mirror/netsampler-goflow2
1
0
Fork 0
mirror of https://github.com/netsampler/goflow2.git synced 2024-05-06 15:54:52 +00:00
Code Issues Projects Releases Wiki Activity
netsampler-goflow2/compose/kcg
History
Louis ae56e41786 refactor: v2 (#150)
2023-08-09 19:47:20 -07:00
..
clickhouse
refactor: v2 (#150)
2023-08-09 19:47:20 -07:00
grafana
refactor: v2 (#150)
2023-08-09 19:47:20 -07:00
prometheus
compose: update Clickhouse schema to match casing of proto (#133)
2023-03-06 20:39:33 -08:00
docker-compose.yml
refactor: v2 (#150)
2023-08-09 19:47:20 -07:00
README.md
Add new compose stack (#5)
2021-05-31 22:10:05 -07:00

README.md

Flows + Kafka + Clicklhouse + Grafana + Prometheus

Clickhouse is a powerful data warehouse.

A sample docker-compose is provided. It's composed of:

  • Apache Kafka
  • Apache Zookeeper
  • GoFlow2
  • Prometheus
  • Clickhouse
  • Grafana

To start the containers, use:

$ docker-compose up

This command will automatically build Grafana and GoFlow2 containers.

GoFlow2 collects NetFlow v9/IPFIX and sFlow packets and sends as a protobuf into Kafka. Zookeeper coordinates Kafka and can also be used by Clickhouse to ensure replication. Prometheus scrapes the metrics of the collector. Clickhouse consumes from Kafka, stores raw data and aggregates over specific columns using MATERIALIZED TABLES and VIEWS defined in a schema file.

You can visualize the data in Grafana at http://localhost:3000 (credentials: admin/admin) with the pre-made dashboards.