mirror of
https://github.com/netsampler/goflow2.git
synced 2024-05-06 15:54:52 +00:00
d53e5f9b5a
This commit resolves an issue with the mapping between the ClickHouse schema and the flow.proto schema. In 3326554, the casing of the proto fields was updated, but the ClickHouse column names were not also updated, resulting in the ClickHouse Kafka engine only being able to successfully deserialize fields that without an underscore. Also updates the provisioned dashboards. Versions of the tools are also updated. Co-authored-by: lspgn <lspgn@users.noreply.github.com>
Flows + Kafka + Clicklhouse + Grafana + Prometheus
Clickhouse is a powerful data warehouse.
A sample docker-compose is provided. It's composed of:
- Apache Kafka
- Apache Zookeeper
- GoFlow2
- Prometheus
- Clickhouse
- Grafana
To start the containers, use:
$ docker-compose up
This command will automatically build Grafana and GoFlow2 containers.
GoFlow2 collects NetFlow v9/IPFIX and sFlow packets and sends as a protobuf into Kafka.
Zookeeper coordinates Kafka and can also be used by Clickhouse to ensure replication.
Prometheus scrapes the metrics of the collector.
Clickhouse consumes from Kafka, stores raw data and aggregates over specific columns
using MATERIALIZED TABLES and VIEWS defined in a schema file.
You can visualize the data in Grafana at http://localhost:3000 (credentials: admin/admin) with the pre-made dashboards.