Jaime Botello
2018-03-17 18:23:49 UTC
Hi,
I'm new to this mailing list and my apologies if this is not the right
place to discuss this.
we are trying to build pmacct netflow collector with avro/kafka support.
After meeting all requirements, avro, kafka among other I was finally able
to run ./configure without errors
***@pmacct:~/pmacct$ ./configure --enable-avro --enable-kafka
--enable-jansson
checking whether to enable Avro support... yes
checking for AVRO... yes
checking for avro_record_get in -lavro... yes
checking avro.h usability... yes
checking avro.h presence... yes
checking for avro.h... yes
***@ip-10-180-221-47:~/pmacct$ nfacctd -f netflow_kafka.conf -d
DEBUG: [netflow_kafka.conf] plugin name/type: 'default'/'core'.
DEBUG: [netflow_kafka.conf] plugin name/type: 'default_kafka'/'kafka'.
DEBUG: [netflow_kafka.conf] aggregate:src_host, dst_host, src_port,
dst_port, proto, tos, src_as, dst_as, peer_src_ip, peer_dst_ip, in_iface,
out_iface, src_net, dst_net, src_mask, dst_mask, tcpflags, sampling_rate,
timestamp_start, timestamp_end, timestamp_arrival
DEBUG: [netflow_kafka.conf] nfacctd_port:2055
DEBUG: [netflow_kafka.conf] nfacctd_ip:10.180.222.225
DEBUG: [netflow_kafka.conf] kafka_output:avro
*WARN: [netflow_kafka.conf] 'message_broker_output' set to avro but will
produce no output (missing --enable-avro).*
DEBUG: [netflow_kafka.conf] kafka_topic:netflow
DEBUG: [netflow_kafka.conf] kafka_refresh_time:10
DEBUG: [netflow_kafka.conf] kafka_history:1m
DEBUG: [netflow_kafka.conf] kafka_history_roundoff:m
DEBUG: [netflow_kafka.conf] kafka_broker_host:10.180.221.152
DEBUG: [netflow_kafka.conf] kafka_broker_port:9092
DEBUG: [netflow_kafka.conf] debug:true
INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.1-git
(20180315-00)
INFO ( default/core ): '--enable-mysql' '--enable-kafka'
'--enable-jansson' '--enable-l2' '--enable-ipv6' '--enable-64bit'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/home/ubuntu/pmacct/netflow_kafka.conf'.
INFO ( default_kafka/kafka ): plugin_pipe_size=4096000 bytes
plugin_buffer_size=456 bytes
INFO ( default_kafka/kafka ): ctrl channel: obtained=212992 bytes
target=71856 bytes
After a while and when pmacct is ready to create output, it throws the
following message
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
This is my config file for reference
***@pmacct:~/pmacct$ more netflow_kafka.conf
! ..
plugins: kafka
!
aggregate: src_host, dst_host, src_port, dst_port, proto, tos, src_as,
dst_as, peer_src_ip, peer_dst_ip, in_iface, out_iface, src_net, dst_net,
src_mask, dst_mask, tcpflags, sampling_rate, timestamp_start,
timestamp_end, timestamp_arrival
!
nfacctd_port: 2055
nfacctd_ip: 10.180.222.225
!
!
kafka_output: avro
kafka_topic: netflow
kafka_refresh_time: 10
kafka_history: 1m
kafka_history_roundoff: m
kafka_broker_host: 10.180.221.152
kafka_broker_port: 9092
thank in advance for any assistance or pointers
I'm new to this mailing list and my apologies if this is not the right
place to discuss this.
we are trying to build pmacct netflow collector with avro/kafka support.
After meeting all requirements, avro, kafka among other I was finally able
to run ./configure without errors
***@pmacct:~/pmacct$ ./configure --enable-avro --enable-kafka
--enable-jansson
checking whether to enable Avro support... yes
checking for AVRO... yes
checking for avro_record_get in -lavro... yes
checking avro.h usability... yes
checking avro.h presence... yes
checking for avro.h... yes
***@ip-10-180-221-47:~/pmacct$ nfacctd -f netflow_kafka.conf -d
DEBUG: [netflow_kafka.conf] plugin name/type: 'default'/'core'.
DEBUG: [netflow_kafka.conf] plugin name/type: 'default_kafka'/'kafka'.
DEBUG: [netflow_kafka.conf] aggregate:src_host, dst_host, src_port,
dst_port, proto, tos, src_as, dst_as, peer_src_ip, peer_dst_ip, in_iface,
out_iface, src_net, dst_net, src_mask, dst_mask, tcpflags, sampling_rate,
timestamp_start, timestamp_end, timestamp_arrival
DEBUG: [netflow_kafka.conf] nfacctd_port:2055
DEBUG: [netflow_kafka.conf] nfacctd_ip:10.180.222.225
DEBUG: [netflow_kafka.conf] kafka_output:avro
*WARN: [netflow_kafka.conf] 'message_broker_output' set to avro but will
produce no output (missing --enable-avro).*
DEBUG: [netflow_kafka.conf] kafka_topic:netflow
DEBUG: [netflow_kafka.conf] kafka_refresh_time:10
DEBUG: [netflow_kafka.conf] kafka_history:1m
DEBUG: [netflow_kafka.conf] kafka_history_roundoff:m
DEBUG: [netflow_kafka.conf] kafka_broker_host:10.180.221.152
DEBUG: [netflow_kafka.conf] kafka_broker_port:9092
DEBUG: [netflow_kafka.conf] debug:true
INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.1-git
(20180315-00)
INFO ( default/core ): '--enable-mysql' '--enable-kafka'
'--enable-jansson' '--enable-l2' '--enable-ipv6' '--enable-64bit'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/home/ubuntu/pmacct/netflow_kafka.conf'.
INFO ( default_kafka/kafka ): plugin_pipe_size=4096000 bytes
plugin_buffer_size=456 bytes
INFO ( default_kafka/kafka ): ctrl channel: obtained=212992 bytes
target=71856 bytes
After a while and when pmacct is ready to create output, it throws the
following message
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
This is my config file for reference
***@pmacct:~/pmacct$ more netflow_kafka.conf
! ..
plugins: kafka
!
aggregate: src_host, dst_host, src_port, dst_port, proto, tos, src_as,
dst_as, peer_src_ip, peer_dst_ip, in_iface, out_iface, src_net, dst_net,
src_mask, dst_mask, tcpflags, sampling_rate, timestamp_start,
timestamp_end, timestamp_arrival
!
nfacctd_port: 2055
nfacctd_ip: 10.180.222.225
!
!
kafka_output: avro
kafka_topic: netflow
kafka_refresh_time: 10
kafka_history: 1m
kafka_history_roundoff: m
kafka_broker_host: 10.180.221.152
kafka_broker_port: 9092
thank in advance for any assistance or pointers
--
*Jaime Botello** (aka Jimbo) *
*Jaime Botello** (aka Jimbo) *