Discussion:
[pmacct-discussion] Avro support issues
Jaime Botello
2018-03-17 18:23:49 UTC
Permalink
Hi,

I'm new to this mailing list and my apologies if this is not the right
place to discuss this.

we are trying to build pmacct netflow collector with avro/kafka support.

After meeting all requirements, avro, kafka among other I was finally able
to run ./configure without errors

***@pmacct:~/pmacct$ ./configure --enable-avro --enable-kafka
--enable-jansson

checking whether to enable Avro support... yes
checking for AVRO... yes
checking for avro_record_get in -lavro... yes
checking avro.h usability... yes
checking avro.h presence... yes
checking for avro.h... yes


***@ip-10-180-221-47:~/pmacct$ nfacctd -f netflow_kafka.conf -d
DEBUG: [netflow_kafka.conf] plugin name/type: 'default'/'core'.
DEBUG: [netflow_kafka.conf] plugin name/type: 'default_kafka'/'kafka'.
DEBUG: [netflow_kafka.conf] aggregate:src_host, dst_host, src_port,
dst_port, proto, tos, src_as, dst_as, peer_src_ip, peer_dst_ip, in_iface,
out_iface, src_net, dst_net, src_mask, dst_mask, tcpflags, sampling_rate,
timestamp_start, timestamp_end, timestamp_arrival
DEBUG: [netflow_kafka.conf] nfacctd_port:2055
DEBUG: [netflow_kafka.conf] nfacctd_ip:10.180.222.225
DEBUG: [netflow_kafka.conf] kafka_output:avro
*WARN: [netflow_kafka.conf] 'message_broker_output' set to avro but will
produce no output (missing --enable-avro).*
DEBUG: [netflow_kafka.conf] kafka_topic:netflow
DEBUG: [netflow_kafka.conf] kafka_refresh_time:10
DEBUG: [netflow_kafka.conf] kafka_history:1m
DEBUG: [netflow_kafka.conf] kafka_history_roundoff:m
DEBUG: [netflow_kafka.conf] kafka_broker_host:10.180.221.152
DEBUG: [netflow_kafka.conf] kafka_broker_port:9092
DEBUG: [netflow_kafka.conf] debug:true
INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.1-git
(20180315-00)
INFO ( default/core ): '--enable-mysql' '--enable-kafka'
'--enable-jansson' '--enable-l2' '--enable-ipv6' '--enable-64bit'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/home/ubuntu/pmacct/netflow_kafka.conf'.
INFO ( default_kafka/kafka ): plugin_pipe_size=4096000 bytes
plugin_buffer_size=456 bytes
INFO ( default_kafka/kafka ): ctrl channel: obtained=212992 bytes
target=71856 bytes


After a while and when pmacct is ready to create output, it throws the
following message

DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro

This is my config file for reference

***@pmacct:~/pmacct$ more netflow_kafka.conf
! ..
plugins: kafka
!
aggregate: src_host, dst_host, src_port, dst_port, proto, tos, src_as,
dst_as, peer_src_ip, peer_dst_ip, in_iface, out_iface, src_net, dst_net,
src_mask, dst_mask, tcpflags, sampling_rate, timestamp_start,
timestamp_end, timestamp_arrival
!
nfacctd_port: 2055
nfacctd_ip: 10.180.222.225
!
!
kafka_output: avro
kafka_topic: netflow
kafka_refresh_time: 10
kafka_history: 1m
kafka_history_roundoff: m
kafka_broker_host: 10.180.221.152
kafka_broker_port: 9092

thank in advance for any assistance or pointers
--
*Jaime Botello** (aka Jimbo) *
Paolo Lucente
2018-03-18 10:55:39 UTC
Permalink
Hi Jaime,

Is it possible you are launching a different nfacctd executable (since
you are not using the path), maybe in /usr/local/sbin? The CL switches
you pass to the configure script do not match with those compiled-in the
executable you launch, ie.:

INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.1-git (20180315-00)
INFO ( default/core ): '--enable-mysql' '--enable-kafka'
'--enable-jansson' '--enable-l2' '--enable-ipv6' '--enable-64bit'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'

You are launching an executable compiled with --enable-mysql
--enable-kafka --enable-jansson. Does this ring a bell?

Paolo
Post by Jaime Botello
Hi,
I'm new to this mailing list and my apologies if this is not the right
place to discuss this.
we are trying to build pmacct netflow collector with avro/kafka support.
After meeting all requirements, avro, kafka among other I was finally able
to run ./configure without errors
--enable-jansson
checking whether to enable Avro support... yes
checking for AVRO... yes
checking for avro_record_get in -lavro... yes
checking avro.h usability... yes
checking avro.h presence... yes
checking for avro.h... yes
DEBUG: [netflow_kafka.conf] plugin name/type: 'default'/'core'.
DEBUG: [netflow_kafka.conf] plugin name/type: 'default_kafka'/'kafka'.
DEBUG: [netflow_kafka.conf] aggregate:src_host, dst_host, src_port,
dst_port, proto, tos, src_as, dst_as, peer_src_ip, peer_dst_ip, in_iface,
out_iface, src_net, dst_net, src_mask, dst_mask, tcpflags, sampling_rate,
timestamp_start, timestamp_end, timestamp_arrival
DEBUG: [netflow_kafka.conf] nfacctd_port:2055
DEBUG: [netflow_kafka.conf] nfacctd_ip:10.180.222.225
DEBUG: [netflow_kafka.conf] kafka_output:avro
*WARN: [netflow_kafka.conf] 'message_broker_output' set to avro but will
produce no output (missing --enable-avro).*
DEBUG: [netflow_kafka.conf] kafka_topic:netflow
DEBUG: [netflow_kafka.conf] kafka_refresh_time:10
DEBUG: [netflow_kafka.conf] kafka_history:1m
DEBUG: [netflow_kafka.conf] kafka_history_roundoff:m
DEBUG: [netflow_kafka.conf] kafka_broker_host:10.180.221.152
DEBUG: [netflow_kafka.conf] kafka_broker_port:9092
DEBUG: [netflow_kafka.conf] debug:true
INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.1-git
(20180315-00)
INFO ( default/core ): '--enable-mysql' '--enable-kafka'
'--enable-jansson' '--enable-l2' '--enable-ipv6' '--enable-64bit'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/home/ubuntu/pmacct/netflow_kafka.conf'.
INFO ( default_kafka/kafka ): plugin_pipe_size=4096000 bytes
plugin_buffer_size=456 bytes
INFO ( default_kafka/kafka ): ctrl channel: obtained=212992 bytes
target=71856 bytes
After a while and when pmacct is ready to create output, it throws the
following message
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created due
to missing --enable-avro
This is my config file for reference
! ..
plugins: kafka
!
aggregate: src_host, dst_host, src_port, dst_port, proto, tos, src_as,
dst_as, peer_src_ip, peer_dst_ip, in_iface, out_iface, src_net, dst_net,
src_mask, dst_mask, tcpflags, sampling_rate, timestamp_start,
timestamp_end, timestamp_arrival
!
nfacctd_port: 2055
nfacctd_ip: 10.180.222.225
!
!
kafka_output: avro
kafka_topic: netflow
kafka_refresh_time: 10
kafka_history: 1m
kafka_history_roundoff: m
kafka_broker_host: 10.180.221.152
kafka_broker_port: 9092
thank in advance for any assistance or pointers
--
*Jaime Botello** (aka Jimbo) *
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
Jaime Botello
2018-03-19 07:06:23 UTC
Permalink
Hi Paolo,

thanks, I decided to build pmacct from scratch and the problem went away.
Thank you

We are having a different problem now with kafka/avro and the encoding. I
will start a new email thread and keep it separated.

--Jaime
Post by Paolo Lucente
Hi Jaime,
Is it possible you are launching a different nfacctd executable (since
you are not using the path), maybe in /usr/local/sbin? The CL switches
you pass to the configure script do not match with those compiled-in the
INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.1-git (20180315-00)
INFO ( default/core ): '--enable-mysql' '--enable-kafka'
'--enable-jansson' '--enable-l2' '--enable-ipv6' '--enable-64bit'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
You are launching an executable compiled with --enable-mysql
--enable-kafka --enable-jansson. Does this ring a bell?
Paolo
Post by Jaime Botello
Hi,
I'm new to this mailing list and my apologies if this is not the right
place to discuss this.
we are trying to build pmacct netflow collector with avro/kafka support.
After meeting all requirements, avro, kafka among other I was finally
able
Post by Jaime Botello
to run ./configure without errors
--enable-jansson
checking whether to enable Avro support... yes
checking for AVRO... yes
checking for avro_record_get in -lavro... yes
checking avro.h usability... yes
checking avro.h presence... yes
checking for avro.h... yes
DEBUG: [netflow_kafka.conf] plugin name/type: 'default'/'core'.
DEBUG: [netflow_kafka.conf] plugin name/type: 'default_kafka'/'kafka'.
DEBUG: [netflow_kafka.conf] aggregate:src_host, dst_host, src_port,
dst_port, proto, tos, src_as, dst_as, peer_src_ip, peer_dst_ip, in_iface,
out_iface, src_net, dst_net, src_mask, dst_mask, tcpflags, sampling_rate,
timestamp_start, timestamp_end, timestamp_arrival
DEBUG: [netflow_kafka.conf] nfacctd_port:2055
DEBUG: [netflow_kafka.conf] nfacctd_ip:10.180.222.225
DEBUG: [netflow_kafka.conf] kafka_output:avro
*WARN: [netflow_kafka.conf] 'message_broker_output' set to avro but will
produce no output (missing --enable-avro).*
DEBUG: [netflow_kafka.conf] kafka_topic:netflow
DEBUG: [netflow_kafka.conf] kafka_refresh_time:10
DEBUG: [netflow_kafka.conf] kafka_history:1m
DEBUG: [netflow_kafka.conf] kafka_history_roundoff:m
DEBUG: [netflow_kafka.conf] kafka_broker_host:10.180.221.152
DEBUG: [netflow_kafka.conf] kafka_broker_port:9092
DEBUG: [netflow_kafka.conf] debug:true
INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.1-git (20180315-00)
INFO ( default/core ): '--enable-mysql' '--enable-kafka'
'--enable-jansson' '--enable-l2' '--enable-ipv6' '--enable-64bit'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/home/ubuntu/pmacct/netflow_kafka.conf'.
INFO ( default_kafka/kafka ): plugin_pipe_size=4096000 bytes
plugin_buffer_size=456 bytes
INFO ( default_kafka/kafka ): ctrl channel: obtained=212992 bytes
target=71856 bytes
After a while and when pmacct is ready to create output, it throws the
following message
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
DEBUG ( default_kafka/kafka ): compose_avro(): AVRO object not created
due
Post by Jaime Botello
to missing --enable-avro
This is my config file for reference
! ..
plugins: kafka
!
aggregate: src_host, dst_host, src_port, dst_port, proto, tos, src_as,
dst_as, peer_src_ip, peer_dst_ip, in_iface, out_iface, src_net, dst_net,
src_mask, dst_mask, tcpflags, sampling_rate, timestamp_start,
timestamp_end, timestamp_arrival
!
nfacctd_port: 2055
nfacctd_ip: 10.180.222.225
!
!
kafka_output: avro
kafka_topic: netflow
kafka_refresh_time: 10
kafka_history: 1m
kafka_history_roundoff: m
kafka_broker_host: 10.180.221.152
kafka_broker_port: 9092
thank in advance for any assistance or pointers
--
*Jaime Botello** (aka Jimbo) *
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
--
*Jaime Botello** (aka Jimbo) *// *Riot Games* : Riot Direct // c:
310-210-9772 // summoner: Riot R3lick
Loading...