[pmacct-discussion] Wait for BGP peering and massiv ipfix data

[Andrey Koblyuk]

Hi All!

Config:

bgp_daemon : true
bgp_stdcomm_pattern_to_asn : MYASN:MYASN
plugins : pgsql[storage]

aggregate[storage] : src_host,dst_host,src_port,dst_port,proto,src_as,dst_as,in_iface,out_iface
sql_optimize_clauses[storage] : true
sql_table[storage] : all_traffic
sql_table_type[storage] : bgp
sql_dont_try_update[storage] : true
sql_use_copy[storage] : true


1) sql_startup_delay does not work for me . I would like to postpone the first data processing/cache purging before BGP peering is up. otherwise the table contains data without information from BGP daemon.
2) is it possible to rotate tables, for example, once a week, simply by creating a new one with a new name (example all_traffic_%m) ? In this case, no aggregation should occur - therefore the parameters are not suitable
sql_history: 1h
sql_history_roundoff: h
sql_table: acct_v4_%w
for me if I correctly understood their purpose.
3) I also noticed a "strange" delay between "Purging cache - END" and psql "select * from all_traffic;". about a minute.. O_O this is a question for nfacctd or postgresql?
4) Interested in your recommendations - what parameters should be changed with a large flow of ipfix data, taking into account the fact that I plan to add plug-ins for aggregating this data to the existing pgsql[storage].

[Paolo Lucente]

Hi Andrey,

Post by Andrey Koblyuk
1) sql_startup_delay does not work for me . I would like to postpone the first data processing/cache purging before BGP peering is up. otherwise the table contains data without information from BGP daemon.

Correlation happens when flow data is received - not upon purge. So
simply delaying the purge is not going to benefit. I wonder, is it such
a big deal though? How often are you restarting the collector and why?

Post by Andrey Koblyuk
2) is it possible to rotate tables, for example, once a week, simply by creating a new one with a new name (example all_traffic_%m) ? In this case, no aggregation should occur - therefore the parameters are not suitable
sql_history: 1h
sql_history_roundoff: h
sql_table: acct_v4_%w
for me if I correctly understood their purpose.

Please elaborate more. 'sql_table: acct_v4_%w' should achieve you the
rotation behaviour you desire. I loose you on the aggregation point you
make.

Post by Andrey Koblyuk
3) I also noticed a "strange" delay between "Purging cache - END" and psql "select * from all_traffic;". about a minute.. O_O this is a question for nfacctd or postgresql?

I'd definitely need more info on this as it's too generic description.
If you suspect a bug and can provide more info, please follow-up by
unicast email.

Post by Andrey Koblyuk
4) Interested in your recommendations - what parameters should be changed with a large flow of ipfix data, taking into account the fact that I plan to add plug-ins for aggregating this data to the existing pgsql[storage].

Mainly buffers (ie. plugin_pipe_szie, plugin_buffer_size or as an
alternative plugin_pipe_zmq along with plugin_pipe_zmq_profile) and,
depending on your aggregation method, the amount of cache entries (ie.
sql_cache_entries). On the buffering part you can read the following:

https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2153-L2168
https://github.com/pmacct/pmacct/blob/master/CONFIG-KEYS#L286-L331

Paolo

[Andrey Koblyuk]

Hi Paolo!

Post by Paolo Lucente

Post by Andrey Koblyuk
1) sql_startup_delay does not work for me . I would like to postpone the first data processing/cache purging before BGP peering is up. otherwise the table contains data without information from BGP daemon.

Correlation happens when flow data is received - not upon purge. So
simply delaying the purge is not going to benefit. I wonder, is it such
a big deal though? How often are you restarting the collector and why?

is there a possibility to run a BGP demon standalone earlier than nfacctd?
with my current configuration, the following occurs - on begin run nfaccd :
NFO ( storage/pgsql ): cache entries=131084 base cache memory=47286240 bytes
INFO ( storage/pgsql ): *** Purging cache - START (PID: 12464) ***
INFO ( storage/pgsql ): *** Purging cache - END (PID: 12464, QN: 51217/51217, ET: 0) ***
INFO ( storage/pgsql ): Finished cache entries (ie. sql_cache_entries). Purging.
INFO ( storage/pgsql ): *** Purging cache - START (PID: 12466) ***
INFO ( storage/pgsql ): *** Purging cache - END (PID: 12466, QN: 137084/137084, ET: 0) ***
INFO ( default/core/BGP ): [MY_RR_IP_ADDR] BGP peers usage: 1/4
INFO ( default/core/BGP ): [MY_RR_IP_ADDR] BGP_OPEN: Local AS: 12593 Remote AS: 12593 HoldTime: 90

can not get into BGP "holdtime event" receive from RR. in "production" - collector always run, does not restart.
a problem only at the start of work (for the time of tests several times it was necessary to observe such a problem).

Post by Paolo Lucente

Post by Andrey Koblyuk
2) is it possible to rotate tables, for example, once a week, simply by creating a new one with a new name (example all_traffic_%m) ? In this case, no aggregation should occur - therefore the parameters are not suitable
for me if I correctly understood their purpose.

Please elaborate more. 'sql_table: acct_v4_%w' should achieve you the
rotation behaviour you desire. I loose you on the aggregation point you
make.

sorry typo. of course %w. The problem is that i need to rotate the table, but do not aggregate this table. why - to reduce the number of entries in the active table (after two days of working with my ipfix flow, the number of records in the sql_table reached ~50 millions). So the question is whether the table will rotate without using aggregation parameters, such as sql_history/sql_history_roundoff?

[Paolo Lucente]

Hi Andrey,

Post by Andrey Koblyuk
is there a possibility to run a BGP demon standalone earlier than nfacctd?

Yes, you have the pmbgpd daemon for that (standalone BGP daemon). But
i think it's not going to solve you anything for what you are trying to
achieve (ie. only solution is not to restart the daemon often enough so
that restarts are impacting on the quality of data). With the standalone
daemon you could save BGP data separately from flow data, then reconcile
them at your leisure as part of post-processing; or you can xconnect BGP
peers through pmbgpd to a nfacctd collector. See:

https://github.com/pmacct/pmacct/blob/master/CONFIG-KEYS#L2473-L2484

Post by Andrey Koblyuk
sorry typo. of course %w. The problem is that i need to rotate the table, but do not aggregate this table. why - to reduce the number of entries in the active table (after two days of working with my ipfix flow, the number of records in the sql_table reached ~50 millions). So the question is whether the table will rotate without using aggregation parameters, such as sql_history/sql_history_roundoff?

There is something i struggle in your elaboration: you say you wish to
not aggregate data in the table but at the same time you say it is a
problem to get ~50 millions in the table. Can you make an example to
describe your desired behaviour?

Paolo

[Andrey Koblyuk]

Paolo, thanks for your reply!

Post by Paolo Lucente

Post by Andrey Koblyuk
sorry typo. of course %w. The problem is that i need to rotate the table, but do not aggregate this table. why - to reduce the number of entries in the active table (after two days of working with my ipfix flow, the number of records in the sql_table reached ~50 millions). So the question is whether the table will rotate without using aggregation parameters, such as sql_history/sql_history_roundoff?

There is something i struggle in your elaboration: you say you wish to
not aggregate data in the table but at the same time you say it is a
problem to get ~50 millions in the table. Can you make an example to
describe your desired behaviour?

I would like to create a table once a week with a new name (for example,acct_%w%Y), which would store non-historical data from
aggregate[storage] : src_host,dst_host,src_port,dst_port,proto,src_as,dst_as,in_iface,out_iface
sql_optimize_clauses[storage] : true
sql_table[storage] : all_traffic
sql_table_type[storage] : bgp
sql_dont_try_update[storage] : true
sql_use_copy[storage] : true

Will the directives sql_table_schema/sql_table_with_dynamic_name work without sql_history_roundoff/sql_history?

[Paolo Lucente]

Hi Andrey,

Post by Andrey Koblyuk
I would like to create a table once a week with a new name (for example,acct_%w%Y), which would store non-historical data from
aggregate[storage] : src_host,dst_host,src_port,dst_port,proto,src_as,dst_as,in_iface,out_iface
sql_optimize_clauses[storage] : true
sql_table[storage] : all_traffic
sql_table_type[storage] : bgp
sql_dont_try_update[storage] : true
sql_use_copy[storage] : true
Will the directives sql_table_schema/sql_table_with_dynamic_name work without sql_history_roundoff/sql_history?

To aggregate flows in the weekly time bin (which i seem to understand is
not what you want, so just mentioning), use the above config including
a definition for sql_history and sql_history_roundoff. To not aggretate
flows (which i seem to understand is what you want), then please add
'timestamp_start, timestamp_end' to your 'aggregate' line. So, in short,
both behaviours are available and can be achieved.

Paolo