[pmacct-discussion] Dynamic SQL data

[Andrey Koblyuk]

Hello!

my config

aggregate[storage] : src_host,dst_host,src_port,dst_port,proto,src_as,dst_as,in_iface,out_iface,timestamp_start,timestamp_end
sql_optimize_clauses[storage] : true
sql_table[storage] : traffic_%d%m%Y
sql_table_schema[storage] : /etc/pmacct/nfacct_schema.psql
sql_table_type[storage] : bgp
sql_dont_try_update[storage] : true
sql_use_copy[storage] : true

/etc/pmacct/nfacct_schema.psql:
CREATE TABLE traffic_%d%m%Y (
id_key BIGSERIAL PRIMARY KEY,
as_src BIGINT NOT NULL DEFAULT 0,
as_dst BIGINT NOT NULL DEFAULT 0,
ip_src inet NOT NULL DEFAULT '0.0.0.0',
ip_dst inet NOT NULL DEFAULT '0.0.0.0',
port_src INT NOT NULL DEFAULT 0,
port_dst INT NOT NULL DEFAULT 0,
ip_proto SMALLINT NOT NULL DEFAULT 0,
iface_in INT NOT NULL,
iface_out INT NOT NULL,
packets INT NOT NULL,
bytes BIGINT NOT NULL,
timestamp_start timestamp without time zone NOT NULL DEFAULT '0001-01-01 00:00:00',
timestamp_start_residual INT NOT NULL DEFAULT 0,
timestamp_end timestamp without time zone NOT NULL DEFAULT '0001-01-01 00:00:00',
timestamp_end_residual INT NOT NULL DEFAULT 0,
stamp_inserted timestamp without time zone NOT NULL DEFAULT CURRENT_TIMESTAMP(0)
);
GRANT SELECT, INSERT, UPDATE, DELETE ON traffic_%d%m%Y TO pmacct;

1) first "purge" create table traffic_01011970, but current date 13032018. Why table create on 01011970?
2) next "purge" print this error in log
INFO ( storage/pgsql ): *** Purging cache - START (PID: 11285) ***
ERROR ( storage/pgsql ): ERROR: relation "traffic_01011970" already exists
INFO ( storage/pgsql ): *** Purging cache - END (PID: 11285, QN: 137084/137084, ET: 1) ***

Why does the table try to be recreated with each purge? what parameter is responsible for this?

[Paolo Lucente]

Hi Andrey,

To your questions:

1) 01011970 corresponds to a UNIX timestamp of zero. The easiest way to
mitigate it is to use 'nfacctd_time_new: true' so to consider the
time of arrival at the collector (that is, nfacctd) as timestamp for
bucketing instead of the flow start time; alternatively - or also in
addition - you can look in your flow data in case there are some with
a flow start time of zero (and if not and you can pin point what gets
purged to that table, we may be facing a bug or maybe an explanation
for it ..).

2) It is a simplistic yet harmless approach to blindly send a CREATE
TABLE, maybe the message back should be made a warning rather than an
error. It's harmless because as you see, data is purged anyway; it's
simplistic because it saves the effort to probe if the table exists.

Paolo

Post by Andrey Koblyuk
Hello!
my config
aggregate[storage] : src_host,dst_host,src_port,dst_port,proto,src_as,dst_as,in_iface,out_iface,timestamp_start,timestamp_end
sql_optimize_clauses[storage] : true
sql_table[storage] : traffic_%d%m%Y
sql_table_schema[storage] : /etc/pmacct/nfacct_schema.psql
sql_table_type[storage] : bgp
sql_dont_try_update[storage] : true
sql_use_copy[storage] : true
CREATE TABLE traffic_%d%m%Y (
id_key BIGSERIAL PRIMARY KEY,
as_src BIGINT NOT NULL DEFAULT 0,
as_dst BIGINT NOT NULL DEFAULT 0,
ip_src inet NOT NULL DEFAULT '0.0.0.0',
ip_dst inet NOT NULL DEFAULT '0.0.0.0',
port_src INT NOT NULL DEFAULT 0,
port_dst INT NOT NULL DEFAULT 0,
ip_proto SMALLINT NOT NULL DEFAULT 0,
iface_in INT NOT NULL,
iface_out INT NOT NULL,
packets INT NOT NULL,
bytes BIGINT NOT NULL,
timestamp_start timestamp without time zone NOT NULL DEFAULT '0001-01-01 00:00:00',
timestamp_start_residual INT NOT NULL DEFAULT 0,
timestamp_end timestamp without time zone NOT NULL DEFAULT '0001-01-01 00:00:00',
timestamp_end_residual INT NOT NULL DEFAULT 0,
stamp_inserted timestamp without time zone NOT NULL DEFAULT CURRENT_TIMESTAMP(0)
);
GRANT SELECT, INSERT, UPDATE, DELETE ON traffic_%d%m%Y TO pmacct;
1) first "purge" create table traffic_01011970, but current date 13032018. Why table create on 01011970?
2) next "purge" print this error in log
INFO ( storage/pgsql ): *** Purging cache - START (PID: 11285) ***
ERROR ( storage/pgsql ): ERROR: relation "traffic_01011970" already exists
INFO ( storage/pgsql ): *** Purging cache - END (PID: 11285, QN: 137084/137084, ET: 1) ***
Why does the table try to be recreated with each purge? what parameter is responsible for this?
--
ANK32-RIPE
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[Andrey Koblyuk]

Paolo, thanks for your work!

Post by Paolo Lucente
1) 01011970 corresponds to a UNIX timestamp of zero. The easiest way to
mitigate it is to use 'nfacctd_time_new: true' so to consider the
time of arrival at the collector (that is, nfacctd) as timestamp for

This parameter is set in the config, but the table is still created with timestamp of zero.

Post by Paolo Lucente
bucketing instead of the flow start time; alternatively - or also in
addition - you can look in your flow data in case there are some with
a flow start time of zero (and if not and you can pin point what gets
purged to that table, we may be facing a bug or maybe an explanation
for it ..).

Except for the standard fields IPFIX in the flow there is nothing. template from router in attach.
How can I make collector on the information from the template when creating a table?
I do not insist, but it seems to me that when using the parameter nfacctd_time_new, the table should be created based on the current date. and it's still some kind of bug. :(

Post by Paolo Lucente
2) It is a simplistic yet harmless approach to blindly send a CREATE
TABLE, maybe the message back should be made a warning rather than an
error. It's harmless because as you see, data is purged anyway; it's
simplistic because it saves the effort to probe if the table exists.

i replace CREATE TABLE in schema file to CREATE TABLE IF NOT EXISTS and get NOTICE , not ERROR. it's good :)

also encountered an unknown problem. on every purge flowdata place to table, but in postgreesql log there are such error messages
2018-03-16 10:32:20.694 EET [4736] ***@pmacct ERROR: invalid input syntax for type inet: ""
2018-03-16 10:32:20.694 EET [4736] ***@pmacct CONTEXT: COPY traffic_01011970, line 91719, column ip_src: ""
2018-03-16 10:32:20.694 EET [4736] ***@pmacct STATEMENT: COPY traffic_01011970 (ip_src, ip_dst, as_src, iface_in, iface_out, as_dst, port_src, port_dst, ip_proto, timestamp_start, timestamp_start_residual, timestamp_end, timestamp_end_residual, packets, bytes) FROM STDIN DELIMITER ','
2018-03-16 10:32:30.045 EET [4738] ***@pmacct ERROR: invalid input syntax for type inet: ""
2018-03-16 10:32:30.045 EET [4738] ***@pmacct CONTEXT: COPY traffic_01011970, line 103998, column ip_src: ""
2018-03-16 10:32:30.045 EET [4738] ***@pmacct STATEMENT: COPY traffic_01011970 (ip_src, ip_dst, as_src, iface_in, iface_out, as_dst, port_src, port_dst, ip_proto, timestamp_start, timestamp_start_residual, timestamp_end, timestamp_end_residual, packets, bytes) FROM STDIN DELIMITER ','
2018-03-16 10:32:39.692 EET [4741] ***@pmacct ERROR: invalid input syntax for type inet: ""
2018-03-16 10:32:39.692 EET [4741] ***@pmacct CONTEXT: COPY traffic_01011970, line 113034, column ip_src: ""
2018-03-16 10:32:39.692 EET [4741] ***@pmacct STATEMENT: COPY traffic_01011970 (ip_src, ip_dst, as_src, iface_in, iface_out, as_dst, port_src, port_dst, ip_proto, timestamp_start, timestamp_start_residual, timestamp_end, timestamp_end_residual, packets, bytes) FROM STDIN DELIMITER ','

each time a different line number, but the same column. is this of a problem?

Post by Paolo Lucente
Paolo

Post by Andrey Koblyuk
Hello!
my config
aggregate[storage] : src_host,dst_host,src_port,dst_port,proto,src_as,dst_as,in_iface,out_iface,timestamp_start,timestamp_end
sql_optimize_clauses[storage] : true
sql_table[storage] : traffic_%d%m%Y
sql_table_schema[storage] : /etc/pmacct/nfacct_schema.psql
sql_table_type[storage] : bgp
sql_dont_try_update[storage] : true
sql_use_copy[storage] : true
CREATE TABLE traffic_%d%m%Y (
id_key BIGSERIAL PRIMARY KEY,
as_src BIGINT NOT NULL DEFAULT 0,
as_dst BIGINT NOT NULL DEFAULT 0,
ip_src inet NOT NULL DEFAULT '0.0.0.0',
ip_dst inet NOT NULL DEFAULT '0.0.0.0',
port_src INT NOT NULL DEFAULT 0,
port_dst INT NOT NULL DEFAULT 0,
ip_proto SMALLINT NOT NULL DEFAULT 0,
iface_in INT NOT NULL,
iface_out INT NOT NULL,
packets INT NOT NULL,
bytes BIGINT NOT NULL,
timestamp_start timestamp without time zone NOT NULL DEFAULT '0001-01-01 00:00:00',
timestamp_start_residual INT NOT NULL DEFAULT 0,
timestamp_end timestamp without time zone NOT NULL DEFAULT '0001-01-01 00:00:00',
timestamp_end_residual INT NOT NULL DEFAULT 0,
stamp_inserted timestamp without time zone NOT NULL DEFAULT CURRENT_TIMESTAMP(0)
);
GRANT SELECT, INSERT, UPDATE, DELETE ON traffic_%d%m%Y TO pmacct;
1) first "purge" create table traffic_01011970, but current date 13032018. Why table create on 01011970?
2) next "purge" print this error in log
INFO ( storage/pgsql ): *** Purging cache - START (PID: 11285) ***
ERROR ( storage/pgsql ): ERROR: relation "traffic_01011970" already exists
INFO ( storage/pgsql ): *** Purging cache - END (PID: 11285, QN: 137084/137084, ET: 1) ***
Why does the table try to be recreated with each purge? what parameter is responsible for this?
--
ANK32-RIPE
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[Paolo Lucente]

Hi Andrey,

Post by Andrey Koblyuk
Except for the standard fields IPFIX in the flow there is nothing. template from router in attach.
How can I make collector on the information from the template when creating a table?
I do not insist, but it seems to me that when using the parameter nfacctd_time_new, the table should be created based on the current date. and it's still some kind of bug. :(

Then you may be missing the sql_history and sql_history_roundoff then.
If also this is not the case then it's either the flow (i'd need a
sample of your flow data, not the template) or a bug.

Post by Andrey Koblyuk

Post by Paolo Lucente
2) It is a simplistic yet harmless approach to blindly send a CREATE
TABLE, maybe the message back should be made a warning rather than an
error. It's harmless because as you see, data is purged anyway; it's
simplistic because it saves the effort to probe if the table exists.

i replace CREATE TABLE in schema file to CREATE TABLE IF NOT EXISTS and get NOTICE , not ERROR. it's good :)

Fantastic, thanks for the feedback.

Paolo

[Andrey Koblyuk]

Post by Paolo Lucente
Then you may be missing the sql_history and sql_history_roundoff then.

Paolo, thank you. Created table has correct name.

Which is strange - with my config
sql_table[storage] : traffic_%d%m%Y
sql_table_schema[storage] : /etc/pmacct/nfacct_schema.psql
sql_table_type[storage] : bgp
sql_dont_try_update[storage] : true
sql_use_copy[storage] : true
sql_history[storage] : 1d
sql_history_roundoff[storage] : d

data purged in table in one minute range (based on stamp_updated?) . as I tested it:
on sql server i periodically(once per 10sec ) run SQL QUERY (9-10 queryes), such as
select * from traffic_16032018 order by id_key desc limit 1;
and see line in attach1.txt. after minute data in table changed and i see (same query) line in attach2.txt.
In the daemon log could pass 10-15 INFO ( storage/pgsql ): Purging cache - START/END calls.
Is this normal behavior?


I will also be very grateful for your comment about the errors in the sql server logs that I sended at the end of my previous email.

[Paolo Lucente]

Hi Andrey,

Data is purged to the database each sql_refresh_time, by default 60
secs so this is expected behaviour. You are free to redefine this to a
different value of course.

On the error from your other email: any chance you can post what is the
problematic column (since you said it's always the same one)? It would
allow to formulate an idea and/or reproduce the issue. I can confirm the
error as it is spit out does not ring a bell to me.

Paolo

Post by Andrey Koblyuk

Post by Paolo Lucente
Then you may be missing the sql_history and sql_history_roundoff then.

Paolo, thank you. Created table has correct name.
Which is strange - with my config
sql_table[storage] : traffic_%d%m%Y
sql_table_schema[storage] : /etc/pmacct/nfacct_schema.psql
sql_table_type[storage] : bgp
sql_dont_try_update[storage] : true
sql_use_copy[storage] : true
sql_history[storage] : 1d
sql_history_roundoff[storage] : d
on sql server i periodically(once per 10sec ) run SQL QUERY (9-10 queryes), such as
select * from traffic_16032018 order by id_key desc limit 1;
and see line in attach1.txt. after minute data in table changed and i see (same query) line in attach2.txt.
In the daemon log could pass 10-15 INFO ( storage/pgsql ): Purging cache - START/END calls.
Is this normal behavior?
I will also be very grateful for your comment about the errors in the sql server logs that I sended at the end of my previous email.
--
ANK32-RIPE
id_key | as_src | as_dst | ip_src | ip_dst | port_src | port_dst | ip_proto | iface_in | iface_out | packets | bytes | timestamp_start | timestamp_start_residual | timestamp_end | timestamp_end_residual | stamp_updated | stamp_inserted
---------+--------+--------+----------------+----------------+----------+----------+----------+----------+-----------+---------+-------+---------------------+--------------------------+---------------------+------------------------+---------------------+---------------------
8751053 | 57774 | 14061 | 176.107.196.58 | 165.227.129.48 | 57818 | 8080 | 6 | 546 | 725 | 1 | 40 | 2018-03-16 13:56:37 | 634000 | 2018-03-16 13:56:37 | 634000 | 2018-03-16 13:59:19 | 2018-03-16 00:00:00
id_key | as_src | as_dst | ip_src | ip_dst | port_src | port_dst | ip_proto | iface_in | iface_out | packets | bytes | timestamp_start | timestamp_start_residual | timestamp_end | timestamp_end_residual | stamp_updated | stamp_inserted
---------+--------+--------+-------------+----------------+----------+----------+----------+----------+-----------+---------+-------+---------------------+--------------------------+---------------------+------------------------+---------------------+---------------------
9126683 | 9198 | 8870 | 5.251.17.62 | 93.171.243.123 | 13463 | 55127 | 6 | 725 | 546 | 2 | 96 | 2018-03-16 13:46:54 | 181000 | 2018-03-16 13:47:00 | 217000 | 2018-03-16 14:00:19 | 2018-03-16 00:00:00

[Andrey Koblyuk]

Hi Paolo,

Post by Paolo Lucente
On the error from your other email: any chance you can post what is the
problematic column (since you said it's always the same one)? It would
allow to formulate an idea and/or reproduce the issue. I can confirm the
error as it is spit out does not ring a bell to me.

This error appears once on each operation
INFO ( storage/pgsql ): *** Purging cache - START (PID: 4927) ***
INFO ( storage/pgsql ): *** Purging cache - END (PID: 4927, QN: 69211/69211, ET: 0) ***

and yes - she always appears at the same column (ip_src). only the line number differs.

2018-03-19 08:47:34.339 EET [4953] ***@pmacct ERROR: invalid input syntax for type inet: ""
2018-03-19 08:47:34.339 EET [4953] ***@pmacct CONTEXT: COPY traffic_19032018, line 9781, column ip_src: ""
2018-03-19 08:47:34.339 EET [4953] ***@pmacct STATEMENT: COPY traffic_19032018 (stamp_updated, stamp_inserted, ip_src, ip_dst, as_src, iface_in, iface_out, as_dst, port_src, port_dst, ip_proto, timestamp_start, timestamp_start_residual, timestamp_end, timestamp_end_residual, packets, bytes) FROM STDIN DELIMITER ','

[Paolo Lucente]

Hi Andrey,

Yes what i meant is: any chance you can try to understand what precisely
is the problematic value? If it's a specific one or there is a set of
values that give problems? Having this info would put us in pole
position for troubleshooting the issue. Since the field type is 'inet'
i'm ruling out this is just about v6 addresses.

Paolo

Post by Andrey Koblyuk
Hi Paolo,

Post by Paolo Lucente
On the error from your other email: any chance you can post what is the
problematic column (since you said it's always the same one)? It would
allow to formulate an idea and/or reproduce the issue. I can confirm the
error as it is spit out does not ring a bell to me.

This error appears once on each operation
INFO ( storage/pgsql ): *** Purging cache - START (PID: 4927) ***
INFO ( storage/pgsql ): *** Purging cache - END (PID: 4927, QN: 69211/69211, ET: 0) ***
and yes - she always appears at the same column (ip_src). only the line number differs.
--
ANK32-RIPE

[Andrey Koblyuk]

Здравствуйте, Paolo.

Post by Paolo Lucente
Hi Andrey,
Yes what i meant is: any chance you can try to understand what precisely
is the problematic value? If it's a specific one or there is a set of
values that give problems? Having this info would put us in pole
position for troubleshooting the issue. Since the field type is 'inet'
i'm ruling out this is just about v6 addresses.

Flow contains only ipv4 data. Results with the debug option enabled:

First purge after daemon start:
Error:
2018-03-21 09:09:22.167 EET [19801] ***@pmacct CONTEXT: COPY traffic_21032018, line 1, column ip_src: ""
Data (a few lines):
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,,,0,0,0,0,0,0,0,2018-03-21 09:09:05,0,1970-01-01 03:00:00,0,0,0 ---??????
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,191.234.72.188,94.125.122.215,12883,725,507,12883,443,49896,6,2018-03-21 09:07:36,164000,2018-03-21 09:08:45,586000,91,53509
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,212.90.184.234,185.16.28.49,12883,507,725,12883,39748,50737,6,2018-03-21 09:08:39,323000,2018-03-21 09:08:40,427000,3,120
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,35.158.154.235,212.90.176.70,12883,725,507,12883,443,38889,6,2018-03-21 09:07:40,917000,2018-03-21 09:08:46,44000,12,1147

Error:
2018-03-21 09:09:33.265 EET [19804] ***@pmacct ERROR: invalid input syntax for type inet: ""
2018-03-21 09:09:33.265 EET [19804] ***@pmacct CONTEXT: COPY traffic_21032018, line 44702, column ip_src: ""
Data:
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,212.90.175.202,31.134.211.47,12883,507,755,56823,0,0,1,2018-03-21 09:08:11,516000,2018-03-21 09:08:20,989000,3,228
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,212.90.162.142,185.158.208.144,12883,507,755,12883,56257,443,6,2018-03-21 09:09:12,776000,2018-03-21 09:09:12,817000,101,6713
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,,,12883,0,0,15497,0,0,0,2018-03-21 09:09:15,0,1970-01-01 03:00:00,0,0,0 ---??????
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,2.62.60.114,171.25.204.96,12389,725,546,12883,56156,37777,6,2018-03-21 09:07:42,644000,2018-03-21 09:07:45,637000,2,104
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,212.90.183.149,120.29.215.38,12883,507,725,12883,53,19131,17,2018-03-21 09:04:59,926000,2018-03-21 09:04:59,995000,25,37500

I see that under strange conditions the daemon generates an incorrect data string that does not contain valid data (except the fields src_as,dst_as,timestamp_start)...

[Paolo Lucente]

Hi Andrey,

So there are some rows that contain empty ip addresses and this is
causing the issue. I don't have a full vision of your config so i can
not guess what could be the cause - please send it over. It could also
be not a config issue but rather some flow spit by the exporting IPFIX
device.

Paolo

Post by Andrey Koblyuk
Здравствуйте, Paolo.

Post by Paolo Lucente
Hi Andrey,
Yes what i meant is: any chance you can try to understand what precisely
is the problematic value? If it's a specific one or there is a set of
values that give problems? Having this info would put us in pole
position for troubleshooting the issue. Since the field type is 'inet'
i'm ruling out this is just about v6 addresses.

DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,,,0,0,0,0,0,0,0,2018-03-21 09:09:05,0,1970-01-01 03:00:00,0,0,0 ---??????
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,191.234.72.188,94.125.122.215,12883,725,507,12883,443,49896,6,2018-03-21 09:07:36,164000,2018-03-21 09:08:45,586000,91,53509
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,212.90.184.234,185.16.28.49,12883,507,725,12883,39748,50737,6,2018-03-21 09:08:39,323000,2018-03-21 09:08:40,427000,3,120
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,35.158.154.235,212.90.176.70,12883,725,507,12883,443,38889,6,2018-03-21 09:07:40,917000,2018-03-21 09:08:46,44000,12,1147
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,212.90.175.202,31.134.211.47,12883,507,755,56823,0,0,1,2018-03-21 09:08:11,516000,2018-03-21 09:08:20,989000,3,228
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,212.90.162.142,185.158.208.144,12883,507,755,12883,56257,443,6,2018-03-21 09:09:12,776000,2018-03-21 09:09:12,817000,101,6713
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,,,12883,0,0,15497,0,0,0,2018-03-21 09:09:15,0,1970-01-01 03:00:00,0,0,0 ---??????
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,2.62.60.114,171.25.204.96,12389,725,546,12883,56156,37777,6,2018-03-21 09:07:42,644000,2018-03-21 09:07:45,637000,2,104
DEBUG ( storage/pgsql ): 2018-03-21 09:09:22,2018-03-21 00:00:00,212.90.183.149,120.29.215.38,12883,507,725,12883,53,19131,17,2018-03-21 09:04:59,926000,2018-03-21 09:04:59,995000,25,37500
I see that under strange conditions the daemon generates an incorrect data string that does not contain valid data (except the fields src_as,dst_as,timestamp_start)...
--
ANK32-RIPE

[Andrey Koblyuk]

This post might be inappropriate. Click to display it.

[Paolo Lucente]

Hi Andrey,

'nfacctd_account_options: true' is likely causing this. You should split
flow records from options - since you would need different schemas for
each. See the 'Va. NetFlow daemon & accounting NetFlow v9/IPFIX options'
chapter in https://github.com/pmacct/pmacct/blob/master/QUICKSTART for
that. See the role of tagging (pre_tag_map), filtering/splitting basing
on the tag (pre_tag_filter) and the two plugin instances.

Paolo

Post by Andrey Koblyuk
Здравствуйте, Paolo.

Post by Paolo Lucente
Hi Andrey,
So there are some rows that contain empty ip addresses and this is
causing the issue. I don't have a full vision of your config so i can
not guess what could be the cause - please send it over. It could also
be not a config issue but rather some flow spit by the exporting IPFIX
device.

pidfile : /var/run/nfacctd.pid
nfacctd_port : 2205
nfacctd_time_new : true
nfacctd_account_options : true
nfacctd_as : bgp
nfacctd_peer_as : bgp
#debug:true
bgp_daemon : true
bgp_stdcomm_pattern_to_asn : XXXXX:XXXXX
bgp_daemon_ip : YY.YY.YY.YY
sql_db : pmacct
sql_host : localhost
sql_user : ZZZZZZZZZZZZ
sql_passwd : ZZZZZZZZZZZZ
plugins : pgsql[storage]
aggregate[storage] : src_host,dst_host,src_port,dst_port,proto,src_as,dst_as,in_iface,out_iface,timestamp_start,timestamp_end
sql_optimize_clauses[storage] : true
sql_table[storage] : traffic_%d%m%Y
sql_table_schema[storage] : /etc/pmacct/nfacct_schema.psql
sql_table_type[storage] : bgp
sql_dont_try_update[storage] : true
sql_use_copy[storage] : true
sql_history[storage] : 1d
sql_history_roundoff[storage] : d
sql_cache_entries[storage] : 131084
If this is a problem with the ipfix stream, is it possible to configure the replacement of null values with 0.0.0.0? or somehow indicate the demon - so that he ignores such flows?
--
ANK32-RIPE