[pmacct-discussion] pmacctd -> netflow/sflow export

[Peter Nixon]

Hi Guys

Has anyone considered adding netflow or sflow export capabilities to XXacctd?
I think they could easily co-exist as extra plugins alongside pgsql, mysql
etc...

This would make pmacct the "onestop show" for netflow/sflow and get rid of the
need to use nprobe (ntop.org) and sflowtool (inmon.com) together with pmacct
for people who don't have netflow/sflow capability in their switches, or who
want to convert between netflow and sflow...

Paolo, How hard would such an addition be?

Cheers

[Ivan A. Beveridge]

Post by Peter Nixon
Has anyone considered adding netflow or sflow export capabilities to XXacctd?
I think they could easily co-exist as extra plugins alongside pgsql, mysql
etc...

Funny you should mention that - I've been thinking of asking that as
recently as today, but I know/assume it would be non-trivial.

I use sfacctd and am likely to need the ability to split our current
(input) sflow streams into multiple (output) sflow streams based on MAC
address.

Post by Peter Nixon
This would make pmacct the "onestop show" for netflow/sflow and get rid of the
need to use nprobe (ntop.org) and sflowtool (inmon.com) together with pmacct
for people who don't have netflow/sflow capability in their switches, or who
want to convert between netflow and sflow...
Paolo, How hard would such an addition be?

This would be *really* cool, because I could do with this functionality
now (WRT sflow), but I'm interested to hear others thoughts (especially
as I don't really have many/any coding skills ;)

Cheers


Ivan
P.S. Converting sflow -> netflow is possible using "sflowtool", but you
lose layer-2 info, obviously.
- --
Ivan Beveridge
<***@linx.net> http://www.linx.net/

[Paolo Lucente]

Hi Ivan and Peter,
while the idea of integrating a kind of sFlow/NetFlow probe has been
already considered (i remember some thoughts recently exchanged with
Sven Anderson about this), i'm somewhat not fully convinced.

In a first instance it will take time as it's absolutely not trivial;
this has been correctly noted by Ivan and somewhat answers to the
question from Peter.

But my point is the following: out there we already have a range of
NetFlow probes, ie. softflowd, nprobe, fprobe to name a few; don't
actually know if exists any for sFlow (out of curiosity, Ivan, can
you confirm this ?). There is also choise for replicators, ie. UDP
samplicator, flow-fanout - which is part of flow-tools.

I know that not every possible scenario is covered, but most common
ones should be adequately fitted by joining these tools, resulting
in a flexible and MODULAR architecture for capturing network data:


------ packets --------- [ netflow probe ] -
|-----
------ NetFlow stream ---------------------- |
|
[ tool A ]------------\ |
[ tool B ]--------------- [ replicator ] ---------
[ pmacct ]------------/

I would underline the modularity: the above scheme shows how you
can pick any of the modules (represented as []) and segregate it
as you like: from a all-in-one box, to each module placed onto a
different box, thus allowing to scale and tailor the capturing
enviroment to even the most busy scenarios.

Integrating either the probe or the replicator into any pmacct
daemon will be certainly useful but would also result in a not
proper approach, in my opinion.

Comments, any thoughts ?

Cheers,
Paolo

[Ivan A. Beveridge]

Hi all,

Post by Paolo Lucente
while the idea of integrating a kind of sFlow/NetFlow probe has been
already considered (i remember some thoughts recently exchanged with
Sven Anderson about this), i'm somewhat not fully convinced.
In a first instance it will take time as it's absolutely not trivial;
this has been correctly noted by Ivan and somewhat answers to the
question from Peter.
But my point is the following: out there we already have a range of
NetFlow probes, ie. softflowd, nprobe, fprobe to name a few; don't
actually know if exists any for sFlow (out of curiosity, Ivan, can
you confirm this ?). There is also choise for replicators, ie. UDP
samplicator, flow-fanout - which is part of flow-tools.

To the best of my knowledge the only open-source software that outputs
sflow data is ntop (I'm not sure whether the "slightly commercialised"
nprobe also does). The only piece of software that I know that converts
sflow -> netflow is inmon's sflowtool. sflowtool will also do fanout
(both sflow -> sflow and sflow -> netflow).

My requirement is to take in sflow data, filter it on MAC address and
output multiple streams based on the filters (1 stream showing in and
out for a particular MAC). This is similar to taking a netflow stream,
splitting into per-port netflow and then exporting these netflow streams
(except more complicated).

Basically what I need must act as a (sflow) collector, filter [I think
pmacct will do both of these] and then act as an agent/probe to convert
that filtered data back to sflow/netflow ... and I can't think that any
tool combination will do this.

I *know* my requirement is non-trivial, and I suspect I'm in a rather
unique position in wanting/needing this (with the possible exception of
other IXes) ... having customers that want to parse their own
sflow/netflow data from our network kit.

Cheers


Ivan
- --
Ivan Beveridge
<***@linx.net> http://www.linx.net/

[Ivan A. Beveridge]

Post by Ivan A. Beveridge

Post by Paolo Lucente
while the idea of integrating a kind of sFlow/NetFlow probe has been
already considered (i remember some thoughts recently exchanged with
Sven Anderson about this), i'm somewhat not fully convinced.
But my point is the following: out there we already have a range of
NetFlow probes, ie. softflowd, nprobe, fprobe to name a few; don't
actually know if exists any for sFlow (out of curiosity, Ivan, can
you confirm this ?). There is also choise for replicators, ie. UDP
samplicator, flow-fanout - which is part of flow-tools.

To the best of my knowledge the only open-source software that outputs
sflow data is ntop (I'm not sure whether the "slightly commercialised"
nprobe also does). The only piece of software that I know that converts
sflow -> netflow is inmon's sflowtool. sflowtool will also do fanout
(both sflow -> sflow and sflow -> netflow).

I should add Como ( http://como.intel-research.net/ ) to that list ..
the sflow v2 support is getting there (I'm trying to help, as an
outsider). Being a "framework" that is where I was/am pinning my hopes
for my particular requirement, but I have not been able to try it yet.

Post by Ivan A. Beveridge
My requirement is to take in sflow data, filter it on MAC address and
output multiple streams based on the filters (1 stream showing in and
out for a particular MAC). This is similar to taking a netflow stream,
splitting into per-port netflow and then exporting these netflow streams
(except more complicated).
Basically what I need must act as a (sflow) collector, filter [I think
pmacct will do both of these] and then act as an agent/probe to convert
that filtered data back to sflow/netflow ... and I can't think that any
tool combination will do this.
I *know* my requirement is non-trivial, and I suspect I'm in a rather
unique position in wanting/needing this (with the possible exception of
other IXes) ... having customers that want to parse their own
sflow/netflow data from our network kit.

Cheers


Ivan
- --
Ivan Beveridge
<***@linx.net> http://www.linx.net/

[Peter Nixon]

Post by Ivan A. Beveridge
Hi all,

Post by Paolo Lucente
while the idea of integrating a kind of sFlow/NetFlow probe has been
already considered (i remember some thoughts recently exchanged with
Sven Anderson about this), i'm somewhat not fully convinced.
In a first instance it will take time as it's absolutely not trivial;
this has been correctly noted by Ivan and somewhat answers to the
question from Peter.
But my point is the following: out there we already have a range of
NetFlow probes, ie. softflowd, nprobe, fprobe to name a few; don't
actually know if exists any for sFlow (out of curiosity, Ivan, can
you confirm this ?). There is also choise for replicators, ie. UDP
samplicator, flow-fanout - which is part of flow-tools.

To the best of my knowledge the only open-source software that outputs
sflow data is ntop (I'm not sure whether the "slightly commercialised"
nprobe also does). The only piece of software that I know that converts
sflow -> netflow is inmon's sflowtool. sflowtool will also do fanout
(both sflow -> sflow and sflow -> netflow).
My requirement is to take in sflow data, filter it on MAC address and
output multiple streams based on the filters (1 stream showing in and
out for a particular MAC). This is similar to taking a netflow stream,
splitting into per-port netflow and then exporting these netflow streams
(except more complicated).
Basically what I need must act as a (sflow) collector, filter [I think
pmacct will do both of these] and then act as an agent/probe to convert
that filtered data back to sflow/netflow ... and I can't think that any
tool combination will do this.

Yes. That was the reason for suggesting a netflow/sflow backend plugin for
pmacct....

Post by Ivan A. Beveridge
I *know* my requirement is non-trivial, and I suspect I'm in a rather
unique position in wanting/needing this (with the possible exception of
other IXes) ... having customers that want to parse their own
sflow/netflow data from our network kit.

Non-trivial yes.. However it is something I think I will need to do also which
is the reason I asked about it :-)

I am also having some technical problems with nprobe that don't exist with
pmacct...

Cheers

[Sven Anderson]

Hi all,

Post by Paolo Lucente
while the idea of integrating a kind of sFlow/NetFlow probe has been
already considered (i remember some thoughts recently exchanged with
Sven Anderson about this), i'm somewhat not fully convinced.

[...]

Post by Paolo Lucente
Integrating either the probe or the replicator into any pmacct
daemon will be certainly useful but would also result in a not
proper approach, in my opinion.
Comments, any thoughts ?

pmacct is already a probe, since you have the pcap interface and, more
important, you do L7 classification, which is clearly a probe-task. As we
discussed before in private mails, I would love to see some features of
pmacct also on a lightweight probe. It would be nice, if pmacct would
evolve to a modular flow processing framework, where you can also compile
a smaller version with pcap import and Netflow/IPFIX/sFlow exporters and
L7 classification API. Maybe it makes sense to tear apart the
probe/exporter and collector parts in general, because in the case you are
running both on the same machine, you anyway don't have performance in
mind, and you can just send it by local IP-traffic from the exporter to
the collector. And the collector part should also offer an abstract data
access API, so that the data processing applications don't depend on the
database and format, that the collector is using.