[pmacct-discussion] uacctd memory usage

[Alex K]

Hi all,

I have a setup with uacctd monitoring traffic of several interfaces through
NFLOG.
With uacctd stopped I see that the server (a relatively small device with 4
GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes up
to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.

Is there any tweaks I can use to put a limit on the memory usage of uacctd.

Thanx,
Alex

[Dariush Marsh-Mossadeghi]

Post by Alex K
Hi all,
I have a setup with uacctd monitoring traffic of several interfaces through NFLOG.
With uacctd stopped I see that the server (a relatively small device with 4 GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes up to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
Is there any tweaks I can use to put a limit on the memory usage of uacctd.
Thanx,
Alex
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

tldr; post the output of the free command when uacctd is running and I’ll do my best to interpret it for you :-)

Is it _really_ using it, or is it just not completely unused?

The linux kernel generally does pretty good job of keeping stuff that might be useful in memory, but getting rid of it very very quickly if the space is needed for something else.
The challenge you face is that most of the userspace tools come with a long list of caveats about what they appear to report vs what’s really happening, this is mainly due to the way the kernel shares memory between processes, and its use of as much memory as possible for various caches.

Some threads worth reading if you’re not familiar with the in’s and out’s of linux kernel memory management...

https://www.linuxatemyram.com/ <https://www.linuxatemyram.com/>
https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage <https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage>
https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/ <https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/>

HTH
Dariush

[Alex K]

Thanx for the reply.

The output of free is the following:

free
total used free shared buff/cache
available
Mem: 4046572 2832576 152012 784240 1061984
204248
Swap: 3906556 1086080 2820476

While is as below when stopped:

free
total used free shared buff/cache
available
Mem: 4046572 520352 3223884 14916 302336
3286952
Swap: 3906556 485040 3421516

Seems that mysql plugins are reserving quite some memory as they list first
in htop when sorted with memory.

Thanx,
Alex


On Mon, Jun 25, 2018 at 2:22 PM, Dariush Marsh-Mossadeghi <

Post by Alex K
Hi all,
I have a setup with uacctd monitoring traffic of several interfaces through NFLOG.
With uacctd stopped I see that the server (a relatively small device with
4 GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes
up to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
Is there any tweaks I can use to put a limit on the memory usage of uacctd.
Thanx,
Alex
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
tldr; post the output of the free command when uacctd is running and I’ll
do my best to interpret it for you :-)
Is it _really_ using it, or is it just not completely unused?
The linux kernel generally does pretty good job of keeping stuff that
might be useful in memory, but getting rid of it very very quickly if the
space is needed for something else.
The challenge you face is that most of the userspace tools come with a
long list of caveats about what they appear to report vs what’s really
happening, this is mainly due to the way the kernel shares memory between
processes, and its use of as much memory as possible for various caches.
Some threads worth reading if you’re not familiar with the in’s and out’s
of linux kernel memory management...
https://www.linuxatemyram.com/
https://stackoverflow.com/questions/4802481/how-to-see-
top-processes-sorted-by-actual-memory-usage
https://stackoverflow.com/questions/3784974/want-to-
know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/
HTH
Dariush
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[Dariush Marsh-Mossadeghi]

OK, so we’re moving from bottom-posting to top-posting
 that’ll make it interesting for other readers ;-)

The output of free doesn’t look desperate, but it is starting to look a bit tight.
You’ve got about a gig of buffer/cache, which the kernel will evict if it needs it.
You’ve got 200M of genuinely free memory.
What is potentially a little concerning is the gig of swap in use, that may or may not be a problem depending on what else is running on the box and how it’s memory use varies over time.

I’m not intimatley familiar with the internals of uacctd’s mysql plugins, but my advice would be:

- Monitor the memory usage. If it doesn’t vary much over time you could go on for years and be just fine. Keep a close watch on swap usage, if that varies a lot or grows over time you’ll want to do something about it.

- Read up on OOM Killer, what it does and how it behaves. If you see OOM Killer entries in your logs, it’s time to think again

- Can you put some more memory in the box? 8GB of RAM will cost you about £60. How much is your time worth ?

HTH
Dariush

Post by Alex K
Thanx for the reply.
free
total used free shared buff/cache available
Mem: 4046572 2832576 152012 784240 1061984 204248
Swap: 3906556 1086080 2820476
free
total used free shared buff/cache available
Mem: 4046572 520352 3223884 14916 302336 3286952
Swap: 3906556 485040 3421516
Seems that mysql plugins are reserving quite some memory as they list first in htop when sorted with memory.
Thanx,
Alex

Post by Alex K
Hi all,
I have a setup with uacctd monitoring traffic of several interfaces through NFLOG.
With uacctd stopped I see that the server (a relatively small device with 4 GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes up to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
Is there any tweaks I can use to put a limit on the memory usage of uacctd.
Thanx,
Alex
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists>

tldr; post the output of the free command when uacctd is running and I’ll do my best to interpret it for you :-)
Is it _really_ using it, or is it just not completely unused?
The linux kernel generally does pretty good job of keeping stuff that might be useful in memory, but getting rid of it very very quickly if the space is needed for something else.
The challenge you face is that most of the userspace tools come with a long list of caveats about what they appear to report vs what’s really happening, this is mainly due to the way the kernel shares memory between processes, and its use of as much memory as possible for various caches.
Some threads worth reading if you’re not familiar with the in’s and out’s of linux kernel memory management...
https://www.linuxatemyram.com/ <https://www.linuxatemyram.com/>
https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage <https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage>
https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/ <https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/>
HTH
Dariush
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[Alex K]

Let me change the posting style... :)

On Mon, Jun 25, 2018 at 3:51 PM, Dariush Marsh-Mossadeghi <

Post by Dariush Marsh-Mossadeghi
OK, so we’re moving from bottom-posting to top-posting
 that’ll make it
interesting for other readers ;-)
The output of free doesn’t look desperate, but it is starting to look a bit tight.
You’ve got about a gig of buffer/cache, which the kernel will evict if it needs it.
You’ve got 200M of genuinely free memory.
What is potentially a little concerning is the gig of swap in use, that
may or may not be a problem depending on what else is running on the box
and how it’s memory use varies over time.
I’m not intimatley familiar with the internals of uacctd’s mysql plugins,
- Monitor the memory usage. If it doesn’t vary much over time you could go
on for years and be just fine. Keep a close watch on swap usage, if that
varies a lot or grows over time you’ll want to do something about it.

Monitoring of last 2 weeks is showing that free mem is fluctuating form 50M
to 220 MB and swaps remains steady. Thats why I am trying to find a
solution.

Post by Dariush Marsh-Mossadeghi
- Read up on OOM Killer, what it does and how it behaves. If you see OOM
Killer entries in your logs, it’s time to think again

I am aware of that. thanx

Post by Dariush Marsh-Mossadeghi
- Can you put some more memory in the box? 8GB of RAM will cost you about
£60. How much is your time worth ?

This is not an option. What I have is 4 GB. This is not just a personal
project. There are going to be thousands of such installations...

HTH
Dariush


On 25 Jun 2018, at 12:32, Alex K <***@gmail.com> wrote:

Thanx for the reply.

The output of free is the following:

free
total used free shared buff/cache
available
Mem: 4046572 2832576 152012 784240 1061984
204248
Swap: 3906556 1086080 2820476

While is as below when stopped:

free
total used free shared buff/cache
available
Mem: 4046572 520352 3223884 14916 302336
3286952
Swap: 3906556 485040 3421516

Seems that mysql plugins are reserving quite some memory as they list first
in htop when sorted with memory.

Thanx,
Alex


On Mon, Jun 25, 2018 at 2:22 PM, Dariush Marsh-Mossadeghi <

Post by Dariush Marsh-Mossadeghi
Hi all,
I have a setup with uacctd monitoring traffic of several interfaces through NFLOG.
With uacctd stopped I see that the server (a relatively small device with
4 GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes
up to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
Is there any tweaks I can use to put a limit on the memory usage of uacctd.
Thanx,
Alex
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
tldr; post the output of the free command when uacctd is running and I’ll
do my best to interpret it for you :-)
Is it _really_ using it, or is it just not completely unused?
The linux kernel generally does pretty good job of keeping stuff that
might be useful in memory, but getting rid of it very very quickly if the
space is needed for something else.
The challenge you face is that most of the userspace tools come with a
long list of caveats about what they appear to report vs what’s really
happening, this is mainly due to the way the kernel shares memory between
processes, and its use of as much memory as possible for various caches.
Some threads worth reading if you’re not familiar with the in’s and out’s
of linux kernel memory management...
https://www.linuxatemyram.com/
https://stackoverflow.com/questions/4802481/how-to-see-top-
processes-sorted-by-actual-memory-usage
https://stackoverflow.com/questions/3784974/want-to-know-
whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/
HTH
Dariush
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Post by Dariush Marsh-Mossadeghi
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[Dariush Marsh-Mossadeghi]

This post might be inappropriate. Click to display it.

[Alex K]

Thanks Dariush. Appreciate your feedback.

I was testing several stripped down kernels by compiling and removing most
of unused modules. The gain I had was in the range of few MB.
Seems that I have to find a more elegant approach at uacctd configuration
since with current setup I am loading several mysql plugins so as to be
able to filter traffic with directions, networks and ports.
I was testing an edge case where these plugins may rise up to 210 instances
thus leading to this memory usage. Normally this will not be done so the
normal case will be using 2.5 GB of RAM instead of 3.5 GB.
Thus in normal cases seems that I am ok.

Cheers


On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi <

Post by Dariush Marsh-Mossadeghi
OK, so it’s effectively an embedded system scenario, with a fixed config
hardware, or similar
No silver bullets or one-liner fixes here :-\
You’ve a number of options to slim down your memory footprint
- Strip Debian of all the packages you don’t need, learn a lot about
kernel tuning, and tune the kernel to your needs.
- Start looking at one of the more embedded systems oriented distros,
rather than OOTB Debian. Although Debian is skinnier than most, there’s a
lot that can be done to minimise footprint. The fact that you’ve got a
64bit Debian9 running on it makes me fairly optimistic that you could run
pretty much any Debian derivative.
- Profiling your userspace software stack and see if there are any
dependencies you don’t need, strip them out. Whether you can do this will
very much depend on the modularity of the components.
All of the above lead you down the path of maintaining forks and compiling
your own packages, to a greater or lesser extent. That’s a maintenance
overhead you want to avoid unless you have to.
Having said all that, if you’re looking at deploying thousands of units,
the economics of software maintenance may stack up for you.
HTH
Dariush
Let me change the posting style... :)

Post by Dariush Marsh-Mossadeghi
OK, so we’re moving from bottom-posting to top-posting
 that’ll make it
interesting for other readers ;-)
The output of free doesn’t look desperate, but it is starting to look a bit tight.
You’ve got about a gig of buffer/cache, which the kernel will evict if it needs it.
You’ve got 200M of genuinely free memory.
What is potentially a little concerning is the gig of swap in use, that
may or may not be a problem depending on what else is running on the box
and how it’s memory use varies over time.
I’m not intimatley familiar with the internals of uacctd’s mysql plugins,
- Monitor the memory usage. If it doesn’t vary much over time you could
go on for years and be just fine. Keep a close watch on swap usage, if that
varies a lot or grows over time you’ll want to do something about it.

Monitoring of last 2 weeks is showing that free mem is fluctuating form
50M to 220 MB and swaps remains steady. Thats why I am trying to find a
solution.

Post by Dariush Marsh-Mossadeghi
- Read up on OOM Killer, what it does and how it behaves. If you see OOM
Killer entries in your logs, it’s time to think again

I am aware of that. thanx

Post by Dariush Marsh-Mossadeghi
- Can you put some more memory in the box? 8GB of RAM will cost you about
£60. How much is your time worth ?

This is not an option. What I have is 4 GB. This is not just a personal
project. There are going to be thousands of such installations
HTH
Dariush
Thanx for the reply.
free
total used free shared buff/cache
available
Mem: 4046572 2832576 152012 784240 1061984
204248
Swap: 3906556 1086080 2820476
free
total used free shared buff/cache
available
Mem: 4046572 520352 3223884 14916 302336
3286952
Swap: 3906556 485040 3421516
Seems that mysql plugins are reserving quite some memory as they list
first in htop when sorted with memory.
Thanx,
Alex

Post by Dariush Marsh-Mossadeghi
Hi all,
I have a setup with uacctd monitoring traffic of several interfaces through NFLOG.
With uacctd stopped I see that the server (a relatively small device with
4 GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes
up to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
Is there any tweaks I can use to put a limit on the memory usage of uacctd.
Thanx,
Alex
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
tldr; post the output of the free command when uacctd is running and I’ll
do my best to interpret it for you :-)
Is it _really_ using it, or is it just not completely unused?
The linux kernel generally does pretty good job of keeping stuff that
might be useful in memory, but getting rid of it very very quickly if the
space is needed for something else.
The challenge you face is that most of the userspace tools come with a
long list of caveats about what they appear to report vs what’s really
happening, this is mainly due to the way the kernel shares memory between
processes, and its use of as much memory as possible for various caches.
Some threads worth reading if you’re not familiar with the in’s and out’s
of linux kernel memory management...
https://www.linuxatemyram.com/
https://stackoverflow.com/questions/4802481/how-to-see-top-p
rocesses-sorted-by-actual-memory-usage
https://stackoverflow.com/questions/3784974/want-to-know-whe
ther-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/
HTH
Dariush

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[Rasto Rickardt]

Hello,

If there is no more RAM available, i would test sqlite3 plugin, as
sqlite3 is more suited for limited resources usage. Youl will possibly
need to change your workflow to export sqlite3 files and load it
somewhere else, but it should be a lot cheaper memory wise.

This one is swapping pretty heavily, i would suggest to set to around 10
for 4GB RAM.

sysctl vm.swappiness=10 so it will start using swap when available RAM
is around 400MB.

r.

Post by Alex K
Thanks Dariush. Appreciate your feedback.
I was testing several stripped down kernels by compiling and removing
most of unused modules. The gain I had was in the range of few MB. 
Seems that I have to find a more elegant approach at uacctd
configuration since with current setup I am loading several mysql
plugins so as to be able to filter traffic with directions, networks and
ports.
I was testing an edge case where these plugins may rise up to 210
instances thus leading to this memory usage. Normally this will not be
done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB.
Thus in normal cases seems that I am ok.
Cheers
On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi
OK, so it’s effectively an embedded system scenario, with a fixed
config hardware, or similar
No silver bullets or one-liner fixes here :-\
You’ve a number of options to slim down your memory footprint
- Strip Debian of all the packages you don’t need, learn a lot about
kernel tuning, and tune the kernel to your needs.
- Start looking at one of the more embedded systems oriented
distros, rather than OOTB Debian. Although Debian is skinnier than
most, there’s a lot that can be done to minimise footprint. The fact
that you’ve got a 64bit Debian9 running on it makes me fairly
optimistic that you could run pretty much any Debian derivative.
- Profiling your userspace software stack and see if there are any
dependencies you don’t need, strip them out. Whether you can do this
will very much depend on the modularity of the components.
All of the above lead you down the path of maintaining forks and
compiling your own packages, to a greater or lesser extent. That’s a
maintenance overhead you want to avoid unless you have to.
Having said all that, if you’re looking at deploying thousands of
units, the economics of software maintenance may stack up for you.
HTH
Dariush 

Post by Alex K
Let me change the posting style... :)
On Mon, Jun 25, 2018 at 3:51 PM, Dariush
OK, so we’re moving from bottom-posting to top-posting…
that’ll make it interesting for other readers ;-)
The output of free doesn’t look desperate, but it is starting
to look a bit tight.
You’ve got about a gig of buffer/cache, which the kernel will
evict if it needs it.
You’ve got 200M of genuinely free memory.
What is potentially a little concerning is the gig of swap in
use, that may or may not be a problem depending on what else
is running on the box and how it’s memory use varies over time.
I’m not intimatley familiar with the internals of uacctd’s
- Monitor the memory usage. If it doesn’t vary much over time
you could go on for years and be just fine. Keep a close watch
on swap usage, if that varies a lot or grows over time you’ll
want to do something about it.
Monitoring of last 2 weeks is showing that free mem is fluctuating
form 50M to 220 MB and swaps remains steady. Thats why I am trying
to find a solution.
- Read up on OOM Killer, what it does and how it behaves. If
you see OOM Killer entries in your logs, it’s time to think again 
I am aware of that. thanx
- Can you put some more memory in the box? 8GB of RAM will
cost you about £60. How much is your time worth ?
This is not an option. What I have is 4 GB. This is not just a
personal project. There are going to be thousands of such
installations
HTH
Dariush

Thanx for the reply. 
The output of free is the following: 
free
            total        used        free        shared    
buff/cache   available
Mem:        4046572     2832576      152012      784240    
1061984      204248
Swap:       3906556     1086080     2820476
While is as below when stopped: 
free
              total        used        free      shared 
buff/cache   available
Mem:        4046572      520352     3223884       14916     
302336     3286952
Swap:       3906556      485040     3421516
Seems that mysql plugins are reserving quite some memory as they
list first in htop when sorted with memory. 
Thanx, 
Alex
On Mon, Jun 25, 2018 at 2:22 PM, Dariush

Hi all, 
I have a setup with uacctd monitoring traffic of several
interfaces through NFLOG. 
With uacctd stopped I see that the server (a relatively
small device with 4 GB of RAM) consumes 450MB of RAM. Once I
start uacctd the mem usage goes up to 3.5 GB. I am using
mysql plugin and this is running on Debian9 64 bit.
Is there any tweaks I can use to put a limit on the memory
usage of uacctd. 
Thanx, 
Alex
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>

tldr; post the output of the free command when uacctd is
running and I’ll do my best to interpret it for you :-)
Is it _really_ using it, or is it just not completely unused?
The linux kernel generally does pretty good job of keeping
stuff that might be useful in memory, but getting rid of it
very very quickly if the space is needed for something else.
The challenge you face is that most of the userspace tools
come with a long list of caveats about what they appear to
report vs what’s really happening, this is mainly due to the
way the kernel shares memory between processes, and its use
of as much memory as possible for various caches. 
Some threads worth reading if you’re not familiar with the
in’s and out’s of linux kernel memory management...
https://www.linuxatemyram.com/
https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage
<https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage>
https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/
<https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/>
HTH
Dariush

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

_______________________________________________
pmacct-discussion mailing

[Alex K]

Post by Rasto Rickardt
Hello,
If there is no more RAM available, i would test sqlite3 plugin, as
sqlite3 is more suited for limited resources usage. Youl will possibly
need to change your workflow to export sqlite3 files and load it
somewhere else, but it should be a lot cheaper memory wise.
Thanx. Seems that I have to stick with mysql at least at this phase.

This one is swapping pretty heavily, i would suggest to set to around 10

Post by Rasto Rickardt
for 4GB RAM.
sysctl vm.swappiness=10 so it will start using swap when available RAM
is around 400MB.

This is already done.

Post by Rasto Rickardt
r.

Post by Alex K
Thanks Dariush. Appreciate your feedback.
I was testing several stripped down kernels by compiling and removing
most of unused modules. The gain I had was in the range of few MB.
Seems that I have to find a more elegant approach at uacctd
configuration since with current setup I am loading several mysql
plugins so as to be able to filter traffic with directions, networks and
ports.
I was testing an edge case where these plugins may rise up to 210
instances thus leading to this memory usage. Normally this will not be
done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB.
Thus in normal cases seems that I am ok.
Cheers
On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi
OK, so it’s effectively an embedded system scenario, with a fixed
config hardware, or similar
No silver bullets or one-liner fixes here :-\
You’ve a number of options to slim down your memory footprint
- Strip Debian of all the packages you don’t need, learn a lot about
kernel tuning, and tune the kernel to your needs.
- Start looking at one of the more embedded systems oriented
distros, rather than OOTB Debian. Although Debian is skinnier than
most, there’s a lot that can be done to minimise footprint. The fact
that you’ve got a 64bit Debian9 running on it makes me fairly
optimistic that you could run pretty much any Debian derivative.
- Profiling your userspace software stack and see if there are any
dependencies you don’t need, strip them out. Whether you can do this
will very much depend on the modularity of the components.
All of the above lead you down the path of maintaining forks and
compiling your own packages, to a greater or lesser extent. That’s a
maintenance overhead you want to avoid unless you have to.
Having said all that, if you’re looking at deploying thousands of
units, the economics of software maintenance may stack up for you.
HTH
Dariush

Post by Alex K
Let me change the posting style... :)
On Mon, Jun 25, 2018 at 3:51 PM, Dariush
OK, so we’re moving from bottom-posting to top-posting

that’ll make it interesting for other readers ;-)
The output of free doesn’t look desperate, but it is starting
to look a bit tight.
You’ve got about a gig of buffer/cache, which the kernel will
evict if it needs it.
You’ve got 200M of genuinely free memory.
What is potentially a little concerning is the gig of swap in
use, that may or may not be a problem depending on what else
is running on the box and how it’s memory use varies over time.
I’m not intimatley familiar with the internals of uacctd’s
- Monitor the memory usage. If it doesn’t vary much over time
you could go on for years and be just fine. Keep a close watch
on swap usage, if that varies a lot or grows over time you’ll
want to do something about it.
Monitoring of last 2 weeks is showing that free mem is fluctuating
form 50M to 220 MB and swaps remains steady. Thats why I am trying
to find a solution.
- Read up on OOM Killer, what it does and how it behaves. If
you see OOM Killer entries in your logs, it’s time to think

again

Post by Alex K

Post by Alex K
I am aware of that. thanx
- Can you put some more memory in the box? 8GB of RAM will
cost you about £60. How much is your time worth ?
This is not an option. What I have is 4 GB. This is not just a
personal project. There are going to be thousands of such
installations
HTH
Dariush

Post by Alex K
Thanx for the reply.
free
total used free shared
buff/cache available
Mem: 4046572 2832576 152012 784240
1061984 204248
Swap: 3906556 1086080 2820476
free
total used free shared
buff/cache available
Mem: 4046572 520352 3223884 14916
302336 3286952
Swap: 3906556 485040 3421516
Seems that mysql plugins are reserving quite some memory as they
list first in htop when sorted with memory.
Thanx,
Alex
On Mon, Jun 25, 2018 at 2:22 PM, Dariush

Post by Alex K
Hi all,
I have a setup with uacctd monitoring traffic of several
interfaces through NFLOG.
With uacctd stopped I see that the server (a relatively
small device with 4 GB of RAM) consumes 450MB of RAM. Once I
start uacctd the mem usage goes up to 3.5 GB. I am using
mysql plugin and this is running on Debian9 64 bit.
Is there any tweaks I can use to put a limit on the memory
usage of uacctd.
Thanx,
Alex
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>

tldr; post the output of the free command when uacctd is
running and I’ll do my best to interpret it for you :-)
Is it _really_ using it, or is it just not completely unused?
The linux kernel generally does pretty good job of keeping
stuff that might be useful in memory, but getting rid of it
very very quickly if the space is needed for something else.
The challenge you face is that most of the userspace tools
come with a long list of caveats about what they appear to
report vs what’s really happening, this is mainly due to the
way the kernel shares memory between processes, and its use
of as much memory as possible for various caches.
Some threads worth reading if you’re not familiar with the
in’s and out’s of linux kernel memory management...
https://www.linuxatemyram.com/
https://stackoverflow.com/questions/4802481/how-to-see-

top-processes-sorted-by-actual-memory-usage

Post by Alex K

Post by Alex K

Post by Alex K
<https://stackoverflow.com/questions/4802481/how-to-see-

top-processes-sorted-by-actual-memory-usage>

Post by Alex K

Post by Alex K

Post by Alex K
https://stackoverflow.com/questions/3784974/want-to-

know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/

Post by Alex K

Post by Alex K

Post by Alex K
<https://stackoverflow.com/questions/3784974/want-to-

know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/

Post by Alex K

Post by Alex K

Post by Alex K
HTH
Dariush

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[Dariush Marsh-Mossadeghi]

Rasto's comments got me thinking


Not being privy to your application and it’s architecture this may not work for you at all.

I’ve had success in the past using rabbitmq to offload flow logs.
We used it to deliver netflow to ELK stacks running on AWS, and it worked really well.
It has the major advantage of operational resilience.

Something like this:

netflow sources -> nfacctd -> rabbitmq -> big long unreliable network path across public internet -> rabbitmq -> logstash -> elasticsearch

as ever YMMV :-)
Dariush

Post by Rasto Rickardt
Hello,
If there is no more RAM available, i would test sqlite3 plugin, as
sqlite3 is more suited for limited resources usage. Youl will possibly
need to change your workflow to export sqlite3 files and load it
somewhere else, but it should be a lot cheaper memory wise.
Thanx. Seems that I have to stick with mysql at least at this phase.
This one is swapping pretty heavily, i would suggest to set to around 10
for 4GB RAM.
sysctl vm.swappiness=10 so it will start using swap when available RAM
is around 400MB.
This is already done.
r.

Post by Alex K
Thanks Dariush. Appreciate your feedback.
I was testing several stripped down kernels by compiling and removing
most of unused modules. The gain I had was in the range of few MB.
Seems that I have to find a more elegant approach at uacctd
configuration since with current setup I am loading several mysql
plugins so as to be able to filter traffic with directions, networks and
ports.
I was testing an edge case where these plugins may rise up to 210
instances thus leading to this memory usage. Normally this will not be
done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB.
Thus in normal cases seems that I am ok.
Cheers
On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi
OK, so it’s effectively an embedded system scenario, with a fixed
config hardware, or similar
No silver bullets or one-liner fixes here :-\
You’ve a number of options to slim down your memory footprint
- Strip Debian of all the packages you don’t need, learn a lot about
kernel tuning, and tune the kernel to your needs.
- Start looking at one of the more embedded systems oriented
distros, rather than OOTB Debian. Although Debian is skinnier than
most, there’s a lot that can be done to minimise footprint. The fact
that you’ve got a 64bit Debian9 running on it makes me fairly
optimistic that you could run pretty much any Debian derivative.
- Profiling your userspace software stack and see if there are any
dependencies you don’t need, strip them out. Whether you can do this
will very much depend on the modularity of the components.
All of the above lead you down the path of maintaining forks and
compiling your own packages, to a greater or lesser extent. That’s a
maintenance overhead you want to avoid unless you have to.
Having said all that, if you’re looking at deploying thousands of
units, the economics of software maintenance may stack up for you.
HTH
Dariush

Post by Alex K
Let me change the posting style... :)
On Mon, Jun 25, 2018 at 3:51 PM, Dariush
OK, so we’re moving from bottom-posting to top-posting

that’ll make it interesting for other readers ;-)
The output of free doesn’t look desperate, but it is starting
to look a bit tight.
You’ve got about a gig of buffer/cache, which the kernel will
evict if it needs it.
You’ve got 200M of genuinely free memory.
What is potentially a little concerning is the gig of swap in
use, that may or may not be a problem depending on what else
is running on the box and how it’s memory use varies over time.
I’m not intimatley familiar with the internals of uacctd’s
- Monitor the memory usage. If it doesn’t vary much over time
you could go on for years and be just fine. Keep a close watch
on swap usage, if that varies a lot or grows over time you’ll
want to do something about it.
Monitoring of last 2 weeks is showing that free mem is fluctuating
form 50M to 220 MB and swaps remains steady. Thats why I am trying
to find a solution.
- Read up on OOM Killer, what it does and how it behaves. If
you see OOM Killer entries in your logs, it’s time to think again
I am aware of that. thanx
- Can you put some more memory in the box? 8GB of RAM will
cost you about £60. How much is your time worth ?
This is not an option. What I have is 4 GB. This is not just a
personal project. There are going to be thousands of such
installations
HTH
Dariush

Post by Alex K
Thanx for the reply.
free
total used free shared
buff/cache available
Mem: 4046572 2832576 152012 784240
1061984 204248
Swap: 3906556 1086080 2820476
free
total used free shared
buff/cache available
Mem: 4046572 520352 3223884 14916
302336 3286952
Swap: 3906556 485040 3421516
Seems that mysql plugins are reserving quite some memory as they
list first in htop when sorted with memory.
Thanx,
Alex
On Mon, Jun 25, 2018 at 2:22 PM, Dariush

Post by Alex K
Hi all,
I have a setup with uacctd monitoring traffic of several
interfaces through NFLOG.
With uacctd stopped I see that the server (a relatively
small device with 4 GB of RAM) consumes 450MB of RAM. Once I
start uacctd the mem usage goes up to 3.5 GB. I am using
mysql plugin and this is running on Debian9 64 bit.
Is there any tweaks I can use to put a limit on the memory
usage of uacctd.
Thanx,
Alex
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists>
<http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists>>

tldr; post the output of the free command when uacctd is
running and I’ll do my best to interpret it for you :-)
Is it _really_ using it, or is it just not completely unused?
The linux kernel generally does pretty good job of keeping
stuff that might be useful in memory, but getting rid of it
very very quickly if the space is needed for something else.
The challenge you face is that most of the userspace tools
come with a long list of caveats about what they appear to
report vs what’s really happening, this is mainly due to the
way the kernel shares memory between processes, and its use
of as much memory as possible for various caches.
Some threads worth reading if you’re not familiar with the
in’s and out’s of linux kernel memory management...
https://www.linuxatemyram.com/ <https://www.linuxatemyram.com/>
https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage <https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage>
<https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage <https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage>>
https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/ <https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/>
<https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/ <https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/>>
HTH
Dariush

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists>
<http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists>>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists>

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists>

[Alex K]

I have not experience with RabbitMQ but I've seen this guy around on high
volume data transfer scenarios.
A message queuing approach sounds more resilient to me but this will need
consideration at a later point. Thanx though for the thoughts.
What I currently do is: iptables -> uacctd -> mysql -> processing of mysql
and further aggregation -> json file -> push to central hub (rsync) ->
import to DB -> visualize... (I've been playing like this for at least 3
years without issues)
It is interesting to hear how you tackle such issues.

Alex


On Mon, Jun 25, 2018 at 7:51 PM, Dariush Marsh-Mossadeghi <

Post by Dariush Marsh-Mossadeghi
Rasto's comments got me thinking

Not being privy to your application and it’s architecture this may not
work for you at all.
I’ve had success in the past using rabbitmq to offload flow logs.
We used it to deliver netflow to ELK stacks running on AWS, and it worked really well.
It has the major advantage of operational resilience.
netflow sources -> nfacctd -> rabbitmq -> big long unreliable network path
across public internet -> rabbitmq -> logstash -> elasticsearch
as ever YMMV :-)
Dariush

Post by Rasto Rickardt
Hello,
If there is no more RAM available, i would test sqlite3 plugin, as
sqlite3 is more suited for limited resources usage. Youl will possibly
need to change your workflow to export sqlite3 files and load it
somewhere else, but it should be a lot cheaper memory wise.
Thanx. Seems that I have to stick with mysql at least at this phase.

This one is swapping pretty heavily, i would suggest to set to around 10

Post by Rasto Rickardt
for 4GB RAM.
sysctl vm.swappiness=10 so it will start using swap when available RAM
is around 400MB.

This is already done.

Post by Rasto Rickardt
r.

Post by Alex K
Thanks Dariush. Appreciate your feedback.
I was testing several stripped down kernels by compiling and removing
most of unused modules. The gain I had was in the range of few MB.
Seems that I have to find a more elegant approach at uacctd
configuration since with current setup I am loading several mysql
plugins so as to be able to filter traffic with directions, networks and
ports.
I was testing an edge case where these plugins may rise up to 210
instances thus leading to this memory usage. Normally this will not be
done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB.
Thus in normal cases seems that I am ok.
Cheers
On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi
OK, so it’s effectively an embedded system scenario, with a fixed
config hardware, or similar
No silver bullets or one-liner fixes here :-\
You’ve a number of options to slim down your memory footprint
- Strip Debian of all the packages you don’t need, learn a lot about
kernel tuning, and tune the kernel to your needs.
- Start looking at one of the more embedded systems oriented
distros, rather than OOTB Debian. Although Debian is skinnier than
most, there’s a lot that can be done to minimise footprint. The fact
that you’ve got a 64bit Debian9 running on it makes me fairly
optimistic that you could run pretty much any Debian derivative.
- Profiling your userspace software stack and see if there are any
dependencies you don’t need, strip them out. Whether you can do this
will very much depend on the modularity of the components.
All of the above lead you down the path of maintaining forks and
compiling your own packages, to a greater or lesser extent. That’s a
maintenance overhead you want to avoid unless you have to.
Having said all that, if you’re looking at deploying thousands of
units, the economics of software maintenance may stack up for you.
HTH
Dariush

Post by Alex K
Let me change the posting style... :)
On Mon, Jun 25, 2018 at 3:51 PM, Dariush
OK, so we’re moving from bottom-posting to top-posting

that’ll make it interesting for other readers ;-)
The output of free doesn’t look desperate, but it is starting
to look a bit tight.
You’ve got about a gig of buffer/cache, which the kernel will
evict if it needs it.
You’ve got 200M of genuinely free memory.
What is potentially a little concerning is the gig of swap in
use, that may or may not be a problem depending on what else
is running on the box and how it’s memory use varies over time.
I’m not intimatley familiar with the internals of uacctd’s
- Monitor the memory usage. If it doesn’t vary much over time
you could go on for years and be just fine. Keep a close watch
on swap usage, if that varies a lot or grows over time you’ll
want to do something about it.
Monitoring of last 2 weeks is showing that free mem is fluctuating
form 50M to 220 MB and swaps remains steady. Thats why I am trying
to find a solution.
- Read up on OOM Killer, what it does and how it behaves. If
you see OOM Killer entries in your logs, it’s time to think

again

Post by Alex K

Post by Alex K
I am aware of that. thanx
- Can you put some more memory in the box? 8GB of RAM will
cost you about £60. How much is your time worth ?
This is not an option. What I have is 4 GB. This is not just a
personal project. There are going to be thousands of such
installations
HTH
Dariush

Post by Alex K
Thanx for the reply.
free
total used free shared
buff/cache available
Mem: 4046572 2832576 152012 784240
1061984 204248
Swap: 3906556 1086080 2820476
free
total used free shared
buff/cache available
Mem: 4046572 520352 3223884 14916
302336 3286952
Swap: 3906556 485040 3421516
Seems that mysql plugins are reserving quite some memory as they
list first in htop when sorted with memory.
Thanx,
Alex
On Mon, Jun 25, 2018 at 2:22 PM, Dariush

Post by Alex K
Hi all,
I have a setup with uacctd monitoring traffic of several
interfaces through NFLOG.
With uacctd stopped I see that the server (a relatively
small device with 4 GB of RAM) consumes 450MB of RAM. Once I
start uacctd the mem usage goes up to 3.5 GB. I am using
mysql plugin and this is running on Debian9 64 bit.
Is there any tweaks I can use to put a limit on the memory
usage of uacctd.
Thanx,
Alex
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>

tldr; post the output of the free command when uacctd is
running and I’ll do my best to interpret it for you :-)
Is it _really_ using it, or is it just not completely unused?
The linux kernel generally does pretty good job of keeping
stuff that might be useful in memory, but getting rid of it
very very quickly if the space is needed for something else.
The challenge you face is that most of the userspace tools
come with a long list of caveats about what they appear to
report vs what’s really happening, this is mainly due to the
way the kernel shares memory between processes, and its use
of as much memory as possible for various caches.
Some threads worth reading if you’re not familiar with the
in’s and out’s of linux kernel memory management...
https://www.linuxatemyram.com/
https://stackoverflow.com/questions/4802481/how-to-see-top-

processes-sorted-by-actual-memory-usage

Post by Alex K

Post by Alex K

Post by Alex K
<https://stackoverflow.com/questions/4802481/how-to-see-top

-processes-sorted-by-actual-memory-usage>

Post by Alex K

Post by Alex K

Post by Alex K
https://stackoverflow.com/questions/3784974/want-to-know-

whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/

Post by Alex K

Post by Alex K

Post by Alex K
<https://stackoverflow.com/questions/3784974/want-to-know-

whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/>

Post by Alex K

Post by Alex K

Post by Alex K
HTH
Dariush

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists