Discussion:
kernel: Out of Memory:Killed process xxxxx (clamd).
(too old to reply)
Meni Shapiro
2004-09-10 18:01:48 UTC
Permalink
hi guys,
I got clamd running on a rh9 machine with mimedefang & sendmail 8.12.8 (yes
i know...should upgrade to 8.13.x ....)
i got a problem that sometimes apears every 30 minutes!! and sometimes after
few days!!(up to 2 weeks!)
i get this line in /var/log/messages:
"Sep 10 19:16:46 xxxx kernel: Out of Memory: Killed process xxxxx (clamd)."

this is just one example...BUT that happens all the time.
I feagure it is a memory problem (dough?!?) BUT i question is:
How do i force mimedefang/sendmail to process the mails and NOT queue them?
This is a large organisation and mail can't be queue too much?
(last time it queued 7000 messages over the weekend before i noticed and
restarted the clamd manualy)



Sincerely,

Meni Shapiro
***@pcb.co.il



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Nigel Horne
2004-09-10 17:47:55 UTC
Permalink
Post by Meni Shapiro
hi guys,
I got clamd running on a rh9 machine with mimedefang & sendmail 8.12.8 (yes
i know...should upgrade to 8.13.x ....)
i got a problem that sometimes apears every 30 minutes!! and sometimes after
few days!!(up to 2 weeks!)
"Sep 10 19:16:46 xxxx kernel: Out of Memory: Killed process xxxxx (clamd)."
this is just one example...BUT that happens all the time.
How do i force mimedefang/sendmail to process the mails and NOT queue them?
Please ask on the mimedefang mailing list, not here.


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Fajar A. Nugraha
2004-09-14 03:36:00 UTC
Permalink
Post by Nigel Horne
Post by Meni Shapiro
hi guys,
I got clamd running on a rh9 machine with mimedefang & sendmail 8.12.8 (yes
i know...should upgrade to 8.13.x ....)
[snip]
Post by Nigel Horne
Post by Meni Shapiro
"Sep 10 19:16:46 xxxx kernel: Out of Memory: Killed process xxxxx (clamd)."
[snip]
Post by Nigel Horne
Post by Meni Shapiro
How do i force mimedefang/sendmail to process the mails and NOT queue them?
Please ask on the mimedefang mailing list, not here.
You should really look what CAUSED your system to be out of memory.
Getting mail to pass thru on scanner errors is really just a workaround,
and that too may fail
if the problem is "out of memory" or "out of disk space".

If you use clamd as promary scanner, like I do, and discover that clamd
used up over 3 GB of memory, like I did,
then the only solution is to put memory limitation on clamd.

daemontools (http://cr.yp.to/daemontools.html) has a series of programs,
including
softlimit, which limits a program's memory usage, and service, which can
automatically starts
a daemon if it gets killed.

I use the combination of service + softlimit, which ensures clamd is
running and only using certain
amount of memory (64 M seems like a good choice here), and clamdwatch.pl
to make sure clamd
is actually running (i.e not hung).

If you have another primary scanner (clamd is your backup), then you
should stick to it for now.
Clamd works great for lots of people, but some have reported memory
leaks on latest stable (0.75.1),
which could cause your system to be "out of memory".

Regards,

Fajar


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Meni Shapiro
2004-09-14 05:30:29 UTC
Permalink
Hi Fajar,
Thanks for you answer. It's the most usefull i got 'till today.
I will take a look at the tools you suguested...
The leak is from clamd...i checked 'top' and saw how it swallows all
avialble memory until it is killed by kernel.
It is indeed the primary AV filter, and i thought of upgrading to 0.75
but since you said it suffers from the same problems (memory leaking) i
will the the 'softlimit'.
Thanks again.
Sincerely,

Meni Shapiro
Post by Fajar A. Nugraha
Post by Nigel Horne
Post by Meni Shapiro
hi guys,
I got clamd running on a rh9 machine with mimedefang & sendmail 8.12.8 (yes
i know...should upgrade to 8.13.x ....)
[snip]
Post by Nigel Horne
Post by Meni Shapiro
"Sep 10 19:16:46 xxxx kernel: Out of Memory: Killed process xxxxx (clamd)."
[snip]
Post by Nigel Horne
Post by Meni Shapiro
How do i force mimedefang/sendmail to process the mails and NOT queue them?
Please ask on the mimedefang mailing list, not here.
You should really look what CAUSED your system to be out of memory.
Getting mail to pass thru on scanner errors is really just a workaround,
and that too may fail
if the problem is "out of memory" or "out of disk space".
If you use clamd as promary scanner, like I do, and discover that clamd
used up over 3 GB of memory, like I did,
then the only solution is to put memory limitation on clamd.
daemontools (http://cr.yp.to/daemontools.html) has a series of programs,
including
softlimit, which limits a program's memory usage, and service, which can
automatically starts
a daemon if it gets killed.
I use the combination of service + softlimit, which ensures clamd is
running and only using certain
amount of memory (64 M seems like a good choice here), and clamdwatch.pl
to make sure clamd
is actually running (i.e not hung).
If you have another primary scanner (clamd is your backup), then you
should stick to it for now.
Clamd works great for lots of people, but some have reported memory
leaks on latest stable (0.75.1),
which could cause your system to be "out of memory".
Regards,
Fajar
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Clamav-users mailing list
https://lists.sourceforge.net/lists/listinfo/clamav-users
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Trog
2004-09-14 07:38:57 UTC
Permalink
Post by Fajar A. Nugraha
Clamd works great for lots of people, but some have reported memory
leaks on latest stable (0.75.1),
which could cause your system to be "out of memory".
A few people (out of the thousands who run ClamAV) have reported "memory
leaks" in stable versions of clamd.

However, none of those people have submitted a report from a memory
debugging tool to show where the leak occurs on their systems, despite
being asked to by the development team. None of the development team
have seen such a leak.

Until one of the people complaining produces a useful report, nothing
can be done. It is just as likely a leak in a system library than in
clamd.

-trog
Jason Haar
2004-09-14 08:42:45 UTC
Permalink
Post by Trog
A few people (out of the thousands who run ClamAV) have reported "memory
leaks" in stable versions of clamd.
However, none of those people have submitted a report from a memory
debugging tool to show where the leak occurs on their systems, despite
being asked to by the development team. None of the development team
have seen such a leak.
I tried to help out with valgrind as you suggested - but within 10 mins it
took 1.5Gb of RAM on my workstation (I wasn't going to put it up on
production now was I? :-) and - well - I turned it off. I really don't have
the equipment to handle running 1.5Gb debugging processes...
Post by Trog
Until one of the people complaining produces a useful report, nothing
can be done. It is just as likely a leak in a system library than in
clamd.
Could be: but I've seen it on Rh8 and Fedora-Core2 - quite different Linux
systems as far as libraries/etc go.

I hope someone else can help out - there is a problem that needs solving
there.
--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Trog
2004-09-14 09:34:24 UTC
Permalink
Post by Jason Haar
Post by Trog
A few people (out of the thousands who run ClamAV) have reported "memory
leaks" in stable versions of clamd.
However, none of those people have submitted a report from a memory
debugging tool to show where the leak occurs on their systems, despite
being asked to by the development team. None of the development team
have seen such a leak.
I tried to help out with valgrind as you suggested - but within 10 mins it
took 1.5Gb of RAM on my workstation (I wasn't going to put it up on
production now was I? :-) and - well - I turned it off. I really don't have
the equipment to handle running 1.5Gb debugging processes...
On my 512MB FC-1 system:

%MEM VSZ RSS COMMAND
12.8 1566524 66012 valgrind --leak-check=yes --tool=memcheck clamd

The virtual size may get up to 1.5GB, but the resident size (which is
the actual amount of memory it is using) shouldn't. So, unless you are
running with No-Overcommit Memory settings, which certainly isn't
required on a workstation (and is of arguable use at all) it shouldn't
be an issue.

Apart from that, even running it for 10 mins may be enough to show a
problem. Did you generate a result, or just kill it?

-trog
Thomas Lamy
2004-09-14 10:07:20 UTC
Permalink
Post by Jason Haar
Post by Trog
A few people (out of the thousands who run ClamAV) have reported "memory
leaks" in stable versions of clamd.
However, none of those people have submitted a report from a memory
debugging tool to show where the leak occurs on their systems, despite
being asked to by the development team. None of the development team
have seen such a leak.
I tried to help out with valgrind as you suggested - but within 10 mins it
took 1.5Gb of RAM on my workstation (I wasn't going to put it up on
production now was I? :-) and - well - I turned it off. I really don't have
the equipment to handle running 1.5Gb debugging processes...
One of the downsides of valgrind - it doesn't free any memory at runtime
(to check for double-free()s). I've not seen tools where one can switch
this off yet (but I've only used valgrind on Linux and purify on Solaris
machines).
Post by Jason Haar
Post by Trog
Until one of the people complaining produces a useful report, nothing
can be done. It is just as likely a leak in a system library than in
clamd.
Could be: but I've seen it on Rh8 and Fedora-Core2 - quite different Linux
systems as far as libraries/etc go.
From the different posts here I bet there are library issues in BSD, as
that OS is number one when it comes to leakage complains. I don't know
current Solaris releases, but Sol7 actually was a PITA.
With Linux being my development platform, I see no runtime leaks there.
Post by Jason Haar
I hope someone else can help out - there is a problem that needs solving
there.
All of the team members are aware of that. But as trog already wrote:
Until someone comes up with either a mail that triggers the leak or some
mem debugger's output, we're stuck.

Running valgrind on a production server is a no-no, as you already observed.

For quite a while (6 weeks) I collected each and every mail on one of my
MXes. I checked them "offline" for leaks using a shell wrapper, which
checked clams memory usage between each feeded mail, but found really
nothing.
I'll start that grabber again once I upgraded disk space on another MX here.

Thomas


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Timo Schöler
2004-09-14 11:40:17 UTC
Permalink
Post by Thomas Lamy
Post by Jason Haar
Post by Trog
A few people (out of the thousands who run ClamAV) have reported "memory
leaks" in stable versions of clamd.
However, none of those people have submitted a report from a memory
debugging tool to show where the leak occurs on their systems, despite
being asked to by the development team. None of the development team
have seen such a leak.
I tried to help out with valgrind as you suggested - but within 10 mins it
took 1.5Gb of RAM on my workstation (I wasn't going to put it up on
production now was I? :-) and - well - I turned it off. I really don't have
the equipment to handle running 1.5Gb debugging processes...
One of the downsides of valgrind - it doesn't free any memory at
runtime (to check for double-free()s). I've not seen tools where one
can switch this off yet (but I've only used valgrind on Linux and
purify on Solaris machines).
Post by Jason Haar
Post by Trog
Until one of the people complaining produces a useful report, nothing
can be done. It is just as likely a leak in a system library than in
clamd.
Could be: but I've seen it on Rh8 and Fedora-Core2 - quite different Linux
systems as far as libraries/etc go.
From the different posts here I bet there are library issues in BSD,
as that OS is number one when it comes to leakage complains. I don't
know current Solaris releases, but Sol7 actually was a PITA.
With Linux being my development platform, I see no runtime leaks there.
Post by Jason Haar
I hope someone else can help out - there is a problem that needs solving
there.
Until someone comes up with either a mail that triggers the leak or
some mem debugger's output, we're stuck.
Running valgrind on a production server is a no-no, as you already observed.
For quite a while (6 weeks) I collected each and every mail on one of
my MXes. I checked them "offline" for leaks using a shell wrapper,
which checked clams memory usage between each feeded mail, but found
really nothing.
I'll start that grabber again once I upgraded disk space on another MX here.
Thomas
hi,

running NetBSD 2.0 BETA (as of 20040909) i have not the smallest
problem (clamd w/amavisd-new, SA, dspam, and razor) regarding to
'extraordinary memory usage'. neither on a i386 MP system (uptime
approx. 30 days, 50k messages/day) nor on my private mail-gw (Sun Ultra
1, same OS).
--
mit vorzueglichster Hochachtung/best regards,

Timo Schoeler
//macfinity -- finest IT services | Triftstrasse 39 | 13353 Berlin |
Germany
Fon ++49 30 25 20 30 20 | Fax ++49 30 25 20 30 19
PGP data http://www.macfinity.net/~tis/contact/PGPPKB_timo.schoeler.txt



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Matt
2004-09-14 12:13:28 UTC
Permalink
Post by Thomas Lamy
From the different posts here I bet there are library issues in BSD, as
that OS is number one when it comes to leakage complains.
More specifically, it tends to be FreeBSD 5* systems which have the most
complaints. FreeBSD 4* systems have been rock solid with Clam upto just in
my experience.
If someone is going to run a devel status OS in a production
environment, i.e: the FreeBSD 5* range, then high memory usage is but one
of many things to be concerned about.

Matt



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Nigel Horne
2004-09-14 13:08:16 UTC
Permalink
Post by Thomas Lamy
For quite a while (6 weeks) I collected each and every mail on one of my
MXes. I checked them "offline" for leaks using a shell wrapper, which
checked clams memory usage between each feeded mail, but found really
nothing.
I'll start that grabber again once I upgraded disk space on another MX here.
That sounds like an excellent idea. Can you share the wrapper with users here
who are seeing a memory leak? If it shows a problem it will help us to track it down and plug it.
Post by Thomas Lamy
Thomas
-Nigel
--
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK. ICQ#20252325
***@despammed.com http://www.bandsman.co.uk


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Jason Haar
2004-09-15 05:14:05 UTC
Permalink
For the record I just want to say that I think "using excessive memory" is
more correct than "memory leak".

The reason I thought clamd had a memory leak was because I'd run it under
softlimits (set to say 40M) and clamd would end up (after mins,hours or
days) hung at 39xxxM. The logs would show it to be in an infinite loop -
attempting to malloc more RAM but being stopped due to the ulimit.

I then upped it to 130M - the highest I was willing to go (not even Squid in
our environment uses that much RAM!) and it would still happen. So I thought
"memory leak"

However, I loved clamd so much that I decided not to run it under
softlimits, and - three months later - the darn thing still works 100% fine
:-)


So yes - it does seem to leap up to large amounts of RAM on occassion, but
it does come back down - and it stops LOTS of viruses on our e-mail servers.

We use it under Qmail-Scanner in conjunction with two commercial AVs. ClamAV
gets called first, and if it thinks the message is CLEAN - then the other
two get a go. So far ClamAV has caught ALL viruses - the only ones that it
missed (and were caught by one of the other two) have so far all been
corrupted viruses (there have been so few I checked by hand).

ClamAV rulz. :-)
--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Fajar A. Nugraha
2004-09-14 09:30:33 UTC
Permalink
Post by Trog
Post by Fajar A. Nugraha
Clamd works great for lots of people, but some have reported memory
leaks on latest stable (0.75.1),
which could cause your system to be "out of memory".
A few people (out of the thousands who run ClamAV) have reported "memory
leaks" in stable versions of clamd.
Yes, I was one of them :)
Post by Trog
However, none of those people have submitted a report from a memory
debugging tool to show where the leak occurs on their systems, despite
being asked to by the development team. None of the development team
have seen such a leak.
True. I tried the memory detector you suggested (dmalloc and mpatrol),
it didn't work on clamd (Solaris/Sparc).
Perhaps there are some required compiling skill that I don't have ....
Post by Trog
Until one of the people complaining produces a useful report, nothing
can be done. It is just as likely a leak in a system library than in
clamd.
I agree.
Still, the problem is there.
If there is anything I can run on my system to help diagnose this leak,
I'd gladly do it.
Provided that it is "easy enough" to install (as easy as installing php,
apache, or clamav).
dmalloc and mpatrol apparently does not fall on this category, and since
valgrind only works
on Linux I can't use it on Solaris/sparc,

If you need access to a server which experience this "leak", I can
provide it.
(normal, non-root account).

Regards,

Fajar


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Nigel Horne
2004-09-14 12:27:59 UTC
Permalink
Post by Fajar A. Nugraha
and since
valgrind only works
on Linux
For the record, this is wrong. 1) Valgrind works on FreeBSD.
2) It's only x86, so it doesn't work on all Linux's.
Post by Fajar A. Nugraha
I can't use it on Solaris/sparc,
This is true.
Post by Fajar A. Nugraha
Fajar
-Nigel
--
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK. ICQ#20252325
***@despammed.com http://www.bandsman.co.uk


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
D Walsh
2004-09-14 09:34:44 UTC
Permalink
Post by Trog
Post by Fajar A. Nugraha
Clamd works great for lots of people, but some have reported memory
leaks on latest stable (0.75.1),
which could cause your system to be "out of memory".
A few people (out of the thousands who run ClamAV) have reported "memory
leaks" in stable versions of clamd.
However, none of those people have submitted a report from a memory
debugging tool to show where the leak occurs on their systems, despite
being asked to by the development team. None of the development team
have seen such a leak.
Until one of the people complaining produces a useful report, nothing
can be done. It is just as likely a leak in a system library than in
clamd.
-trog
Would you consider the following a sign of a memory leak?

ID name user cpu threads real mem virtual mem
------------------------------------------------------------------------
----------------------------
1899 freshclam clamav 0.00 1 316.00 KB 27.12 MB
1 hour later
1899 freshclam clamav 0.00 1 316.00 KB 27.12 MB
1901 clamd clamav 0.00 2 19.28 MB 40.37 MB
1 hour later
1901 clamd clamav 0.00 1 19.28 MB 39.86 MB

Other than using memory it doesn't appear to be leaking, this is after
91 hours of uptime and the values have barely changed and if it should
every crash, a full memory report will be forwarded for examination
cause the report is automatically generated by crashreporter.

I suspect if you are experiencing a memory leak it is probably System
Library related since I can watch an app to see what grows and this one
seems fairly stable in that respect.

-- Dale



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Nigel Horne
2004-09-14 12:32:45 UTC
Permalink
Post by D Walsh
Would you consider the following a sign of a memory leak?
ID name user cpu threads real mem virtual mem
------------------------------------------------------------------------
----------------------------
1899 freshclam clamav 0.00 1 316.00 KB 27.12 MB
1 hour later
1899 freshclam clamav 0.00 1 316.00 KB 27.12 MB
1901 clamd clamav 0.00 2 19.28 MB 40.37 MB
1 hour later
1901 clamd clamav 0.00 1 19.28 MB 39.86 MB
No, since internally the code could have called "free()", but that doesn't necessarily
mean that the memory is released to the operating system.
Post by D Walsh
-- Dale
-Nigel
--
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK. ICQ#20252325
***@despammed.com http://www.bandsman.co.uk


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Fajar A. Nugraha
2004-09-15 03:38:52 UTC
Permalink
Post by Nigel Horne
Post by D Walsh
Would you consider the following a sign of a memory leak?
ID name user cpu threads real mem virtual mem
------------------------------------------------------------------------
----------------------------
1899 freshclam clamav 0.00 1 316.00 KB 27.12 MB
1 hour later
1899 freshclam clamav 0.00 1 316.00 KB 27.12 MB
1901 clamd clamav 0.00 2 19.28 MB 40.37 MB
1 hour later
1901 clamd clamav 0.00 1 19.28 MB 39.86 MB
No, since internally the code could have called "free()", but that doesn't necessarily
mean that the memory is released to the operating system.
Okay. So here's the problem now.
On my system (Solaris9) "top" shows these values :

Memory: 1024M real, 169M free, 3464M swap in use, 3378M swap free

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
12406 exim 9 59 0 51M 17M sleep 18.8H 14.06% clamd

RES shows clamd uses 17M of memory, which is about right.
SIZE shows a greater value (51M), which is still normal.
I assume it's because the "free()" doesn't immediately return memory to
the OS.
The PROBLEM is that on some occations SIZE reached over 3GB, RES stays
at 17-21M,
but the memory is never returned to the OS (e.g. new processes will
complain "out of memory"
or "no stack space available"). If I kill clamd at that point, all
returns to normal.

Which brings my earlier suggestion. Is there any way to put a built-in
memory limiter
(not external program like softlimit) to clamd?

MySQL's mysqld is a good example on how memory limitation is used. It
has some
paramaters, which controls memory usage. On crashes, it shows how much
(max) memory
it can use (something like
"key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
= 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
")
And based on my experince, it never uses over that amount of memory.
MySQL's mysqld_safe script is also a good example on how it can
automatically
restarts the daemon.

The point is, I understand that clamd sometimes need a lot of memory,
and the high memory usage
on SIZE may not be leak at all, but some of us have very limited amount
of memory and swap file,
and would like some mechanism to limit clamd's memory usage.

Sure, softlimit + svc works fine. But it will be even better if clamd
has the limitation built-in.

Regards,

Fajar


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
D Walsh
2004-09-15 05:18:21 UTC
Permalink
Post by Fajar A. Nugraha
Post by Nigel Horne
Post by D Walsh
Would you consider the following a sign of a memory leak?
ID name user cpu threads real mem virtual mem
---------------------------------------------------------------------
--- ----------------------------
1899 freshclam clamav 0.00 1 316.00 KB 27.12 MB
1 hour later
1899 freshclam clamav 0.00 1 316.00 KB 27.12 MB 1901 clamd
clamav 0.00 2 19.28 MB 40.37 MB
1 hour later
1901 clamd clamav 0.00 1 19.28 MB 39.86 MB
No, since internally the code could have called "free()", but that doesn't necessarily
mean that the memory is released to the operating system.
Okay. So here's the problem now.
Memory: 1024M real, 169M free, 3464M swap in use, 3378M swap free
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
12406 exim 9 59 0 51M 17M sleep 18.8H 14.06% clamd
RES shows clamd uses 17M of memory, which is about right.
SIZE shows a greater value (51M), which is still normal.
I assume it's because the "free()" doesn't immediately return memory
to the OS.
The PROBLEM is that on some occations SIZE reached over 3GB, RES stays
at 17-21M,
but the memory is never returned to the OS (e.g. new processes will
complain "out of memory"
or "no stack space available"). If I kill clamd at that point, all
returns to normal.
Well, I must admit the following.

I sat down in front of a Solaris 9 system, installed clamav as
instructed and yes indeed there appears to be a problem with the
implementation of free(), in 30 mins of sending e-mail from the EICAR
test site memory did climb to 2.87gb and did not clear itself.

I disconnected it from the network to stop further communication in an
attempt to clear memory but it's been an hour and it has not decreased.

While I was waiting, I installed Solaris 8 on another system, ran the
same tests and while memory did hit 2.47gb, within 5 mins it returned
to 48mb.

This leads me to believe that the problem is occurring in Solaris 9
due to changes in the OS itself and perhaps clamav should consider
using a different avenue with this OS.

Aside from that, all I can tell you is I'm not impressed with this OS
and I guess I prefer the Darwin/FreeBSD environment because I'm used to
it, know what tools are available and can muck around with some
confidence without blowing things up.
Post by Fajar A. Nugraha
Which brings my earlier suggestion. Is there any way to put a built-in
memory limiter
(not external program like softlimit) to clamd?
MySQL's mysqld is a good example on how memory limitation is used. It
has some
paramaters, which controls memory usage. On crashes, it shows how much
(max) memory
it can use (something like
"key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
")
And based on my experince, it never uses over that amount of memory.
MySQL's mysqld_safe script is also a good example on how it can
automatically
restarts the daemon.
The point is, I understand that clamd sometimes need a lot of memory,
and the high memory usage
on SIZE may not be leak at all, but some of us have very limited
amount of memory and swap file,
and would like some mechanism to limit clamd's memory usage.
Sure, softlimit + svc works fine. But it will be even better if clamd
has the limitation built-in.
Regards,
Fajar
-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Clamav-users mailing list
https://lists.sourceforge.net/lists/listinfo/clamav-users
-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Fajar A. Nugraha
2004-09-15 05:48:57 UTC
Permalink
Post by D Walsh
I sat down in front of a Solaris 9 system, installed clamav as
instructed and yes indeed there appears to be a problem with the
implementation of free(), in 30 mins of sending e-mail from the EICAR
test site memory did climb to 2.87gb and did not clear itself.
[snip]
Post by D Walsh
This leads me to believe that the problem is occurring in Solaris 9
due to changes in the OS itself and perhaps clamav should consider
using a different avenue with this OS.
I use Solaris 6, 8, and 9 with ClamAV. Problem exists in all.
Post by D Walsh
Aside from that, all I can tell you is I'm not impressed with this OS
and I guess I prefer the Darwin/FreeBSD environment because I'm used
to it, know what tools are available and can muck around with some
confidence without blowing things up.
If it were up to me, I would've put Linuxes on all our Sun. However, it
is not my decision alone to make, so I'm stuck with Solaris :)

Regards,

Fajar


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Thomas Lamy
2004-09-15 06:36:30 UTC
Permalink
Post by Fajar A. Nugraha
Post by D Walsh
I sat down in front of a Solaris 9 system, installed clamav as
instructed and yes indeed there appears to be a problem with the
implementation of free(), in 30 mins of sending e-mail from the EICAR
test site memory did climb to 2.87gb and did not clear itself.
[snip]
Post by D Walsh
This leads me to believe that the problem is occurring in Solaris 9
due to changes in the OS itself and perhaps clamav should consider
using a different avenue with this OS.
I use Solaris 6, 8, and 9 with ClamAV. Problem exists in all.
Post by D Walsh
Aside from that, all I can tell you is I'm not impressed with this OS
and I guess I prefer the Darwin/FreeBSD environment because I'm used
to it, know what tools are available and can muck around with some
confidence without blowing things up.
If it were up to me, I would've put Linuxes on all our Sun. However, it
is not my decision alone to make, so I'm stuck with Solaris :)
Regards,
Fajar
I'll step done inot the cellar and reactivate my old Ultra-1 (shiver)
with solaris 6 and purify, perhaps I find something. No promises.

Thomas



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
D Walsh
2004-09-15 06:53:51 UTC
Permalink
Post by Fajar A. Nugraha
Post by D Walsh
I sat down in front of a Solaris 9 system, installed clamav as
instructed and yes indeed there appears to be a problem with the
implementation of free(), in 30 mins of sending e-mail from the EICAR
test site memory did climb to 2.87gb and did not clear itself.
[snip]
Post by D Walsh
This leads me to believe that the problem is occurring in Solaris 9
due to changes in the OS itself and perhaps clamav should consider
using a different avenue with this OS.
I use Solaris 6, 8, and 9 with ClamAV. Problem exists in all.
Well, I only tried it, not impressed with the environment yet many seem
to like it, ce la vie....
Post by Fajar A. Nugraha
Post by D Walsh
Aside from that, all I can tell you is I'm not impressed with this OS
and I guess I prefer the Darwin/FreeBSD environment because I'm used
to it, know what tools are available and can muck around with some
confidence without blowing things up.
If it were up to me, I would've put Linuxes on all our Sun. However,
it is not my decision alone to make, so I'm stuck with Solaris :)
Regards,
Fajar
-- Dale



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
David Champion
2004-09-15 08:03:16 UTC
Permalink
Post by D Walsh
I sat down in front of a Solaris 9 system, installed clamav as
instructed and yes indeed there appears to be a problem with the
implementation of free(), in 30 mins of sending e-mail from the EICAR
test site memory did climb to 2.87gb and did not clear itself.
This does not by itself indicate a problem in Solaris's free().

There are two things to learn here. First, what your profile indicates
is that clamd has allocated a lot of memory, and the arrangement of
those segments is such that it cannot all be returned to the heap yet.
This C program illustrates, somewhat.

#include <stdlib.h>
#include <unistd.h>
#define PS_ARGS "-eopid,vsz,rss,comm" /* for solaris */
/* #define PS_ARGS "auwx"*/ /* for others */
main()
{
void *big, *small; /* two references to memory */
char cmd[256]; /* a command string */
sprintf(cmd, "ps " PS_ARGS " | grep %d | grep -v grep", getpid());
system(cmd); /* show basic usage */
big = malloc(1024*1024*1024); /* allocate a big lump */
system(cmd); /* show usage */
small = malloc(1024*1024); /* allocate a small lump */
system(cmd); /* usage is larger */
free(big); /* free big first */
system(cmd); /* usage *might* not decrease */
free(small); /* free small */
system(cmd); /* usage *might* decrease */
}

Unix memory management works in a particular way. All allocated memory
is taken from a free pool in the process's address space called the
heap. There's a pointer called the break (brk) that indicates what
belongs to the process, and what belongs to the system. Anything on one
side is the process's, and anything on the other side is the system's.
If you allocate memory in segments, the break moves to make space with
each allocation. (For small allocations, the malloc manager usually
tries to anticipate larger needs; it doesn't take only small chunks
from the system.) Free()ing memory just means telling malloc that it's
available for reuse. It doesn't necessarily mean it goes back to the
system -- the break needs to be reset for that to happen. So the malloc
manager needs enough smarts to rearrange memory so that the break can be
moved without releasing memory that's in use. In the demo code above,
freeing "big" doesn't necessarily mean that the 1 GB can be returned to
the heap, because "small" is in its way. The malloc manager needs to
relocate intervening segments -- i.e., "small" -- to make that work.
Relocating means doing a lot of memory copying, so the more complex your
segment list -- the more small malloc()s you've done -- the harder this
problem gets.

Most modern mallocs can probably handle the case above -- it's not too
hard, since there's plenty of space in the memory occupied by big for
small to relocate to, easily. But if you swap the free(big) and the
free(small), this gets harder. Not all mallocs can do that, and when
you look at how much memory copying is required for this, it's not
necessarily clear that you want to, unless you really need to to satify
a demand elsewhere in the system. So malloc managers get even smarter,
trying to predict needs and such. The effect is that you don't *always*
get back what you free() when you free() it, but you might get it back
later... and you might not.

Debugging malloc()s can help, if this bothers you. They are designed
to do a lot of relocation work so that all memory free()d is returned
to the heap, so that segmentation faults are triggered whenever free
memory is used, and you can use the debugging data from that to track
the problem. But on the flip side, they typically perform terribly, so
they're not generally suitable for production work.

...

The second thing to realize is that Solaris is actually being very
smart, not dumb. Its memory management system is quite sophisticated. (I
don't mean its malloc() here, so much as the kernel memory management
system -- although the malloc() is surely designed to be cooperative
with that.) Memory that is no longer needed by a process does not
necessarily get returned to the system's free list. Instead it's used
for a variety of caching functions, memory-mapping of filesystems
and swap, a bunch of similar things. If you write a small program to
allocate 100 MB, read a 100 MB file into that memory, free it, and then
repeat, you could well find that the disk is barely touched for the
second round. Sun have white papers on this stuff of you want to read
more about it -- I couldn't do it justice here, but it's really quite
nice.

The dark side of it all is that you can't at all trust "ps" on a Solaris
system, especially on Solaris 9 and up, to tell you how much memory a
process is truly using. But the bright side is that in most cases you
don't really need to, unless you can tell from other indicators that
your system is really stressed for memory. And then you need some more
advanced tools for diagnisis of your system issues, but it's not going
to help you find leaks in clamd. I'll try to dig up references if you
want them badly.
Post by D Walsh
This leads me to believe that the problem is occurring in Solaris 9
due to changes in the OS itself and perhaps clamav should consider
using a different avenue with this OS.
Solaris 9 is much more advanced than 8 in the memory management area,
particularly with respect to the things I mentioned above. Solaris 10
even moreso.
Post by D Walsh
Aside from that, all I can tell you is I'm not impressed with this OS
and I guess I prefer the Darwin/FreeBSD environment because I'm used to
it, know what tools are available and can muck around with some
confidence without blowing things up.
Being used to it is really what it comes down to. If you study it more,
you'll find more reasons to be impressed.


All this is just to throw in with that camp that says these are no
indications of memory leaks. Now back to your regularly-scheduled virus
discussion.
--
-D. ***@uchicago.edu NSIT::ENSS


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Fajar A. Nugraha
2004-09-15 09:16:59 UTC
Permalink
David Champion wrote:
[snip]
Post by David Champion
All this is just to throw in with that camp that says these are no
indications of memory leaks. Now back to your regularly-scheduled virus
discussion.
Your comment has been the most enlightning so far on memory allocation :)
Okay, now suppose that clamd works in a "complicated" way, so that
"The effect is that you don't *always* get back what you free() when you
free()",

Do you have any suggestion as to how to get back the free()d memory?
Will (borrowing Apache's way) using a prefork-kind of daemon, with
limited lifetime
for each child, be better (in sense of memory management) than the current
thread implementation? Or perhaps limiting the lifetime of each thread
sufficient?

Regards,

Fajar


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Trog
2004-09-15 10:09:06 UTC
Permalink
Post by Fajar A. Nugraha
Do you have any suggestion as to how to get back the free()d memory?
Will (borrowing Apache's way) using a prefork-kind of daemon, with
limited lifetime
for each child, be better (in sense of memory management) than the current
thread implementation? Or perhaps limiting the lifetime of each thread
sufficient?
Apache has been moving away from the "prefork-kind of daemon" towards
threads for a number of years.

The lifetime of threads in clamd is limited by the workload. If they
don't have any work to do for a period of time, then they exit.

Have you tried the current CVS version?????? If not, do.

-trog
Graham Toal
2004-09-15 15:57:38 UTC
Permalink
Post by Trog
Post by Fajar A. Nugraha
Do you have any suggestion as to how to get back the free()d memory?
Will (borrowing Apache's way) using a prefork-kind of daemon, with
limited lifetime
for each child, be better (in sense of memory management) than the current
thread implementation? Or perhaps limiting the lifetime of each thread
sufficient?
Apache has been moving away from the "prefork-kind of daemon" towards
threads for a number of years.
The lifetime of threads in clamd is limited by the workload. If they
don't have any work to do for a period of time, then they exit.
Just as an FYI, when I wrote my smtp filter daemon, I gave some
thought to preforking, using threads, etc - but for the first implementation
I just let xinetd kick it off as a regular on-demand process.

Much to my surprise it has performed quite acceptably, at a reasonably
large and reasonably busy site (roughly 35,000 mails per day per pserver -
we're using two MX servers for business continuity, but on days when
only one has been running it has handled 70,000 mails per day just as
handily) Load average during the busy part of the day stays around .5
(This is a 2 processor box so a LA of 2 would be fully loaded, rather
than 1 on a single processor system, i.e. it's about 25% loaded.)

The smtp filter invokes uvscan, clamscan, spamc and spamprobe on
each mail. Only spamd runs as a daemon, the rest are simple processes
kicked off on demand.

So unless you have a site with more than a couple of thousand users,
I'm not at all sure that tweaks like you're talking about are worth
the programming complexity. I think the benefits are soon overtaken
by the speed increases of newer hardware.


Graham


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Fajar A. Nugraha
2004-09-16 04:01:51 UTC
Permalink
Post by Graham Toal
Just as an FYI, when I wrote my smtp filter daemon, I gave some
thought to preforking, using threads, etc - but for the first implementation
I just let xinetd kick it off as a regular on-demand process.
[snip]
Post by Graham Toal
So unless you have a site with more than a couple of thousand users,
I'm not at all sure that tweaks like you're talking about are worth
the programming complexity. I think the benefits are soon overtaken
by the speed increases of newer hardware.
Try over one million users, cause that's what I have here :)
Every little bit of saving counts.

Regards,

Fajar


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
Fajar A. Nugraha
2004-09-16 03:58:44 UTC
Permalink
Post by Trog
Apache has been moving away from the "prefork-kind of daemon" towards
threads for a number of years.
Yes, but in case you didn't notice prefork is STILL the default MPM if
no specific one
is chosen. It's for "compatibility purposes" mostly, for modules that
are not thread-safe yet.
So it's still pretty much alive.
Post by Trog
The lifetime of threads in clamd is limited by the workload. If they
don't have any work to do for a period of time, then they exit.
What if they have lots of things to do all the time? (e.g. busy
mailservers).
The 3G memory usage that I talk about happens on the busiest server.
The not-so-busy only uses hundreds of MB max.
It might be a good idea to force-limit thread lifetime to a number of scans
if it indeed helps return memory back to the OS (not sure about this one
though.
Logically it should work).
Post by Trog
Have you tried the current CVS version?????? If not, do.
I did. Upgraded daily, in fact. I build (and use) daily CVS snapshot for
many
platforms, including Solaris (available on clamav.or.id).

Most recent CVS snapshot still have this problem (e.g memory not
returned to the OS).
This clamd has been running for 7 hours, on a not-so-busy maliserver.

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
1706 exim 9 58 0 41M 8064K sleep 16:43 4.30% clamd

Regards,

Fajar


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
Trog
2004-09-17 08:05:33 UTC
Permalink
Post by Fajar A. Nugraha
Post by Trog
Apache has been moving away from the "prefork-kind of daemon" towards
threads for a number of years.
Yes, but in case you didn't notice prefork is STILL the default MPM if
no specific one
is chosen. It's for "compatibility purposes" mostly, for modules that
are not thread-safe yet.
So it's still pretty much alive.
Sure, and I still use it extensively. But, the point is that you were
using the fact that Apache uses fork()ing as an argument for not using
threads, when in fact they are moving towards threads away from
fork()ing.
Post by Fajar A. Nugraha
Post by Trog
The lifetime of threads in clamd is limited by the workload. If they
don't have any work to do for a period of time, then they exit.
What if they have lots of things to do all the time? (e.g. busy
mailservers).
Then they will keep processing work requests.
Post by Fajar A. Nugraha
The 3G memory usage that I talk about happens on the busiest server.
The not-so-busy only uses hundreds of MB max.
It might be a good idea to force-limit thread lifetime to a number of scans
if it indeed helps return memory back to the OS (not sure about this one
though.
Logically it should work).
Won't make any difference. The thread manager is completely separate
from the scanning engine. A memory leak in the scanning engine won't get
magically recovered by thread termination.

You can limit the number of concurrent threads, and hence memory by
using the MaxThreads directive. That also limits the number of
concurrent scans.
Post by Fajar A. Nugraha
Post by Trog
Have you tried the current CVS version?????? If not, do.
I did. Upgraded daily, in fact. I build (and use) daily CVS snapshot for
many
platforms, including Solaris (available on clamav.or.id).
Most recent CVS snapshot still have this problem (e.g memory not
returned to the OS).
This clamd has been running for 7 hours, on a not-so-busy maliserver.
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
1706 exim 9 58 0 41M 8064K sleep 16:43 4.30% clamd
It's using only 8M of memory. Nothing wrong with that.

-trog
Fajar A. Nugraha
2004-09-18 10:37:40 UTC
Permalink
Post by Trog
You can limit the number of concurrent threads, and hence memory by
using the MaxThreads directive. That also limits the number of
concurrent scans.
I did. MaxThreads 32
Post by Trog
Post by Fajar A. Nugraha
This clamd has been running for 7 hours, on a not-so-busy maliserver.
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
1706 exim 9 58 0 41M 8064K sleep 16:43 4.30% clamd
It's using only 8M of memory. Nothing wrong with that.
If you look at the RES it's using 8M. I believe I mentioned earlier that
the problem is not in RES,
but in SIZE. This time SIZE is 41M and RES 8M, my system still runs fine.
But SIZE can get realy, really big (over 3 GB used by clamd, on a system
with 1 GB memory and 4 GB swap),
and the system will start complaining out-of-memory when all that time
RES only shows 8-21 MB.

I suspect (might be wrong) that clamd DID call all necessary free()s,
but the OS is unable
to return the memory back to system.
IF using fork instead of threads could help return this memory back to
the OS
(again, I might be wrong about this), isn't it worth to try ?

If that indeed helps on some platform, then wouldn't providing both ways
(fork and threads)
and let users pick which one they want (something like --with-mpm=
directive in apache)
be a good thing?

Regards,

Fajar


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
David Champion
2004-09-16 04:47:17 UTC
Permalink
Post by Fajar A. Nugraha
Do you have any suggestion as to how to get back the free()d memory?
Will (borrowing Apache's way) using a prefork-kind of daemon, with
limited lifetime
for each child, be better (in sense of memory management) than the current
thread implementation? Or perhaps limiting the lifetime of each thread
sufficient?
I wouldn't say better, necessarily. Using processes instead of
lightweight processes (LWPs, or "threads") for your threads of execution
will certainly return memory to the system more often. It will also
consume more memory per thread of execution -- realize that all the
BSS and heap data for a single process will be reduplicated for each
process, and that you'll have a process for each concurrent e-mail
message being scanned. Currently a lot of that is shared once among
all concurrent messages. It will also force the kernel to spend more
time mapping and unmapping memory pages, and you won't get the same I/O
cache benefit. It might retain less memory at some sample points (though
not at others), but I can't say that it will make your system perform
better, which is the real issue, I suspect.

Remember: ps tells you that some amount of memory is mapped to the
process. It does not tell you what that memory is being used for. If
that's all used to accelerate disk I/O, that seems good, not bad.

The problem mentioned in the subject was someone else's issue, running
clamd on Red Hat 9, in conjunction MIMEdefang. I don't know much about
that combination, and can't speak to it. I'm not sure what problem
you're seeing -- but big numbers next to "clamd" in ps output isn't a
problem, per se, and putting monitors and such around clamd to make it
terminate early can hurt your overall performance. (Sorry if I've missed
details to your situation -- I don't usually get around to everything on
this list.)

Certainly, I think that in general it benefits you to use multi-threaded
(MT) code, and to retain all the memory you can in the MT process,
*without* returning it to the system. At small e-mail volumes this
doesn't matter much, but as your volume increases, this matters more,
and more rapidly. A kernel can spend a lot of time spawning processes
and remapping memory, but MT code and memory conservation are strategies
to minimize this.

If you're experiencing software failures or memory shortages in other
applications on the server, you need to look at general performance
tuning for the whole system, and maybe break services out onto multiple
systems. Performance tuning is a holistic process -- there are a lot of
interrelated things to take into account, and you might well need to
broaden your toolset. Sar will help, as will a variety of analysis tools
that work on sar data.

There are good books on the topic, too. Adrian Cockroft, Richard
McDougall, and Jim Mauro all have excellent materials on the topic,
besides having been instrumental in the Solaris VM architecture. Among
other resources, they refer their students to:
http://www.solarisinternals.com/
http://www.sun.com/blueprints/
http://www.sun.com/bigadmin/
--
-D. ***@uchicago.edu NSIT::ENSS


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
Mar Matthias Darin
2004-09-16 05:28:00 UTC
Permalink
Post by Fajar A. Nugraha
Okay, now suppose that clamd works in a "complicated" way, so that
"The effect is that you don't *always* get back what you free() when you
free()",
Do you have any suggestion as to how to get back the free()d memory?
Will (borrowing Apache's way) using a prefork-kind of daemon, with limited
lifetime
for each child, be better (in sense of memory management) than the current
thread implementation? Or perhaps limiting the lifetime of each thread
sufficient?
From experience with pthreads and Linux v2.4, pthreads was a royal pain. I
initially used threads as a method of a limited lifetime model for my
firewall design... I kept getting unusual and unpredictable segfaults. The
process would run anywhere from 2 days to several months, then for no
appearent reason,segfault in a routine that had been tested a thousand times
under high stress conditions and not failed.

I confirmed that pthreads leaks memory in the management thread when calls
were made to usleep. (nanosleep -> _nanosleep). GDB brought up the error
after to weeks of running the process in a debug state.

After moving to fork() and named pipes, the same code hasn't broken once in
nearly a year of hard testing. My tested often included 10 or more icmp
floods of at least 65535 packets. I drove my load to 240 during the test...

Now the forked process uses and frees memory thousands of time per second
with no issue...

Pthreads work well for light duty non-daemon processes... If its heavy duty
Trog
2004-09-17 08:27:40 UTC
Permalink
Post by Mar Matthias Darin
Post by Fajar A. Nugraha
Do you have any suggestion as to how to get back the free()d memory?
Will (borrowing Apache's way) using a prefork-kind of daemon, with limited
lifetime
for each child, be better (in sense of memory management) than the current
thread implementation? Or perhaps limiting the lifetime of each thread
sufficient?
From experience with pthreads and Linux v2.4, pthreads was a royal pain. I
initially used threads as a method of a limited lifetime model for my
firewall design... I kept getting unusual and unpredictable segfaults. The
process would run anywhere from 2 days to several months, then for no
appearent reason,segfault in a routine that had been tested a thousand times
under high stress conditions and not failed.
Such things are generally due to memory usage bugs in the code, they
just don't trigger very often.

My pthread'ed web proxy has been running very stable on RH 6.2 on kernel
2.2.19) for a very long time, current stats:

connections(24104670) requests(54869840) threads(4/24)

[The threads stat means there are 24 worker threads started, and 4 of
them are currently actively doing something useful at this moment in
time - in this model, all networking is non-blocking, so threads don't
wait for network I/O - this means that the churn rate for threads is
very high, while the actual number of threads remains relatively low.]
Post by Mar Matthias Darin
After moving to fork() and named pipes, the same code hasn't broken once in
nearly a year of hard testing. My tested often included 10 or more icmp
floods of at least 65535 packets. I drove my load to 240 during the test...
Now the forked process uses and frees memory thousands of time per second
with no issue...
Pthreads work well for light duty non-daemon processes... If its heavy duty
It depends how good your model and implementation are in my experience.

-trog
Ralf Hildebrandt
2004-09-15 06:31:20 UTC
Permalink
Post by Fajar A. Nugraha
Which brings my earlier suggestion. Is there any way to put a
built-in memory limiter (not external program like softlimit) to
clamd?
Why add code to clamd when a good unix-like solution already exists?
--
Ralf Hildebrandt (i.A. des IT-Zentrum) ***@charite.de
Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-916
IT-Zentrum Standort CBF AIM. ralfpostfix


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Fajar A. Nugraha
2004-09-15 07:00:34 UTC
Permalink
Post by Ralf Hildebrandt
Post by Fajar A. Nugraha
Which brings my earlier suggestion. Is there any way to put a
built-in memory limiter (not external program like softlimit) to
clamd?
Why add code to clamd when a good unix-like solution already exists?
Because softlimit is a hack. Because current clamd implementation is not
to "die" on
memory allocation error, but sleep. Because clamd is not only used on
unixen systems,
but also on Windows. Because it will be much simpler if the ability is
built-in on clamd.


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Ralf Hildebrandt
2004-09-15 07:58:41 UTC
Permalink
Post by Fajar A. Nugraha
Because softlimit is a hack.
It is not a hack. It is common pratice to run programs using least
privilege and with limited resource to prevent runaway conditions.
Post by Fajar A. Nugraha
Because current clamd implementation is not to "die" on
memory allocation error, but sleep.
It doesn't die, it's being killed by the kernel.
--
Ralf Hildebrandt (i.A. des IT-Zentrum) ***@charite.de
Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-916
IT-Zentrum Standort CBF AIM. ralfpostfix


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Jason Haar
2004-09-15 09:13:41 UTC
Permalink
Post by Ralf Hildebrandt
Post by Fajar A. Nugraha
Because current clamd implementation is not to "die" on
memory allocation error, but sleep.
It doesn't die, it's being killed by the kernel.
No - clamd does a malloc and that fails. Then instead of dying (which would
be the proper thing to do IMHO), it sleeps a few microsecs and then tries to
malloc the memory again. Infinite loop occurs...

I don't understand why it does that. I mean, if it failed to get memory
once, why expect it to get better? At least have it try only 3 times or
something - and then exit.

[people running softlimits would almost invariably also be calling clamd
under a supervise script, so if clamd died, it would be auto-restarted.
That's the condition we are trying to achieve]
--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Ralf Hildebrandt
2004-09-15 11:27:16 UTC
Permalink
Post by Jason Haar
Post by Ralf Hildebrandt
Post by Fajar A. Nugraha
Because current clamd implementation is not to "die" on
memory allocation error, but sleep.
It doesn't die, it's being killed by the kernel.
No - clamd does a malloc and that fails. Then instead of dying (which would
be the proper thing to do IMHO), it sleeps a few microsecs and then tries to
malloc the memory again. Infinite loop occurs...
Ok, THAT's bad - and should be fixed.
Post by Jason Haar
[people running softlimits would almost invariably also be calling clamd
under a supervise script, so if clamd died, it would be auto-restarted.
That's the condition we are trying to achieve]
Yep.
--
Ralf Hildebrandt (i.A. des IT-Zentrum) ***@charite.de
Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-916
IT-Zentrum Standort CBF AIM. ralfpostfix


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Trog
2004-09-15 11:55:24 UTC
Permalink
Post by Ralf Hildebrandt
Post by Jason Haar
Post by Ralf Hildebrandt
Post by Fajar A. Nugraha
Because current clamd implementation is not to "die" on
memory allocation error, but sleep.
It doesn't die, it's being killed by the kernel.
No - clamd does a malloc and that fails. Then instead of dying (which would
be the proper thing to do IMHO), it sleeps a few microsecs and then tries to
malloc the memory again. Infinite loop occurs...
Ok, THAT's bad - and should be fixed.
If it were true it would be. Please point me at some code in clamd that
does that.

-trog
Ralf Hildebrandt
2004-09-15 14:17:58 UTC
Permalink
Post by Trog
Post by Ralf Hildebrandt
Ok, THAT's bad - and should be fixed.
If it were true it would be. Please point me at some code in clamd that
does that.
That was not my claim, but the other person's.
--
Ralf Hildebrandt (i.A. des IT-Zentrum) ***@charite.de
Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-916
IT-Zentrum Standort CBF AIM. ralfpostfix


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Trog
2004-09-15 14:44:12 UTC
Permalink
Post by Ralf Hildebrandt
Post by Trog
Post by Ralf Hildebrandt
Ok, THAT's bad - and should be fixed.
If it were true it would be. Please point me at some code in clamd that
does that.
That was not my claim, but the other person's.
I know, I believe I correctly kept the attribution. You merely believed
it at face value.

-trog
Ralf Hildebrandt
2004-09-15 17:45:15 UTC
Permalink
Post by Trog
Post by Ralf Hildebrandt
That was not my claim, but the other person's.
I know, I believe I correctly kept the attribution. You merely believed
it at face value.
Fact: We've been running clamd for a week now, scanning 130.000 mails
per week. It has not died on us, nor is it using huge amounts of memory:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1509 amavis 9 0 44084 16m 15m S 0.0 1.6 0:14.31 clamd
1510 amavis 9 0 44084 16m 15m S 0.0 1.6 0:00.30 clamd
5146 amavis 9 0 44084 16m 15m S 0.0 1.6 1:51.37 clamd
10478 amavis 9 0 44084 16m 15m S 0.0 1.6 0:02.41 clamd

If it would, I'd surely report it properly.

Question: Why do I see 4 clamd processes?

/usr/local/etc/clamav.conf:
LogFile /var/log/clamd.log
LogFileMaxSize 20M
LogTime
LogSyslog
PidFile /var/run/clamd.pid
DataDirectory /var/lib/clamav
LocalSocket /var/amavis/clamd
FixStaleSocket
MaxThreads 30
MaxDirectoryRecursion 15
User amavis
ScanMail
ScanArchive
ArchiveMaxFileSize 10M
ArchiveMaxRecursion 5
ArchiveMaxFiles 1000
--
Ralf Hildebrandt (i.A. des IT-Zentrum) ***@charite.de
Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-916
IT-Zentrum Standort CBF AIM. ralfpostfix


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Fajar A. Nugraha
2004-09-16 04:07:40 UTC
Permalink
Post by Ralf Hildebrandt
Fact: We've been running clamd for a week now, scanning 130.000 mails
per week.
With that amount you shouldn't have any problem.
Try 1157851 mail per day (that's yesterday's count on one of my MTAs).
Post by Ralf Hildebrandt
Question: Why do I see 4 clamd processes?
Might be threads implementation on 2.4 kernel. I get that too on my
Aurora Linux.
On 2.6 kernel you only see one process.

Regards,

Fajar



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
Mike Lambert
2004-09-14 13:13:22 UTC
Permalink
Post by Trog
Post by Fajar A. Nugraha
Clamd works great for lots of people, but some have reported memory
leaks on latest stable (0.75.1),
which could cause your system to be "out of memory".
A few people (out of the thousands who run ClamAV) have reported "memory
leaks" in stable versions of clamd.
I am one of them.
Post by Trog
However, none of those people have submitted a report from a memory
debugging tool to show where the leak occurs on their systems, despite
being asked to by the development team.
If I were an experienced software developer, I would without hesitation.
But, I am only a humble admin.
Post by Trog
None of the development team have seen such a leak.
Sucks to be me.
Post by Trog
Until one of the people complaining produces a useful report, nothing
can be done. It is just as likely a leak in a system library than in
clamd.
Various versions of clamd have experienced random crashing and what I
call "memory leaks" on FreeBSD 4.9, the major symptom of which is
expanding memory usage (both VSZ and RSS) as reported by ps, until the
process consumes all available memory. In other words, clamd starts by
consuming about 12 MB, then, as viruses are found, memory usage grows.
For a while, I was killing and restarting clamd every few days because
the process had consumed over 250MB of memory (both VSZ and RSS). No
other process running on any FreeBSD system I manage has shown this
behavior. None. Maybe I'm just lucky.

My solution is to watch the CVS change log for "fixed memory leak", then
try that code. I have been running ClamAV devel-20040827/489/14
successfully. Memory usage has been stable for almost two weeks, and
the process has yet to crash.

PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
82524 mailnull 2 0 34936K 33636K poll 51:59 0.00% 0.00% clamd

Whatever was fixed between 20040803 and 20040824 stopped the crashing (I
would be more specific but I was on vacation), and fixes between
20040824 and 20040827 appear to have resolved the expanding memory usage
issues on my system.

For what it is worth, I offer my thanks and appreciation to the clamav
development team for their hard work on this project. I wish I could be
more helpful.

Regards,
Mike Lambert



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Nigel Horne
2004-09-14 09:23:29 UTC
Permalink
Post by Meni Shapiro
Hi Fajar,
Thanks for you answer. It's the most usefull i got 'till today.
I will take a look at the tools you suguested...
The leak is from clamd...i checked 'top' and saw how it swallows all
avialble memory until it is killed by kernel.
That is not evidence of a memory leak. It is evidence of as lot of memory
being used at runtime which is a very different thing.
Post by Meni Shapiro
Meni Shapiro
--
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK. ICQ#20252325
***@despammed.com http://www.bandsman.co.uk


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Fajar A. Nugraha
2004-09-14 10:04:06 UTC
Permalink
Post by Nigel Horne
Post by Meni Shapiro
Hi Fajar,
Thanks for you answer. It's the most usefull i got 'till today.
I will take a look at the tools you suguested...
The leak is from clamd...i checked 'top' and saw how it swallows all
avialble memory until it is killed by kernel.
That is not evidence of a memory leak. It is evidence of as lot of memory
being used at runtime which is a very different thing.
BTW, what IS the evidence of memory leak?
Would you call memory usage of 128MB leak?
Would you call clamd memory usage of 3GB leak?
Is there anumber which says "x amount of memory used by clamd is normal" ?
I know that the amount of memory used should be varied depending on
system activity,
but when clamd uses 1 or 2 GB memory when it does nothing (well, it WAS
very busy
earlier, but it's doing nothing now) is _weird_

Though I probably should rephrase my statements from now, and no longer
use the phrase "memory leak" but "high memory usage" instead.

Regards,

Fajar


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Trog
2004-09-14 10:32:53 UTC
Permalink
Post by Fajar A. Nugraha
Post by Nigel Horne
That is not evidence of a memory leak. It is evidence of as lot of memory
being used at runtime which is a very different thing.
BTW, what IS the evidence of memory leak?
There is no substantiated evidence at this point.
Post by Fajar A. Nugraha
Would you call memory usage of 128MB leak?
Would you call clamd memory usage of 3GB leak?
Neither. I would call losing reference to allocated memory a memory
leak.
Post by Fajar A. Nugraha
Is there anumber which says "x amount of memory used by clamd is normal" ?
No, it depends entirely upon your usage.
Post by Fajar A. Nugraha
I know that the amount of memory used should be varied depending on
system activity,
but when clamd uses 1 or 2 GB memory when it does nothing (well, it WAS
very busy
earlier, but it's doing nothing now) is _weird_
If you're scanning multiple 1GB files concurrently, then your going to
use 1-2GB of memory.

It's up to your systems malloc/free implementation to decide when memory
is released back to the system, not clams. So memory not going down
during inactive periods is not "_weird_", it is entirely normal
behaviour.

-trog
Fajar A. Nugraha
2004-09-14 11:07:32 UTC
Permalink
Post by Trog
Post by Fajar A. Nugraha
I know that the amount of memory used should be varied depending on
system activity,
but when clamd uses 1 or 2 GB memory when it does nothing (well, it WAS
very busy
earlier, but it's doing nothing now) is _weird_
If you're scanning multiple 1GB files concurrently, then your going to
use 1-2GB of memory.
That's just it. I put a size limit on my mail system, BEFORE clamd has a
chance to scan it,
so I know for a fact that no mail ever exceeds 50MB.
Perhaps MaxThreads 32 has something to do with the 3 GB memory usage ....
Post by Trog
It's up to your systems malloc/free implementation to decide when memory
is released back to the system, not clams. So memory not going down
during inactive periods is not "_weird_", it is entirely normal
behaviour.
Okay. So lets say that clamd IS leak-free.
Lets say that the 3 GB is, in fact, memory that clamd used but system's
"free" didn't return to the OS.

Now comes a question : what does clamd (the devel versions) do when it
cannot allocate additional memory ?
I was under the impressions that old versions simply "returns an error
and wait" instead of just died.
Which makes running daemontools alone insufficient.

If that were true, is there a plan to change that behaviour to
"die/panic on memory errors"?
Is there any plan to implement some kind of built-in memory-limiter on
clamd?

Regards,

Fajar


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Trog
2004-09-14 11:59:19 UTC
Permalink
Post by Fajar A. Nugraha
Post by Trog
If you're scanning multiple 1GB files concurrently, then your going to
use 1-2GB of memory.
That's just it. I put a size limit on my mail system, BEFORE clamd has a
chance to scan it,
so I know for a fact that no mail ever exceeds 50MB.
Perhaps MaxThreads 32 has something to do with the 3 GB memory usage ....
MaxThreads doesn't have a direct correlation to memory usage. clamd will
not start threads it doesn't need, and unused threads will exit if there
is no work for them to do.
Post by Fajar A. Nugraha
Now comes a question : what does clamd (the devel versions) do when it
cannot allocate additional memory ?
I was under the impressions that old versions simply "returns an error
and wait" instead of just died.
Which makes running daemontools alone insufficient.
It'll return an error and attempt to continue in most instances.
Post by Fajar A. Nugraha
If that were true, is there a plan to change that behaviour to
"die/panic on memory errors"?
Is there any plan to implement some kind of built-in memory-limiter on
clamd?
As Nigel has stated on more than one occassion, memory usage in the
current development tree is much more predictable than in the current
stable version.

-trog
Lutz Petersen
2004-09-14 23:43:25 UTC
Permalink
Post by Trog
Post by Fajar A. Nugraha
Would you call memory usage of 128MB leak?
Would you call clamd memory usage of 3GB leak?
Neither. I would call losing reference to allocated memory a memory
leak.
Ok, but just two things:

The clam conf is set to limit the amount of memory that should be
scanned. If there is no memory leak, otherwise it could be that
something is wrong with that code/configuration ?

But: If not running clamav as a daemon (clamscan instead of clamdscan),
the problem seems not to appear (or isn't yet seen). Every clamscan-
process terminating frees allocated memory of the process. But running
clamdscan, from time to time the clamd grows (in memory-usage). So this
_looks_ like a memory leak.

But also: I ran valgrind on one server, but the problem didn't appear
for some days. As it appeared, the hole server crashed - murphy..
So I have to do more valgrind-runs to get some useful output. Version
is last stable (.75.1), and no BSD.
Post by Trog
Post by Fajar A. Nugraha
Is there anumber which says "x amount of memory used by clamd is normal" ?
No, it depends entirely upon your usage.
Let us say, the configuration is set to maximum scan of 10 mb - what
would be (nearby) the amount of used memory to be expected on a system
where no leaks appear ?



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Nigel Horne
2004-09-14 13:04:16 UTC
Permalink
Post by Fajar A. Nugraha
Post by Nigel Horne
Post by Meni Shapiro
Hi Fajar,
Thanks for you answer. It's the most usefull i got 'till today.
I will take a look at the tools you suguested...
The leak is from clamd...i checked 'top' and saw how it swallows all
avialble memory until it is killed by kernel.
That is not evidence of a memory leak. It is evidence of as lot of memory
being used at runtime which is a very different thing.
BTW, what IS the evidence of memory leak?
A malloc with no corresponding free.
Post by Fajar A. Nugraha
Would you call memory usage of 128MB leak?
Not necessarily.
Post by Fajar A. Nugraha
Would you call clamd memory usage of 3GB leak?
Not necessaily.
Post by Fajar A. Nugraha
Is there anumber which says "x amount of memory used by clamd is normal" ?
Amount of memory used has nothing to do with memory leaks. A memory leak
can be of 1 byte. If I malloc 3Gb and then free it, no leak has occured, but on some
systems a "ps" will show that 3Gb is in use.
Post by Fajar A. Nugraha
Though I probably should rephrase my statements from now, and no longer
use the phrase "memory leak" but "high memory usage" instead.
That is different.
High memory usage != memory leak
You will notice a reduction in the development version compared with the current release.
If you want to help to test please download from http://www.clamav.net.

Note, I am not saying there is no memory leak. I am saying that I am yet to see evidence
of one or to be sent any data that produces on.
Post by Fajar A. Nugraha
Regards,
Fajar
-Nigel
--
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK. ICQ#20252325
***@despammed.com http://www.bandsman.co.uk


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Fajar A. Nugraha
2004-09-15 03:21:14 UTC
Permalink
Post by Nigel Horne
Post by Fajar A. Nugraha
Though I probably should rephrase my statements from now, and no longer
use the phrase "memory leak" but "high memory usage" instead.
That is different.
High memory usage != memory leak
Actually, that is my point. Right now I'm not sure whether there IS any
memory leak at all on clamd,
but I know that on some occations clamd uses up LOTS of memory. Hence, the
phrase "high memory usage" is more suitable.

Regards,

Fajar



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
Meni Shapiro
2004-09-20 18:06:33 UTC
Permalink
----- Original Message -----
From: "Nigel Horne" <***@bandsman.co.uk>
To: <clamav-***@lists.sourceforge.net>
Sent: Tuesday, September 14, 2004 11:23 AM
Subject: Re: [Clamav-users] kernel: Out of Memory:Killed process xxxxx
(clamd).
Post by Nigel Horne
Post by Meni Shapiro
Hi Fajar,
Thanks for you answer. It's the most usefull i got 'till today.
I will take a look at the tools you suguested...
The leak is from clamd...i checked 'top' and saw how it swallows all
avialble memory until it is killed by kernel.
That is not evidence of a memory leak. It is evidence of as lot of memory
being used at runtime which is a very different thing.
OK....
Please let me know how to findout what it is because it is killed by kernel
every 1/2h , so i put a little script in cron.hourly to restart clamd.
i run clamAV on 2 systems:
1) linux crux 1.2 kernel 2.4.26 works fine!
2) R.H.9 kernel 2.4.20-8 SUCKS!!!
any suguestions????

Sincerely,

Meni Shapiro
Post by Nigel Horne
Post by Meni Shapiro
Meni Shapiro
--
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK. ICQ#20252325
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Clamav-users mailing list
https://lists.sourceforge.net/lists/listinfo/clamav-users
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Nigel Horne
2004-09-14 09:22:20 UTC
Permalink
Post by Fajar A. Nugraha
If you have another primary scanner (clamd is your backup), then you
should stick to it for now.
Clamd works great for lots of people, but some have reported memory
leaks on latest stable (0.75.1),
which could cause your system to be "out of memory".
There have been no memory leaks reported for months now.
Let me rephrase that, there have been reports of leaks but
1) No hard evidence was ever produced
2) Everyone (and I use that word advisedly) who reported leaks in the
last 2 months has misunderstood the phrase "memory leak".

As for the issue of memory used during runtime (which is different, this
is where the confusion comes in), the next version will be better. If you want to
help to test it, you can do so by downloading the CVS or daily snapshot from
http://www.clamav.net.
Post by Fajar A. Nugraha
Regards,
Fajar
-Nigel
--
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK. ICQ#20252325
***@despammed.com http://www.bandsman.co.uk


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
Loading...