From jay at splitstreams.com Tue Jul 28 12:15:07 2009 From: jay at splitstreams.com (Jay Deiman) Date: Tue, 28 Jul 2009 11:15:07 -0500 Subject: [Pymilter] memory leak in 0.9.2? Message-ID: <4A6F240B.3080905@splitstreams.com> Hello again, I've put together a milter using the python-milter 0.9.2 lib to do uri and ip BL checks on message bodies. I'm very happy with everything at this point in terms of performance and functionality, with one exception. I'm in the final stages of testing at this point and tested the milter on our most heavily loaded mail server (FreeBSD 7.0 running postfix 2.6.1) and I noticed after running it for close to 24 hours that my memory usage creeped all the way up to 144MB of usage. I'm wondering, before I start the painstaking process of tracking down memory leaks if there is a known leak that is perhaps fixed for 0.9.3? Thank you, Jay Deiman -- Jay Deiman \033:wq! From stuart at bmsi.com Tue Jul 28 13:04:05 2009 From: stuart at bmsi.com (Stuart D. Gathman) Date: Tue, 28 Jul 2009 13:04:05 -0400 (EDT) Subject: [Pymilter] memory leak in 0.9.2? In-Reply-To: <4A6F240B.3080905@splitstreams.com> References: <4A6F240B.3080905@splitstreams.com> Message-ID: On Tue, 28 Jul 2009, Jay Deiman wrote: > all the way up to 144MB of usage. I'm wondering, before I start the > painstaking process of tracking down memory leaks if there is a known leak > that is perhaps fixed for 0.9.3? The only thing fixed so far in miltermodule.c in 0.9.3 is an unitialized optional parameter (crash if you don't pass it). I guess the first question to investigate is if the leak is in the Python API (missing a Py_DECREF somewhere) or the libmilter API. I wonder if there is a way to tell how much of the 144M is allocated as: 1) python objects 2) malloc/free via libmilter 3) threads To see if we're on the same page: What version of python? (I'm on 2.4). What version of sendmail is libmilter.a from? (I'm on 8.14.3) I think it might be a good general diagnostic feature to count milter_context create/dealloc in miltermodule.c, and provide a way to query from python. Any other diagnostic counts that might be handy? -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Tue Jul 28 13:13:05 2009 From: stuart at bmsi.com (Stuart D. Gathman) Date: Tue, 28 Jul 2009 13:13:05 -0400 (EDT) Subject: [Pymilter] memory leak in 0.9.2? In-Reply-To: References: <4A6F240B.3080905@splitstreams.com> Message-ID: On Tue, 28 Jul 2009, Stuart D. Gathman wrote: > I guess the first question to investigate is if the leak is in the > Python API (missing a Py_DECREF somewhere) or the libmilter API. I wonder if > there is a way to tell how much of the 144M is allocated as: > > 1) python objects > 2) malloc/free via libmilter > 3) threads There is also the possibility that the 144M is in a global datastructure allocated by python code. For instance, I keep ip sets, CBV result maps, and other global datastructures which use quite a bit of memory. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Tue Jul 28 17:32:31 2009 From: stuart at bmsi.com (Stuart D. Gathman) Date: Tue, 28 Jul 2009 17:32:31 -0400 (EDT) Subject: [Pymilter] memory leak in 0.9.2? In-Reply-To: References: <4A6F240B.3080905@splitstreams.com> <4A6F5245.8060507@splitstreams.com> <4A6F5FD3.7030903@splitstreams.com> Message-ID: On Tue, 28 Jul 2009, Stuart D. Gathman wrote: > I guess I'm not likely to reproduce it here then. I added a getdiag() > method to the module. It returns a tuple. Currently, the first two > elements are counts of contextNew and contextDel. The counts are > incremented and returned only with GIL locked (PyEval_AcquireThread). > I'll test it, and then commit to CVS. Tested and commited to CVS. When running synchronously, del count is always one less than new count. When running asynchronously, they get farther apart as more threads are active. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From jay at splitstreams.com Wed Jul 29 12:09:07 2009 From: jay at splitstreams.com (Jay Deiman) Date: Wed, 29 Jul 2009 11:09:07 -0500 Subject: [Pymilter] memory leak in 0.9.2? In-Reply-To: References: <4A6F240B.3080905@splitstreams.com> <4A6F5245.8060507@splitstreams.com> <4A6F5FD3.7030903@splitstreams.com> <4A6F6F7C.30606@splitstreams.com> Message-ID: <4A707423.9050000@splitstreams.com> Stuart D. Gathman wrote: > On Tue, 28 Jul 2009, Jay Deiman wrote: > >> Yes, it is the x86_64 build. > > It could be a 64-bit issue. I will try to find time to compile and run > on linux x86_64 to see if it fails. > Well, this morning I got a machine updated to FreeBSD 7.2 (which has 8.14.3) and everything seems to be running smooth with that. I'm just going to chalk this up to an 8.14.2 issue and move on. Thanks for all the help with this, Jay -- Jay Deiman \033:wq! From stuart at bmsi.com Wed Jul 29 12:27:41 2009 From: stuart at bmsi.com (Stuart D. Gathman) Date: Wed, 29 Jul 2009 12:27:41 -0400 (EDT) Subject: [Pymilter] memory leak in libmilter-8.14.2 Message-ID: I have committed changes to CVS for pymilter to add milter.getversion() to retrieve the runtime libmilter version, and to disable Milter.Base.negotiate() when the libmilter runtime version is less than (1,0,1). Sendmail bumped it from (1,0,0) to (1,0,1) due to this bug. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.