From m.mach at uptime.at Wed Dec 3 08:14:49 2008 From: m.mach at uptime.at (Michal Mach) Date: Wed, 03 Dec 2008 14:14:49 +0100 Subject: [Pymilter] pymilter segfaults In-Reply-To: References: <49103BA9.5090401@uptime.at> Message-ID: <49368649.8080509@uptime.at> Stuart D. Gathman wrote: > On Tue, 4 Nov 2008, Michal Mach wrote: > > >> mimemilter.py[23082]: segfault at 0000000000000130 rip 00002acb089e8900 rsp >> 00000000410010f0 error 4 >> >> Similar problem was discussed in this mailing list here: >> >> http://www.bmsi.com/pipermail/pymilter/2007-January/000143.html >> > > Never heard back from that guy, so I assumed it was the mysql module. > (No one else has had the problem since.) > > >> Can you give me some suggestions what could be the problem? >> If you need more info please drop me an mail. >> > > One thing to try is to get version 1.9 of milter_module.c from here: > > http://pymilter.cvs.sourceforge.net/pymilter/milter/miltermodule.c?view=log > > and rebuild milter.so ("python setup.py build" and move into place or adjust > PYTHON_PATH). If the problem goes away, then that means that more than one > thread was accessing a milter ctx (maybe because of something libmilter does). > I'll document that and revert. > > Hi, So i downloaded the older miltermodule.c module and recompiled the debian package, but we still get the segfaults. I am attaching some pieces of output from logs when the segfault happens. It seams that it happens when calling close methods from different threads one after the other. But i don't now if this is the problem. Log.1 2008-12-03 05:50:49 [INFO ] [1191237968] [Dummy-13] hello from localhost.localdomain 2008-12-03 05:50:49 [INFO ] [1191237968] [Dummy-13] Mail address 'test at example.com'. [Accepting] 2008-12-03 05:50:49 [INFO ] [1098918224] [Dummy-2] Called close for message with id: undefined 2008-12-03 05:50:49 [INFO ] [1191237968] [Dummy-13] connect from localhost at ('127.0.0.1', 0) 2008-12-03 05:50:49 [INFO ] [1191237968] [Dummy-13] hello from localhost 2008-12-03 05:50:49 [INFO ] [1191237968] [Dummy-13] Mail address 'test at example.com'. [Accepting] 2008-12-03 05:50:49 [INFO ] [1098918224] [Dummy-2] Called close for message with id: undefined 2008-12-03 05:51:02 [INFO ] [1098918224] [Dummy-2] Called close for message with id: undefined 2008-12-03 05:51:12 [INFO ] [1090525520] [Dummy-17] Called close for message with id: undefined Log.2 2008-12-03 06:41:09 [INFO ] [1107310928] [Dummy-2] Interdomain disclaimer applied for user: test at example.com 2008-12-03 06:41:12 [INFO ] [1098918224] [Dummy-1] Called close for message with id: 48BA2A36F 2008-12-03 06:41:16 [INFO ] [1107310928] [Dummy-2] connect from localhost at ('127.0.0.1', 0) 2008-12-03 06:41:16 [INFO ] [1107310928] [Dummy-2] hello from localhost 2008-12-03 06:41:16 [INFO ] [1098918224] [Dummy-1] Disclaimer already present. Mail queue id: 666F4A370 2008-12-03 06:41:16 [INFO ] [1098918224] [Dummy-1] Called close for message with id: 666F4A370 2008-12-03 06:41:36 [INFO ] [1090525520] [Dummy-3] Called close for message with id: undefined Log.3 2008-12-03 00:19:04 [INFO ] [1149274448] [Dummy-6] hello from localhost.localdomain 2008-12-03 00:19:04 [INFO ] [1149274448] [Dummy-6] Mail address 'test at example.com'. [Accepting] 2008-12-03 00:19:04 [INFO ] [1098918224] [Dummy-1] Called close for message with id: undefined 2008-12-03 00:19:04 [INFO ] [1149274448] [Dummy-6] connect from localhost at ('127.0.0.1', 0) 2008-12-03 00:19:04 [INFO ] [1149274448] [Dummy-6] hello from localhost 2008-12-03 00:19:04 [INFO ] [1149274448] [Dummy-6] Mail address 'test at example.com'. [Accepting] 2008-12-03 00:19:04 [INFO ] [1098918224] [Dummy-1] Called close for message with id: undefined 2008-12-03 00:19:14 [INFO ] [1090525520] [Dummy-8] Called close for message with id: undefined Log.4 2008-12-02 18:46:54 [INFO ] [1115703632] [Dummy-3] Mail address 'test at example.com'. [Accepting] 2008-12-02 18:46:55 [INFO ] [1098918224] [Dummy-1] Called close for message with id: undefined 2008-12-02 18:46:56 [INFO ] [1115703632] [Dummy-3] connect from localhost at ('127.0.0.1', 0) 2008-12-02 18:46:56 [INFO ] [1115703632] [Dummy-3] hello from localhost 2008-12-02 18:46:56 [INFO ] [1115703632] [Dummy-3] Mail address 'test at example.com'. [Accepting] 2008-12-02 18:46:56 [INFO ] [1098918224] [Dummy-1] Called close for message with id: undefined 2008-12-02 18:47:16 [INFO ] [1090525520] [Dummy-4] Called close for message with id: undefined Log.5 2008-12-02 21:57:58 [INFO ] [1107310928] [Dummy-1] hello from example.com 2008-12-02 21:57:58 [INFO ] [1107310928] [Dummy-1] Mail address 'test at example.com'. [Accepting] 2008-12-02 21:57:58 [INFO ] [1166059856] [Dummy-9] Called close for message with id: undefined 2008-12-02 21:57:58 [INFO ] [1107310928] [Dummy-1] connect from localhost at ('127.0.0.1', 0) 2008-12-02 21:57:58 [INFO ] [1107310928] [Dummy-1] hello from localhost 2008-12-02 21:57:58 [INFO ] [1107310928] [Dummy-1] Mail address 'test at example.com'. [Accepting] 2008-12-02 21:57:58 [INFO ] [1166059856] [Dummy-9] Called close for message with id: undefined 2008-12-02 21:58:18 [INFO ] [1090525520] [Dummy-5] Called close for message with id: undefined From stuart at bmsi.com Wed Dec 3 22:35:50 2008 From: stuart at bmsi.com (Stuart D. Gathman) Date: Wed, 3 Dec 2008 22:35:50 -0500 (EST) Subject: [Pymilter] Re: pymilter support for sendmail 8.14 CHGFROM In-Reply-To: <20081204020635.GL27124@monkey.uchicago.edu> References: <20081204020635.GL27124@monkey.uchicago.edu> Message-ID: On Wed, 3 Dec 2008, David Champion wrote: > I needed the ability to alter the sender in a milter I'm developing. > This is a new feature in sendmail 8.14, so I upgraded and added support > to pymilter. I hope it's correct, and that you find it useful. As I mentioned on the pymilter list, that was already in CVS. It will be in 0.8.12 as soon as I can cut a release. Sorry for the extra work. We can compare work to see if either of us missed something. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From dgc at uchicago.edu Thu Dec 4 00:53:10 2008 From: dgc at uchicago.edu (David Champion) Date: Wed, 3 Dec 2008 23:53:10 -0600 Subject: [Pymilter] Re: pymilter support for sendmail 8.14 CHGFROM In-Reply-To: References: <20081204020635.GL27124@monkey.uchicago.edu> Message-ID: <20081204055310.GM27124@monkey.uchicago.edu> > As I mentioned on the pymilter list, that was already in CVS. It will be > in 0.8.12 as soon as I can cut a release. Sorry for the extra work. We > can compare work to see if either of us missed something. Oh, heh. The pymilter list, you say. I guess I should join that. :) -- -D. dgc at uchicago.edu NSIT University of Chicago From stuart at bmsi.com Thu Dec 4 13:08:10 2008 From: stuart at bmsi.com (Stuart D. Gathman) Date: Thu, 4 Dec 2008 13:08:10 -0500 (EST) Subject: [Pymilter] pymilter segfaults In-Reply-To: <49368649.8080509@uptime.at> References: <49103BA9.5090401@uptime.at> <49368649.8080509@uptime.at> Message-ID: On Wed, 3 Dec 2008, Michal Mach wrote: > So i downloaded the older miltermodule.c module and recompiled the debian > package, but we still get the segfaults. > > I am attaching some pieces of output from logs when the segfault happens. It > seams that it happens when calling close methods from different threads one > after the other. But i don't now if this is the problem. Is your python code calling the close callback? It should be called by libmilter only. If your python code calls the close method, then it destroys the milter context - which would lead to segfault when libmilter next attempts a callback. This would be a bug - the callbacks should detect if the context has already been destroyed and throw an exception. ... > 2008-12-03 05:50:49 [INFO ] [1098918224] [Dummy-2] Called close for message > with id: undefined ... > 2008-12-03 06:41:12 [INFO ] [1098918224] [Dummy-1] Called close for message > with id: 48BA2A36F What is the id you are printing? -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Thu Dec 4 13:53:25 2008 From: stuart at bmsi.com (Stuart D. Gathman) Date: Thu, 4 Dec 2008 13:53:25 -0500 (EST) Subject: [Pymilter] pymilter segfaults In-Reply-To: References: <49103BA9.5090401@uptime.at> <49368649.8080509@uptime.at> Message-ID: On Thu, 4 Dec 2008, Stuart D. Gathman wrote: > Is your python code calling the close callback? It should be called by > libmilter only. If your python code calls the close method, then > it destroys the milter context - which would lead to segfault when libmilter > next attempts a callback. This would be a bug - the callbacks should detect > if the context has already been destroyed and throw an exception. It is tricky because it is really designed for only 1 thread at a time to use a milter context. Let me talk to myself for a bit to try and reason it out: 1. The Python close code is just calling getpriv/setpriv. It doesn't mess with the libmilter ctx. 2. The libmilter close callback doesn't delete the python milter object. It destroys the associated PyThreadState and unlinks from libmilter ctx, but the milter object stays around until references drop to 0. getpriv/setpriv from python will continue to work. 3. Should python code attempt to invoke other context methods that require a libmilter ctx, they call _find_context. _find_context assumes that the interpreter is locked *and* that libmilter will not be messing with the libmilter ctx since it is waiting for a callback to complete. 4. But suppose that python code invokes a context method when *not* in a callback? Or suppose that libmilter gets tired of waiting for a callback and closes the context asyncronously? In that case, the libmilter context might get destroyed while python is trying to use it, causing a segfault. Does 4 sound like it might be happening in your milter? (Does anyone know if libmilter might call close while another callback hasn't returned?) -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Thu Dec 4 14:09:34 2008 From: stuart at bmsi.com (Stuart D. Gathman) Date: Thu, 4 Dec 2008 14:09:34 -0500 (EST) Subject: [Pymilter] pymilter segfaults In-Reply-To: References: <49103BA9.5090401@uptime.at> <49368649.8080509@uptime.at> Message-ID: On Thu, 4 Dec 2008, Stuart D. Gathman wrote: > 4. But suppose that python code invokes a context method when *not* in > a callback? Or suppose that libmilter gets tired of waiting for > a callback and closes the context asyncronously? In that case, the libmilter > context might get destroyed while python is trying to use it, causing > a segfault. > > Does 4 sound like it might be happening in your milter? (Does anyone know > if libmilter might call close while another callback hasn't returned?) But milter_wrap_close protects against 4 by calling PyEval_AcquireThread before unlinking from libmilter context and destroying PyThreadState. And we handle wrap_close getting called before wrap_connect also by checking if there is a milter_ContextObject. It seems pretty watertight, but apparently there is a leak or bad assumption somewhere. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Thu Dec 4 14:53:06 2008 From: stuart at bmsi.com (Stuart D. Gathman) Date: Thu, 4 Dec 2008 14:53:06 -0500 (EST) Subject: [Pymilter] pymilter segfaults In-Reply-To: References: <49103BA9.5090401@uptime.at> <49368649.8080509@uptime.at> Message-ID: On Thu, 4 Dec 2008, Stuart D. Gathman wrote: > It seems pretty watertight, but apparently there is a leak or bad > assumption somewhere. Let's try to note things that are different about your setup, and see if that yields a clue. It looks like you are on a 64-bit system. Also, I am still on python2.4, although many other pymilter users are likely on python2.5. Does anyone else use pymilter on 64-bit? I will try to get python2.5 installed on some production mail servers to see if that gives me the problem. Since python2.5 tweaks the python C-API, it might pay to examine the C-API changes for possible effects on pymilter. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Thu Dec 4 14:59:14 2008 From: stuart at bmsi.com (Stuart D. Gathman) Date: Thu, 4 Dec 2008 14:59:14 -0500 (EST) Subject: [Pymilter] pymilter segfaults In-Reply-To: References: <49103BA9.5090401@uptime.at> <49368649.8080509@uptime.at> Message-ID: On Thu, 4 Dec 2008, Stuart D. Gathman wrote: > I will try to get python2.5 installed on some production mail servers to > see if that gives me the problem. Since python2.5 tweaks the python > C-API, it might pay to examine the C-API changes for possible effects > on pymilter. The two likely segfault inducing changes are: 1) http://www.python.org/doc/2.5.2/whatsnew/pep-353.html#pep-353 Must use Py_ssize_t instead of int in many situations. Can break working 2.4 code with 2.5 on 64-bit. 2) Changes to obmalloc means that PyMem and PyObject calls must be carefully matched. This would break working (but incorrect) 2.4 code on 2.5. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Thu Dec 4 15:23:47 2008 From: stuart at bmsi.com (Stuart D. Gathman) Date: Thu, 4 Dec 2008 15:23:47 -0500 (EST) Subject: [Pymilter] pymilter segfaults In-Reply-To: References: <49103BA9.5090401@uptime.at> <49368649.8080509@uptime.at> Message-ID: On Thu, 4 Dec 2008, Stuart D. Gathman wrote: > The two likely segfault inducing changes are: > > 1) http://www.python.org/doc/2.5.2/whatsnew/pep-353.html#pep-353 > Must use Py_ssize_t instead of int in many situations. Can break > working 2.4 code with 2.5 on 64-bit. The only function affected by the change according to ssizecheck.py is generic_env_wrapper(), and all Py_ssize_t args are autoconverted by compiler (passed as non-varargs). > 2) Changes to obmalloc means that PyMem and PyObject calls must be > carefully matched. This would break working (but incorrect) > 2.4 code on 2.5. PyMem_* is never used. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Fri Dec 5 15:16:18 2008 From: stuart at bmsi.com (Stuart D. Gathman) Date: Fri, 5 Dec 2008 15:16:18 -0500 (EST) Subject: [Pymilter] Splitting milter module Message-ID: I have submitted a request to sourceforge to copy the milter CVS module to pymilter. This will allow splitting that source module into pymilter (the extension module and Milter package) and milter (the all singing all dancing python milter that I use in production). I believe this will be better for projects depending on pymilter. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From m.mach at uptime.at Mon Dec 22 08:18:34 2008 From: m.mach at uptime.at (Michal Mach) Date: Mon, 22 Dec 2008 14:18:34 +0100 Subject: [Pymilter] pymilter segfaults In-Reply-To: References: <49103BA9.5090401@uptime.at> <49368649.8080509@uptime.at> Message-ID: <494F93AA.7000908@uptime.at> Stuart D. Gathman wrote: > On Wed, 3 Dec 2008, Michal Mach wrote: > > >> So i downloaded the older miltermodule.c module and recompiled the debian >> package, but we still get the segfaults. >> >> I am attaching some pieces of output from logs when the segfault happens. It >> seams that it happens when calling close methods from different threads one >> after the other. But i don't now if this is the problem. >> > > Is your python code calling the close callback? It should be called by > libmilter only. If your python code calls the close method, then > it destroys the milter context - which would lead to segfault when libmilter > next attempts a callback. This would be a bug - the callbacks should detect > if the context has already been destroyed and throw an exception. > No, sorry i meant when the pymilter is calling the close method. I am not calling any of the milter methods in my code. > ... > >> 2008-12-03 05:50:49 [INFO ] [1098918224] [Dummy-2] Called close for message >> with id: undefined >> > ... > >> 2008-12-03 06:41:12 [INFO ] [1098918224] [Dummy-1] Called close for message >> with id: 48BA2A36F >> > > What is the id you are printing? > > This is the id which i get when calling getsymval("i") in eoh callback. When the method getsymval("i") dosn't returned an id, i print "undefined".