[Pymilter] no memory release when assigning chunks to self.fp in milter-template.py

[joao reis]

Sorry for the delay to reply. The close() method is executed, because 
one of the tests I did was creating a file on disk in self.fp, 
processing and deleting the file in close() method. All emails sent were 
created and deleted, so the close () is certainly running.
How could I explicitly test item 3? Could you give me an example, please?

This doesn't just happen with BytesIO, it happens with StringIO, writing 
the content in variables or creating a file. If I comment on 
self.fp.write (chunk) the memory does not grow.

I've tried to use memory profiling modules but didn't work corretly 
since it is a daemon...

Any ideias to advance?

Att,

João Reis.

On 1/5/20 9:02 PM, Stuart D Gathman wrote:
> On Fri, 3 Jan 2020, joao reis wrote:
>
>> milter-template.py seems to accumulate memory constantly while saving 
>> the message chunks in self.fp.
>>
>> Here is a minor adaptation of the milter-template running on python3 
>> (I noticed the same problem with pymilter running on python2) follows 
>> (https://github.com/joaoreis81/milter-template) with few 
>> modifications that reproduce the problem. On a reasonably loaded 
>> postfix server, after 30 minutes of messaging the process memory 
>> increased to 500MB
>
> Postfix vs sendmail is a big difference.  What normally happens
> on sendmail is:
>
> 1. MTA send SMFI_CLOSE message, 2. libmilter calls close_callback, 
> which calls close method of python
>     object
> 3. libmilter deletes milter_Context, releasing python object
>    (there is also an SMFICTX object that points to milter_Context
>     and is managed by libmilter)
>
> You can investigate whether 1 and 2 is happening by adding some kind of
> logging to your close() method.
>
> You can investigate whether 3 is happening by logging the result
> of Milter.getdiag() periodically.  This returns a count of 
> milter_Context NEW and milter_context DEL.
>
> I would normally suspect some user introduced global data structure,
> but you have already determined that this only happens when writing
> to a BytesIO object: commenting out self.fp.write(chunk) stops (most 
> of?) memory growth.
>
> You may have found a bug in BytesIO.  I know I've hit many in the 
> email package over the years (some reported almost 2 decades ago and
> still not fixed - probably for fear of breaking something that depends
> on the bugs...)

<!DSPAM:170117BA6692091877400068>