[Pymilter] no memory release when assigning chunks to self.fp in milter-template.py

[Stuart D Gathman]

On Fri, 3 Jan 2020, joao reis wrote:

> milter-template.py seems to accumulate memory constantly while saving the 
> message chunks in self.fp.
>
> Here is a minor adaptation of the milter-template running on python3 (I 
> noticed the same problem with pymilter running on python2) follows 
> (https://github.com/joaoreis81/milter-template) with few modifications that 
> reproduce the problem. On a reasonably loaded postfix server, after 30 
> minutes of messaging the process memory increased to 500MB

Postfix vs sendmail is a big difference.  What normally happens
on sendmail is:

1. MTA send SMFI_CLOSE message, 
2. libmilter calls close_callback, which calls close method of python
 	object
3. libmilter deletes milter_Context, releasing python object
    (there is also an SMFICTX object that points to milter_Context
     and is managed by libmilter)

You can investigate whether 1 and 2 is happening by adding some kind of
logging to your close() method.

You can investigate whether 3 is happening by logging the result
of Milter.getdiag() periodically.  This returns a count of 
milter_Context NEW and milter_context DEL.

I would normally suspect some user introduced global data structure,
but you have already determined that this only happens when writing
to a BytesIO object: commenting out self.fp.write(chunk) stops 
(most of?) memory growth.

You may have found a bug in BytesIO.  I know I've hit many in the 
email package over the years (some reported almost 2 decades ago and
still not fixed - probably for fear of breaking something that depends
on the bugs...)