[Pymilter] no memory release when assigning chunks to self.fp in milter-template.py

[Marco]

Hello,

   this worry me too... I'm a newby in Python. I'm testing the milter 
which I'm developing for memory leakage. While Python seems to require a 
lot of memory, I can't see a really leakage at the moment.

If I sent 600 mails, each of 1MB in 1 minute I see about 85MB in RSS and 
about a max of 352MB of VSZ. This last amount of memory doesn't seem to 
increase for each new mail I send.

So, if after a while I resend another bunch of 600 mails, the VSZ 
doesn't increase no more. It seems a sort of pool.

I use Postfix with Red Hat 7.

I tried to run gc.collect, but it doesn't seem to change something. 
Really, I even found a missed <file>.close() in my code, but it seems to 
be irrelevant. It seems that python manages the memory itself quite 
well, but it releases the RAM "when it wants", not really just after a 
close().

I'm still testing this.

Kind Regards
Marco

Il 03/01/2020 19:44, joao reis ha scritto:
> Hi,
> 
> milter-template.py seems to accumulate memory constantly while saving 
> the message chunks in self.fp.
> 
> Here is a minor adaptation of the milter-template running on python3 (I 
> noticed the same problem with pymilter running on python2) follows 
> (https://github.com/joaoreis81/milter-template) with few modifications 
> that reproduce the problem. On a reasonably loaded postfix server, after 
> 30 minutes of messaging the process memory increased to 500MB
> 
> In my last tests, I am running milter-template as a container and the 
> increase in memory occurs both when running directly in the operating 
> system and the container environment. We can certainly exclude this as a 
> possible cause of the problem described.
> 
> As stated earlier, all file descriptors cleanup tests were done but 
> unsuccessful for memory release, it looks like a "memory cancer" really.
> 
> I have been using Red Hat based operating systems (fedora, centos) and 
> the problem always comes up. I will perform some tests changing 
> distribution to verify the result.
> 
> I am passing this thread to the mailing list as directed.
> 
> 
> Thanks in advance
> João Reis.
> _______________________________________________
> Pymilter mailing list
> Pymilter at gathman.org
> https://gathman.org/mailman/listinfo/pymilter