[Pymilter] milter multiprocess (or multithread)

[Stuart D. Gathman]

On Tue, 12 Nov 2019, Marco wrote:

> I would like to ask a question as a beginner on Python.
>
> milter-template.py seems to be an example of multiprocess milter:
>
> https://github.com/sdgathman/pymilter/blob/python3-branch/milter-template.py
>
> For python3 I had to change something, such as BytesIO in place of 
> StringIO... but I managed to work with it.

Thanks, I should update it since py2 is officially obsolete soon.

> "# each connection runs in its own thread and has its own myMilter
> # instance."
>
> This is what I want to achieve. I would like to write a milter with a TCP 
> socket where each client connection (or mail) open a new process.

A thread is different than a process.  A process can have multiple threads.
Threads also run in parallel - but share the same address space within
a process.

> The milter make several things, it writes to RedisDB, ask something with REST 
> API and finally decides if the mail has to be accepted or rejected.

> I would like to manage these operation in parallel, with a process for each 
> new mail passed to the milter. Each process doesn't need to wait some results 
> from other processes.

This can be done with threads or processes.  A caveat with threads
unique to C-Python is the GIL (Global Interpreter Lock).  Python code
in threads does not run in parallel, but IO does and C libraries (like
pymilter) can.

> But trying this template if I well understand I see that only two processes 
> are opened. A master process and one child which handle all the connections:

The master process in the example has a thread for each connection.  As an
illustration, it creates one Process "thread" (almost the same API as 
Thread threads) to do logging.  Thus the connection threads don't have
to wait for the logging to complete.  This is not actually useful in
practice, as I/O is already in parallel between threads, but just
an example to give you ideas.

The things you mentioned should not require any additional processes 
to happen in parallel.  Each connection has its own thread.

But maybe you don't want connection threads (and thus sendmail/postfix)
waiting for RedisDB.  You could have one background process to write to
RedisDB, and connection threads will just queue up data for it (similar
to the logging process in the example).  The drawback to such
asynchronous writes is that the connection thread doesn't wait to find
out if the write succeeded.  E.g. if the RedisDB process crashes, the
connection thread has already gone on its merry way ...

> Could you help me to understand how to achieve the result to open new process 
> (milter instance) for every new mail sent to the milter socket? Or, 
> alternatively, a new process for every new client (MTA) connected to the 
> milter.

You already get a new thread for every connection out of the box.  You
almost certainly don't want to create a process for each connection.

-- 
 	      Stuart D. Gathman <stuart at gathman.org>
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.