Python Forum
performance in network host of manager in multiprocessing - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Networking (https://python-forum.io/forum-12.html)
+--- Thread: performance in network host of manager in multiprocessing (/thread-36569.html)



performance in network host of manager in multiprocessing - rmaleki - Mar-06-2022

Hi
I run my own project with gunicorn (multi worker) and I wanted to share objects between processes. I used BaseManager from the multiprocessing python package. my project is Django and this is part of code:

def get_bloom_filter_shared_state():
    manager = BaseManager(
        ('127.0.0.1', '8004'),
        b'blabla'
    )
    m_bloom_filter = BloomFilter()
    a_bloom_filter = BloomFilter()

    # n_share_state: memory sharing already exists or new.
    n_share_state = False

    # register in manager
    manager.register('m_bloom_filter', lambda: m_bloom_filter)
    manager.register('a_bloom_filter', lambda: a_bloom_filter)

    try:
        manager.get_server()
        manager.start()
        n_share_state = True

    except OSError:
        # Address already in use
        manager.connect()

    return manager.m_bloom_filter(), manager.a_bloom_filter(), n_share_state
Is the performance of the network low in many requests? What is the performance of this host (127.0.0.1)?


RE: performance in network host of manager in multiprocessing - DeaD_EyE - Mar-06-2022

The TCP/IP Stack is fast enough to handle it. A Unix-Socket should reduce overhead.
Using a UNIX-Socket is for free and does not require many code changes.

The most overhead comes from IPC itself.
The Processes receive the data serialized with pickle and use locks around this action, to be atomic.


RE: performance in network host of manager in multiprocessing - rmaleki - Mar-07-2022

Thanks
Your hint helped me a lot.