Inter-Process Communication (IPC) in Python [with Examples] | Apriorit (2023)

Modern software is often based on inter-process communication (IPC). Programs and processes use this mechanism to run in parallel, communicate, and share data with each other, ensuring high performance, efficient data processing, and reliable security.

In this article, we discuss common IPC methods for Python software. You’ll learn what to pay attention to when planning your IPC workflow and find practical examples of inter-process communication in Python with various Python libraries and frameworks.

This article will be useful for project teams that are considering implementing IPC mechanisms in their projects with the use of Python tools.


The need for inter-process communication

Common IPC scenarios and their limitations

Python tools for IPC data serialization

Practical examples of IPC methods


The need for inter-process communication

In monolithic applications, using language-level function or method calls is enough for establishing interactions among application components. But today’s complex solutions, such as microservices-based applications, run on several instances and thus require more advanced data exchange mechanisms.

Inter-process communication (IPC) is a mechanism that allows programs or separate processes to securely communicate with each other, exchange data, and synchronize their activities.

Common benefits of using IPC are:

Inter-Process Communication (IPC) in Python [with Examples] | Apriorit (1)
  • Efficient data sharing — Using IPC, you can enable different processes in your application to access the same data concurrently.
  • Task modularity — You can split a large task into a set of smaller subtasks that can be accomplished by different processes, thus finishing the initial task faster.
  • Optimized code — With IPC, you don’t have to write extra code for managing communication between programs and processes at a low level.

To manage inter-process communication, you can use various methods including:

  • Direct communication — Processes exchange data either by using shared data structures in memory or by passing shared data as parameters to function calls.
  • Message queue — Processes use a shared list of messages to communicate with each other without establishing a direct connection.
  • Message passing — Processes exchange data and synchronize their activities via received and sent messages.
  • Shared memory — Processes use a designated memory region that they can access at the same time.
  • Pipes — Processes read data from and write data to a common channel that works as a shared data buffer.
  • Sockets — Processes establish two-way communication and exchange data using an endpoint for communication.

In this article, we discuss several common ways of managing inter-process communication and offer practical examples of each.

Note that our practical examples show how to do inter-process communication in Python.

Python is famous for being easy to use and offering a large variety of third-party libraries, modules, and frameworks for all kinds of tasks. However, the specifics of inter-process communication within your project might present additional limitations that you need to take into account when choosing your set of tools.

Related services

Custom Python Development Services

Common IPC scenarios and their limitations

Depending on the specifics of your solution, communication among processes can be categorized based on different dimensions:

  • Local or network
  • Synchronous or asynchronous
  • Single-platform or cross-platform

Each of these categories has unique limitations that will influence your choice of third-party tools. Let’s take a closer look at these approaches for inter-process communication in Python.

Local vs network communication

If your code is supposed to run on a single machine, managing IPC is rather simple.

However, if your application requires connecting processes executed on different machines, you need to take an advanced approach. As communication among multiple machines is fundamentally unstable, you need to use special libraries like wrappers over the TCP/IP stack.

Another thing to take into account when configuring network communication is data security. If you plan to transfer data across the network, make sure to implement proper encryption mechanisms to keep that data well protected. In particular, consider using TLS/SSL encryption for socket connections or SHA-256 hashing, AES, or RSA data encryption for other cases.

Synchronous vs asynchronous communication

Process interactions can be synchronous or asynchronous. This distinction is especially important for web applications and microservices-based applications, and it defines the behavior of the data-sending side (the client) after starting a communication with the server:

  • For synchronous processes, the client sends a request and then waits for the server’s response before continuing code execution.
  • For asynchronous processes, the client can continue code execution without waiting for the server’s response.

A request from one client can be processed by either one or multiple services. Thus, for some applications, using a single IPC mechanism might be sufficient, while others will need a combination of IPC mechanisms.

Single-platform vs cross-platform communication

Some types of communication can only be used within a specific operating system or architecture. Common examples of such communication mechanisms are:

While each of these communication mechanisms can be used in an application written in Python, the functionality they are used in will be difficult to port to other platforms, thus making your solution platform-specific.

Therefore, if you are planning to build a cross-platform solution, avoid using platform-specific communication mechanisms. Also, be attentive when choosing a Python library, as there are libraries that use platform-specific IPC mechanisms under the hood.

Choosing the communication mechanism most suitable for a specific task is crucial. But it’s also important to pick the right approach for serializing and deserializing your data so that it can be transferred securely and efficiently..

In the next section, we overview several Python modules and frameworks you can use for serialization and deserialization of IPC data.

Read also:
Extending IDA’s Capabilities with Python: A Practical Example of Disassembling an Xtensa Instruction

Data serialization is key to ensuring the secure storage and transfer of data during inter-process communication. Python offers a wide selection of dedicated modules and frameworks for changing data structures into a suitable format.

Let’s look at four common Python IPC tools you can use to serialize, deserialize, and validate transferred data:

  • JSON module
  • Protobuf framework
  • Pydantic library
  • Struct module

JSON module

Using the standard JSON module, you can encode and decode strings in JSON format to ordinary Python objects.

To see how you can work with this module, let’s first look at an example of a JsonWrapper class:


import ioimport json class JsonWrapper: @staticmethod def encode(obj, encoding): s = json.dumps(obj, ensure_ascii=False).encode(encoding) print("Encoded JSON: {}".format(s)) return s  @staticmethod def decode(json_bytes, encoding): with io.TextIOWrapper(io.BytesIO(json_bytes), encoding=encoding, newline="") as tiow: print("Decoded JSON: {}".format(json_bytes)) return json.load(tiow)

Using this wrapper class, you can create a JSON binary string from a Python object, and vice versa.

Here’s what the code for encoding data to a JSON binary string looks like:


>>> from json_tools import JsonWrapper>>> request_json = {"action" : "hello", "message" : "Hello, World!"}>>> request_data = JsonWrapper.encode(request_json, "utf-8")Encoded JSON: b'{"action": "hello", "message": "Hello, World!"}'>>> request_datab'{"action": "hello", "message": "Hello, World!"}'

And here’s an example for decoding a JSON binary string:


>>> from json_tools import JsonWrapper>>> binary_string = b'{"action" : "hello", "message" : "Hello, World!"}'>>> response = JsonWrapper.decode(binary_string, "utf-8")Decoded JSON: b'{"action" : "hello", "message" : "Hello, World!"}'>>> response{'action': 'hello', 'message': 'Hello, World!'}

This module is easy to use, and it provides means for working with Python objects. However, if you plan on working with the JSON module, keep in mind that it requires additional exception handling for the data encoder and decoder. This module is also not suitable for unstable connections, as JSON decoding is very dependent on the format’s strict structure.

Read also:
Python for Building FinTech Solutions: Benefits, Tools, and Use Cases

Protobuf framework

Protocol Buffers, or simply Protobuf, is a versatile framework from Google that you can use for constructing a domain-specific messaging system.

When working with Protobuf, the usual workflow consists of three steps:

  1. Define objects and messages in a .proto file
  2. Run the compiler command that will create language-specific headers
  3. Include the generated files in the project

Let’s look at each of these steps in practice.

Step 1. Define objects and messages in a .proto file

Inside the project code tree, there should be a .proto file that describes a data structure. As an example, we’ll create an employee.proto file:


syntax = "proto2"; package tutorial; message Employee { required int32 id = 1; required string name = 2; optional string favorite_color = 3;}

For each data structure you want to serialize, you need to add a new message, specifying the name and type for each field in that message. For more information on creating nested data structures and arrays, see Protofub’s official documentation.

Step 2. Run the compiler command to create language-specific headers

At this step, you need to run the protobuf compiler and specify the source directory with your application’s source code, the destination directory where the generated code will be placed, and the path to the .proto file:


protoc -I=<source dir> --python_out=<output dir> <source dir>/employee.proto

The –python_out command-line argument indicates that the output must be a Python module. The compiler may create output files for a number of other languages.

The next command creates an file with multiple internal wrappers, encoders, decoders, and so on:


class Employee(message.Message): __metaclass__ = reflection.GeneratedProtocolMessageType DESCRIPTOR = _EMPLOYEE...

Step 3. Include generated files in the project

Now you can work with the created employee class:


import employee_pb2  class ProtobufWrapper: @staticmethod def encode(id, name): employee = employee_pb2.Employee() = id = name return employee.SerializeToString()  @staticmethod def decode(raw_data): employee = employee_pb2.Person() employee.ParseFromString(raw_data) return employee

It’s noteworthy that when you serialize data using the Protobuf framework, you create a compact binary representation of an object:

Inter-Process Communication (IPC) in Python [with Examples] | Apriorit (2)

Here’s an example of code that launches data serialization:


import employee_pb2class ProtobufWrapper:@staticmethoddef encode(id, name): employee = employee_pb2.Employee() = id = name return employee.SerializeToString() @staticmethoddef decode(raw_data): employee = employee_pb2.Person() employee.ParseFromString(raw_data) return employeeserialized_structure = ProtobufWrapper.encode(1, "John Doe")print(serialized_structure)

With Protobuf, you need to write less code than when using other tools, because data is defined in the .proto file. Auto-generated serialization also simplifies the handling of data structures. Since messages reside in the same file, there’s no need to ensure the work of both client and server code with the same entities. You can also use the same .proto file to generate headers for multiple languages, ensuring seamless Python inter-process communication with code written in C/C++ and other languages.

At the same time, the need to work with the compiler adds extra steps to the standard serialization process. As an alternative to using the compiler, you can store module outputs directly in the source code tree, thus cluttering your code.

Read also:
C# vs Python vs C Comparison Guide: Key Differences, Benefits, and Challenges

Pydantic library

The Pydantic library is a useful tool for data validation that provides type hints at runtime. Using Pydantic models, you can ensure smooth and efficient serialization and deserialization of complex data structures.

Here’s an example of a Pydantic script for data validation during serialization or deserialization:


from typing import Optionalfrom pydantic import BaseModel class User(BaseModel): name: str id: int signup_ts: Optional[datetime] = None colleagues: list[int] = []  external_data = { 'name': 'Example User', 'id': 3, 'signup_ts': '2019-06-01 12:22', 'colleagues': [5, 6],}user = User(**external_data)

If validation fails, the script will end with an error, providing you with details on exactly what went wrong.

The Pydantic library can be used for both loading system settings and validating request data. It allows for validating complex data structures yet has better performance than many similar libraries. However, this approach might be redundant for small data structures with simple types and frequent communications, such as heartbeats or non-critical time synchronization via UDP.

Read also:
The Power of Python: Pros and Cons, Integration Capabilities, and Use Cases

Struct module

In order to communicate with native code like a C/C++ program or a native application binary interface (ABI) library at a low level, high-level Python code requires wrappers that account for type sizes, endianness, and pack alignments.

Using the standard struct module, you can work with raw binary data, packing and unpacking binary streams into usable data structures.

The pack and unpack methods receive a special format string that represents a low-level data structure: endianness, bit width, etc.

Here’s what a C-style Employee.h structure created with this module looks like:


struct Employee { char name[256]; uint16_t id; uint16_t workplace_room_id; uint8_t workplace_id;}

The client and server code would require the following wrappers to pack a Python dictionary to or from binary sequences:


import struct class StructWrapper: @staticmethod def encode(employee): name = employee.get('name') id = employee.get('id') workplace_room_id = employee.get('room_id') workplace_id = employee.get('workplace_id') return struct.pack('<256sHHb', name, id, workplace_room_id, workplace_id)  @staticmethod def decode(raw_data): name, id, workplace_room_id, workplace_id = struct.unpack('<256sHHb', raw_data) return { 'name': name, 'id': id, 'workplace_room_id': workplace_room_id, 'workplace_id': workplace_id }

You can learn more about using format strings when packing and unpacking data in the struct module documentation.

The struct module offers a convenient solution for connecting to low-level ABIs, C/C++ libraries, and legacy code. However, the data encoding and decoding process is hidden and requires precision from a developer. And since the struct module works at a low level, you might need extensive unit test coverage for packing and unpacking operations to discover possible errors.

Since handling of data definitions as strings is not very clear and straightforward, there are other Python modules you can use for working with native code:

Now that you know what tools you can use to manipulate data structures in the desired way, let’s look at some practical examples of IPC methods.

Related services

Practical examples of IPC methods

As we discussed earlier, sockets are one of the advanced methods of inter-process communication. This method is often used in complex software, such as web applications and microservices solutions. In this section, we go over both platform-specific and cross-platform IPC methods based on the use of sockets, providing both client-side and server-side code examples.

Inter-Process Communication (IPC) in Python [with Examples] | Apriorit (3)

BSD socket files

Note: This IPC method is platform-specific and is only suitable for Unix-like operating systems.

In Unix-like platforms, socket files have the .sock extension and are used as communication endpoints for processes, programs, or servers. For communication within the same machine, you can also use Berkeley Software Distribution (BSD) socket files. However, to enable communication across machines, BSD socket files require an additional setup such as file node mapping.

Let’s take a look at an example of client-side code for a BSD socket file:


import osimport socketimport sys # IPC parametersSOCK_FILE = '/tmp/simple-ipc.socket' # Init socket objectif not os.path.exists(SOCK_FILE): print(f"File {SOCK_FILE} doesn't exists") sys.exit(-1) s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)s.connect(SOCK_FILE) # Send requests.sendall(b'Hello, world') # Wait for responsedata = s.recv(1024)print(f'Received bytes: {repr(data)}')

And here’s what the server-side code for this socket looks like:


import osimport socket # IPC parametersSOCK_FILE = '/tmp/simple-ipc.socket' # Setup socketif os.path.exists(SOCK_FILE): os.remove(SOCK_FILE) s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)s.bind(SOCK_FILE)s.listen(0) # Start listening loopwhile True: # Accept 'request' conn, addr = s.accept() print('Connection by client') # Process 'request' while True: data = conn.recv(1024) if not data: break # Send 'response' conn.sendall(data)

You can also take a look at what the example above looks in the console:

Inter-Process Communication (IPC) in Python [with Examples] | Apriorit (4)

This IPC method is rather simple, as the only parameter you need to work with is a unique string. You can use a universally unique identifier so that you don’t have to worry about port number collision.

However, BSD socket files only support a single channel of client–server connection and only in a synchronous way, which may harm your application’s performance.

Read also:
Practical Comparison of the Most Popular API Hooking Libraries: Microsoft Detours, EasyHook, Nektra Deviare, and Mhook

Named pipes

Note: This IPC method is platform-specific and is only suitable for Windows.

Named pipes are used by calling plain file operations to a file with a name that suits the \\.\pipe\<PIPE_NAME> format.
Since Windows wraps most of the low-level work inside file operations, in the Python client, you’ll need to use the open function as well as the file.write and methods:


import structimport time # IPC parametersPIPE_NAME = r'\\.\pipe\simple-ipc-pipe'ENCODING = 'ascii' with open(PIPE_NAME , 'rb+', buffering=0) as f: data = 'Hello, world'.encode(ENCODING) data_len = struct.pack('I', len(data))  f.write(data_len) f.write(data) # Necessary  time.sleep(1)  received_len = struct.unpack('I',[0] received_data = # Also necessary  print(f"Received data: {repr(received_data)}")

Here’s the code for the server-side of the named pipe-based IPC:


import structimport win32pipe, win32file, pywintypes # IPC parametersPIPE_NAME = r'\\.\pipe\simple-ipc-pipe'ENCODING = 'ascii' while True: pipe = win32pipe.CreateNamedPipe(PIPE_NAME, win32pipe.PIPE_ACCESS_DUPLEX, win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_READMODE_MESSAGE | win32pipe.PIPE_WAIT, 1, 65536, 65536, 0, None) try: print("waiting for client") win32pipe.ConnectNamedPipe(pipe, None) print("got client")  request_len = win32file.ReadFile(pipe, 4) request_len = struct.unpack('I', request_len[1])[0] request_data = win32file.ReadFile(pipe, request_len)  # convert to bytes response_data = "Response".encode(ENCODING) response_len = struct.pack('I', len(response_data)) win32file.WriteFile(pipe, response_len) win32file.WriteFile(pipe, response_data) finally: win32file.CloseHandle(pipe)

After running the code above, you will get the following output:

Inter-Process Communication (IPC) in Python [with Examples] | Apriorit (5)

Working with named pipes is just as simple as working with file operations. As with BSD socket files, you only need to work with one parameter, which is a unique string.

However, this IPC method also shares the same drawback as BSD socket files — you can only work with synchronous client–server connections established via a single channel.

Read also:
NFTs for Real Estate: Exploring the Concept and Creating Tokens in Python, JavaScript, and Solidity

Python’s standard socket module provides a way to use classic BSD-style socket objects in Python code. While most of the module’s capabilities are available across Unix, Windows, and macOS, some behavior may be platform-dependent, so make sure to check the documentation for details.

TCP sockets

Using Python’s socket module is the simplest method to configure cross-platform IPC. You can also easily scale it to inter-machine network communication.

To ensure the protection of transferred data, you can use Python’s standard SSL module that contains wrappers for TCP/IP sockets with SSL/TLS encryption.

Let’s take a look at an example of the client-side code for IPC via network sockets when all code runs on the same machine (i.e. using localhost as an address):


import socket # IPC parametersHOST = '' # The server's hostname or IP addressPORT = 65432 # The port used by the server # Create sockets = socket.socket(socket.AF_INET, socket.SOCK_STREAM)s.connect((HOST, PORT)) # Send 'request's.sendall(b'Hello, world') # Wait for 'response'data = s.recv(1024)print(f"Received data: {repr(data)}")

And here’s the server-side code for the same example:


import socket # IPC parametersHOST = '' # Standard loopback interface address (localhost)PORT = 65432 # Port to listen on (non-privileged ports are > 1023) # Create sockets = socket.socket(socket.AF_INET, socket.SOCK_STREAM)s.bind((HOST, PORT))s.listen(0) # Start listening loopwhile True: # Accept 'request' conn, addr = s.accept() print('Connected by', addr) # Process 'request' while True: data = conn.recv(1024) if not data: break # Send 'response' conn.sendall(data)

Now you can keep the server part running and call the client script to initiate data exchange:

Inter-Process Communication (IPC) in Python [with Examples] | Apriorit (6)

TCP-based IPC is a simple and, most importantly, cross-platform way to enable communication between processes and programs running on different platforms and different machines in the network.

However, this method is associated with great data security concerns and requires additional use of SSL/TLS modules to ensure proper data protection. Also, keep in mind that for this method to work, the port number parameter must be unique and synchronized between the server and client code.

Read also:
How to Accelerate Microservices Development: A Practical Guide to Applying Code Generation


Google Remote Procedure Calls (gRPC) is a vast framework for building RPC infrastructure with a focus on performance and support for various platforms and programming languages. As a communication tool, gRPC uses the protocol buffers that we described earlier.

Let’s see how gRPC-based IPC works in practice.

Step 1. Instal the gRPC module and tools package using the following command:


pip install grpcio grpcio-tools

Step 2. Describe a communication protocol using protocol buffers.

The following warehouse.proto file describes an RPC service that sends the CrateRequest structure from the client and returns the CrateResponse structure from the server:


// Definition of the service for getting information about crates inside a warehouseservice WarehouseLookup { // Send an ID and receive the data structure rpc GetCrateInfo(CrateRequest) returns (CrateResponse) {}} // Request structure containing the requested IDmessage CrateRequest { required int64 id = 1;} // Response structure containing the requested datamessage CrateResponse { required int64 width = 1; required int64 height = 2; required int64 length = 3; required int64 weight = 4; optional string label = 5;}

Step 3. Next, you can use gRPC tools to create language-specific files from the warehouse.proto file:


python -m grpc_tools.protoc --python_out=. --pyi_out=. --grpc_python_out=. warehouse.proto -I.

As a result, gRPC will generate two Python files:

  1. with the request and response structures
  2. with the client and server classes

Step 4. Using the client and server classes generated in Step 3, you can now enable communication between the entities.

Here’s what the code for this looks like on the client side:


import grpcimport randomimport warehouse_pb2import warehouse_pb2_grpc crate_id = random.randint(1,1000) print(f"Will try to get info on crate with id={crate_id}") with grpc.insecure_channel('localhost:50051') as channel: stub = warehouse_pb2_grpc.WarehouseLookupStub(channel) response = stub.GetCrateInfo(warehouse_pb2.CrateRequest(id=crate_id)) print("Received crate info: ")print(f" Dimensions: {response.width}x{response.height}x{response.length}")print(f" Weight: {response.weight}")if response.label: print(f" Label: {response.label}")else: print(f" Unlabeled crate")

And here’s the server-side code for this IPC implementation:


import grpcimport warehouse_pb2import warehouse_pb2_grpc from concurrent import futures  class WarehouseLookup(warehouse_pb2_grpc.WarehouseLookupServicer): def GetCrateInfo(self, request, context): # Read lookup on the server goes here # For illustrative purposes, the server will generate some random response import random print(f"Received request from client about crate with id={}") if % 2==0: # Create a response without the label field return warehouse_pb2.CrateResponse(width=random.randint(1,100), height=random.randint(1,100), length=random.randint(1,100), weight=random.randint(1,100)) else: # Create a response with the label field return warehouse_pb2.CrateResponse(width=random.randint(1,100), height=random.randint(1,100), length=random.randint(1,100), weight=random.randint(1,100), label=f"Some crate with id={}")  port = '50051'server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))warehouse_pb2_grpc.add_WarehouseLookupServicer_to_server(WarehouseLookup(), server)server.add_insecure_port('[::]:' + port)server.start()print("Server started, listening on " + port)server.wait_for_termination()

Now you can keep the server part running and call the client script to initiate data exchange:

Inter-Process Communication (IPC) in Python [with Examples] | Apriorit (7)

gRPC is comfortable to work with, as it supports multiple programming languages, allowing for seamless inter-process communication between Python and Java code, as well as with code written in C/C++ and other languages. However, working with this framework requires using the protobuf compiler for data synchronization, which adds to the complexity of the build process.

Another no-socket IPC option you may consider is the FastAPI framework. This framework allows for building Application Programming Interfaces (APIs) with Python based on standard Python 3.7+ type hints.

Related services

API Development and API Integration Services


Understanding what is inter-process communication in Python and how to work with it is key to building secure and efficient products. Programs and processes need to communicate with each other to effectively work with shared data.

Depending on which method is used for inter-process communication in Python when you build your application, you can improve the app’s performance and ensure secure data transfer.

With a wide selection of both standard and third-party IPC tools, Python enables you to work with a variety of IPC methods that can comply with your project’s specific requirements.

Apriorit developers have vast experience creating secure and high-performing applications with efficient IPC mechanisms in place. Whether your business needs a single-platform app or a complex cross-platform solution, our experts can help you deliver it.

Contact us to start improving data exchange processes in your product today!

Top Articles
Latest Posts
Article information

Author: Sen. Ignacio Ratke

Last Updated: 08/03/2023

Views: 6103

Rating: 4.6 / 5 (56 voted)

Reviews: 95% of readers found this page helpful

Author information

Name: Sen. Ignacio Ratke

Birthday: 1999-05-27

Address: Apt. 171 8116 Bailey Via, Roberthaven, GA 58289

Phone: +2585395768220

Job: Lead Liaison

Hobby: Lockpicking, LARPing, Lego building, Lapidary, Macrame, Book restoration, Bodybuilding

Introduction: My name is Sen. Ignacio Ratke, I am a adventurous, zealous, outstanding, agreeable, precious, excited, gifted person who loves writing and wants to share my knowledge and understanding with you.