Week 5: Interprocess communication

• Slides

Last week we learned how to create new processes. But, aside from waitpid, the processes couldn't interact with each other. This week, we'll learn some techniques for communicating between different processes – interprocess communication, or IPC. We'll cover pipes and shared memory with synchronization via semaphores. Later in the course, we'll also talk about Unix domain sockets and signals.

Pipes

A pipe is a one-way data stream. The writer writes to one end and the reader reads from the other end (and there can be multiple writers and readers, though one of each is the typical case).

To create a pipe, call pipe2:

int pipe2(int pipefd[2], int flags);

And you get back a file descriptor for the read end in pipefd[0], and a file descriptor for the write end in pipefd[1]. Then you can use read and write as if it were an actual file.

You can use both ends of the pipe in the same process, and occasionally that is useful. But normally you want one end of the pipe in one process and the other end in another. Solution: call pipe2, then fork:

int pipefd[2];
int r = pipe2(pipefd, 0);
if (r < 0) { bail("pipe2"); }

pid_t pid = fork();
if (pid < 0) {
    bail("fork");
} else if (pid == 0) {
    // child
    close(pipefd[0]);
    // write to pipefd[1]
} else {
    // parent
    close(pipefd[1]);
    // read from pipefd[0]
}

When you fork, you end up with both file descriptors open in both processes. Each process only needs one (the read end or the write end), so it can close the one it doesn't need. File descriptors are per-process, so closing one in the child process does not close it in the parent process, or vice versa.

An important limitation of pipes is that they require the two processes to be related to each other, e.g. parent and child. There is a similar primitive called FIFOs (for "first-in, first-out"; also known as named pipes) which function the same as pipes except that they have an entry in the filesystem so that unrelated processes can discover and use them. See the homework exercises for more.

Shared memory

Shared memory is an efficient form of IPC that avoids copying by mapping the same memory region into multiple processes' address spaces. This lets us transparently share in-memory data structures between unrelated processes, just as if they were threads of the same process. Of course, we'll need some way to synchronize access. For that, we can use a kernel synchronization primitive called named semaphores.

Creating a shared memory region is a three-step process. First, we need to open the shared memory object itself with shm_open, which looks very similar to open:

int fd = shm_open("/my-program-mem", O_CREAT | O_EXCL | O_RDWR, 0600);

The name is not a file path, as shared memory objects have their own namespace (though it may be mounted at some special place in the filesystem such as /dev/shm). The name should always begin with a slash and contain no other slashes.

The shared memory object starts out empty, so we must resize it:

int r = ftruncate(fd, sizeof my_data_structure);

Finally, we can map it into our process's address space:

struct my_data_structure* s = mmap(
    NULL, sizeof my_data_structure, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0,
);
if (s == MAP_FAILED) {
    // handle error
}

Now, we can use s normally, and any writes will be visible to any other process using it (and we will likewise see any other process's writes).

Shared memory and threading

We'll cover multithreading in depth in a later week. For now, just know that a process can have multiple threads of execution, each running independently but sharing the same address space and memory. Which raises the question, why would you use multiple processes and shared memory, when you could have multiple threads in the same process? One reason is to separate concerns between different programs. For instance, you might have a high-performance network application that writes metrics to a shared memory region, and a metrics archiver that reads the same memory region and persists the metrics to disk.

Semaphores

Shared memory lets us share data structures between processes, but we still need a way to synchronize access in the case of multiple writers and readers. For that, we use semaphores:

sem_t* sem = sem_open("/my-program-sem", O_CREAT | O_EXCL, 0600, 1);

Similar to shm_open, the name passed to sem_open is not a file path. The fourth argument is the initial value of the semaphore – setting it to 1 makes the semaphore effectively a lock that only one process can hold at a time.

We can then wait for the semaphore to be available:

int r = sem_wait(&sem);

And when we are done, release it:

int r = sem_post(&sem);

Finally, at the end of our program we should clean everything up:

close(fd);
shm_unlink("/my-program-mem");
sem_unlink("/my-program-sem");

Putting it all together:

const char* mem_pathname = "/my-program-mem";
const char* sem_pathname = "/my-program-sem";

void writer() {
    // NOTE on error handling: At various points we bail without cleaning up, e.g.
    // calling `shm_unlink`. A more robust program should still clean up resources
    // even in case of error.

    int fd = shm_open(mem_pathname, O_CREAT | O_EXCL | O_RDWR, 0600);
    if (fd < 0) { bail("shm_open"); }
    int r = ftruncate(fd, sizeof my_data_structure);
    if (r < 0) { bail("ftruncate"); }

    struct my_data_structure* s;
    s = mmap(NULL, sizeof my_data_structure, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if (s == MAP_FAILED) { bail("mmap"); }

    // After we `mmap`, we can close the shared-memory file descriptor.
    r = close(fd);
    if (r < 0) { bail("close"); }

    sem_t* sem = sem_open(sem_pathname, O_CREAT | O_EXCL, 0600, 1);
    if (sem == SEM_FAILED) { bail("sem_open"); }

    r = sem_wait(&sem);
    if (r < 0) { bail("sem_wait"); }
    // ... use shared data structure ...
    r = sem_post(&sem);
    if (r < 0) { bail("sem_post"); }

    r = sem_unlink(sem_pathname)
    if (r < 0) { bail("sem_unlink"); }

    r = shm_unlink(mem_pathname);
    if (r < 0) { bail("shm_unlink"); }
}

void reader() {
    int fd = shm_open(pathname, O_RDONLY);
    if (fd < 0) { bail("shm_open"); }

    struct my_data_structure* s;
    s = mmap(NULL, sizeof my_data_structure, PROT_READ, MAP_SHARED, fd, 0);
    if (s == MAP_FAILED) { bail("mmap"); }

    // After we `mmap`, we can close the shared-memory file descriptor.
    int r = close(fd);
    if (r < 0) { bail("close"); }

    sem_t* sem = sem_open(sem_pathname, 0);
    if (sem == SEM_FAILED) { bail("sem_open"); }

    r = sem_wait(&sem);
    if (r < 0) { bail("sem_wait"); }
    // ... use shared data structure ...
    r = sem_post(&sem);
    if (r < 0) { bail("sem_post"); }
}

Final project milestone

No milestone this week. Take some time to catch up and work on the homework exercises!

Homework exercises

Note: Not all languages expose shared memory and semaphore primitives directly, so you may need to use a third-party library. For Python, check out posix_ipc in conjunction with the standard library mmap module.

1. (★) List the key differences between pipes and shared memory as forms of IPC.
2. (★) What format should be used for the names of shared memory objects and semaphores?
3. (★) Explain how two different processes end up with opposite ends of a pipe.
4. (★★) Do we get any atomicity guarantees when working with pipes? Read man 7 pipe to find out.
5. (★★) What could go wrong if we don't use semaphores to synchronize access to shared memory? Write an example program to demonstrate the problem.
6. (★★) We briefly mentioned another IPC primitive called FIFOs (first-in, first-out) that overcome some of the limitations of pipes. Research them (man 7 pipe and man 7 fifo may help). What system calls do they use? What are the differences from regular pipes?
7. (★★) What is the buffering behavior of pipes? Pose a hypothesis, then write a test program to find out.
8. (★★) We used mmap for shared memory, but the system call is more versatile than just that. Read the man page and find out what else it can be used for.
9. (★★★) Shared memory is often used for high-performance concurrent applications. Implement a ring buffer in shared memory with one producer process putting work into the buffer, and multiple consumer processes reading from it.
10. (★★★) High-level languages have a way for a parent process to capture its child's stdout and stderr (e.g., capture_output=True in subprocess.run in Python). Use pipe2, fork, and execve to spawn an external program, e.g., echo hello world, and capture its output in a string in the parent program. Bonus: How can you use the same technique to implement shell pipelines?
    Hint

    You'll need to use dup2.