Week 5: Interprocess communication

• Slides

Last week we learned how to create new processes. But, aside from waitpid, the processes couldn't interact with each other. This week, we'll learn some techniques for communicating between different processes – interprocess communication, or IPC. We'll cover pipes and shared memory with synchronization via semaphores. Later in the course, we'll also talk about Unix domain sockets and signals.

Pipes

A pipe is a one-way data stream. The writer writes to one end and the reader reads from the other end (and there can be multiple writers and readers, though one of each is the typical case).

To create a pipe, call pipe2:

int pipe2(int pipefd[2], int flags);

And you get back a file descriptor for the read end in pipefd[0], and a file descriptor for the write end in pipefd[1]. Then you can use read and write as if it were an actual file.

You can use both ends of the pipe in the same process, and occasionally that is useful. But normally you want one end of the pipe in one process and the other end in another. Solution: call pipe2, then fork:

int pipefd[2];
int r = pipe2(pipefd, 0);
if (r < 0) { bail("pipe2"); }

pid_t pid = fork();
if (pid < 0) {
    bail("fork");
} else if (pid == 0) {
    // child
    close(pipefd[0]);
    // write to pipefd[1]
} else {
    // parent
    close(pipefd[1]);
    // read from pipefd[0]
}

When you fork, you end up with both file descriptors open in both processes. Each process only needs one (the read end or the write end), so it can close the one it doesn't need. File descriptors are per-process, so closing one in the child process does not close it in the parent process, or vice versa.

An important limitation of pipes is that they require the two processes to be related to each other, e.g. parent and child. There is a similar primitive called FIFOs (for "first-in, first-out"; also known as named pipes) which function the same as pipes except that they have an entry in the filesystem so that unrelated processes can discover and use them. See the homework exercises for more.

Shared memory

Shared memory is an efficient form of IPC that avoids copying by mapping the same memory region into multiple processes' address spaces. This lets us transparently share in-memory data structures between unrelated processes, just as if they were threads of the same process. Of course, we'll need some way to synchronize access. For that, we can use a kernel synchronization primitive called named semaphores.

Creating a shared memory region is a three-step process. First, we need to open the shared memory object itself with shm_open, which looks very similar to open:

int fd = shm_open("/my-program-mem", O_CREAT | O_EXCL | O_RDWR, 0600);

The name is not a file path, as shared memory objects have their own namespace (though it may be mounted at some special place in the filesystem such as /dev/shm). The name should always begin with a slash and contain no other slashes.

The shared memory object starts out empty, so we must resize it:

int r = ftruncate(fd, sizeof my_data_structure);

Finally, we can map it into our process's address space:

struct my_data_structure* s = mmap(
    NULL, sizeof my_data_structure, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0,
);
if (s == MAP_FAILED) {
    // handle error
}

Now, we can use s normally, and any writes will be visible to any other process using it (and we will likewise see any other process's writes).

Shared memory and threading

We'll cover multithreading in depth in a later week. For now, just know that a process can have multiple threads of execution, each running independently but sharing the same address space and memory. Which raises the question, why would you use multiple processes and shared memory, when you could have multiple threads in the same process? One reason is to separate concerns between different programs. For instance, you might have a high-performance network application that writes metrics to a shared memory region, and a metrics archiver that reads the same memory region and persists the metrics to disk.

Semaphores

Shared memory lets us share data structures between processes, but we still need a way to synchronize access in the case of multiple writers and readers. For that, we use semaphores:

sem_t* sem = sem_open("/my-program-sem", O_CREAT | O_EXCL, 0600, 1);

Similar to shm_open, the name passed to sem_open is not a file path. The fourth argument is the initial value of the semaphore – setting it to 1 makes the semaphore effectively a lock that only one process can hold at a time.

We can then wait for the semaphore to be available:

int r = sem_wait(&sem);

And when we are done, release it:

int r = sem_post(&sem);

Finally, at the end of our program we should clean everything up:

close(fd);
shm_unlink("/my-program-mem");
sem_unlink("/my-program-sem");

Putting it all together:

const char* mem_pathname = "/my-program-mem";
const char* sem_pathname = "/my-program-sem";

void writer() {
    // NOTE on error handling: At various points we bail without cleaning up, e.g.
    // calling `shm_unlink`. A more robust program should still clean up resources
    // even in case of error.

    int fd = shm_open(mem_pathname, O_CREAT | O_EXCL | O_RDWR, 0600);
    if (fd < 0) { bail("shm_open"); }
    int r = ftruncate(fd, sizeof my_data_structure);
    if (r < 0) { bail("ftruncate"); }

    struct my_data_structure* s;
    s = mmap(NULL, sizeof my_data_structure, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if (s == MAP_FAILED) { bail("mmap"); }

    // After we `mmap`, we can close the shared-memory file descriptor.
    r = close(fd);
    if (r < 0) { bail("close"); }

    sem_t* sem = sem_open(sem_pathname, O_CREAT | O_EXCL, 0600, 1);
    if (sem == SEM_FAILED) { bail("sem_open"); }

    r = sem_wait(&sem);
    if (r < 0) { bail("sem_wait"); }
    // ... use shared data structure ...
    r = sem_post(&sem);
    if (r < 0) { bail("sem_post"); }

    r = sem_unlink(sem_pathname)
    if (r < 0) { bail("sem_unlink"); }

    r = shm_unlink(mem_pathname);
    if (r < 0) { bail("shm_unlink"); }
}

void reader() {
    int fd = shm_open(pathname, O_RDONLY);
    if (fd < 0) { bail("shm_open"); }

    struct my_data_structure* s;
    s = mmap(NULL, sizeof my_data_structure, PROT_READ, MAP_SHARED, fd, 0);
    if (s == MAP_FAILED) { bail("mmap"); }

    // After we `mmap`, we can close the shared-memory file descriptor.
    int r = close(fd);
    if (r < 0) { bail("close"); }

    sem_t* sem = sem_open(sem_pathname, 0);
    if (sem == SEM_FAILED) { bail("sem_open"); }

    r = sem_wait(&sem);
    if (r < 0) { bail("sem_wait"); }
    // ... use shared data structure ...
    r = sem_post(&sem);
    if (r < 0) { bail("sem_post"); }
}

Final project milestone

No milestone this week. Take some time to catch up and work on the homework exercises!

Homework exercises

Note: Not all languages expose shared memory and semaphore primitives directly, so you may need to use a third-party library. For Python, check out posix_ipc in conjunction with the standard library mmap module.

1. (★) List the key differences between pipes and shared memory as forms of IPC.
   Solution

   Pipes can only be used between related processes (parent and child), while shared memory can be used by unrelated processes. Pipes let you pass a stream of data in one direction, while shared memory lets you share a fixed-size chunk of memory bidirectionally. Shared memory generally requires synchronization for safe access, while pipes do not (unless the same pipe has multiple writers).
2. (★) What format should be used for the names of shared memory objects and semaphores?
   Solution

   They should begin with a slash and contain no other slashes, e.g. /my-shared-memory-object. Each has their own namespace which is separate from the filesystem, so don't worry about name collision with files.
3. (★) Explain how two different processes end up with opposite ends of a pipe.
   Solution

   When pipe2 is called, it returns the two ends of the pipe to a single process. That process can then call fork, after which the parent and child processes will each have both ends of the pipe. The parent and child can then close the end they don't need.
4. (★★) Do we get any atomicity guarantees when working with pipes? Read man 7 pipe to find out.
   Solution

   Writes up to the value PIPE_BUF (4096 bytes on Linux) are guaranteed to be atomic, i.e. not interleaved with writes from other writers. You can dynamically retrieve the value of PIPE_BUF with fpathconf for portability.
5. (★★) What could go wrong if we don't use semaphores to synchronize access to shared memory? Write an example program to demonstrate the problem.
   Solution

   See shm_example.c from in class. Without synchronized access, you could read partially updated states or corrupt the data structure with multiple simultaneous writes.
6. (★★) We briefly mentioned another IPC primitive called FIFOs (first-in, first-out) that overcome some of the limitations of pipes. Research them (man 7 pipe and man 7 fifo may help). What system calls do they use? What are the differences from regular pipes?
   Solution

   FIFOs are very similar to pipes, except they have an entry on the filesystem so that unrelated processes can use them. They are thus often called named pipes. You use mkfifo (technically this is a C library function, and the actual syscall is mknod) to create the FIFO and open to open it – after that, they behave just like unnamed pipes. Note that open will block until the other end is opened by another process.
7. (★★) What is the buffering behavior of pipes? Pose a hypothesis, then write a test program to find out.
   Solution

   Plausible hypotheses: (a) no buffering, writes will block until the child reads, (b) the kernel buffers up to N bytes, then writes block, or (c) the kernel will buffer as much as you write to it. pipebuffer.c shows that (b) is correct. On our shared server, the buffer has a size of 65,536 bytes.
8. (★★) We used mmap for shared memory, but the system call is more versatile than just that. Read the man page and find out what else it can be used for.
   Solution

   Another handy use of mmap is to map a file into your address space so you can access it like an array, which is sometimes easier to work with than read and write.
9. (★★★) Shared memory is often used for high-performance concurrent applications. Implement a ring buffer in shared memory with one producer process putting work into the buffer, and multiple consumer processes reading from it.
   Solution

   https://github.com/iafisher/cs644/blob/master/week5/solutions/ringbuffer.c
10. (★★★) High-level languages have a way for a parent process to capture its child's stdout and stderr (e.g., capture_output=True in subprocess.run in Python). Use pipe2, fork, and execve to spawn an external program, e.g., echo hello world, and capture its output in a string in the parent program. Bonus: How can you use the same technique to implement shell pipelines?
    Hint

    You'll need to use dup2.
    Solution

    See pyfork.py for a solution in Python. The trick is to create a pipe before calling fork, then use dup2 in the child process to replace stdout with the write end of the pipe. The parent can then read the child's output from the other end of the pipe. Shell pipelines are similar: you make the first program's standard output the write end of a pipe, whose read end is the second program's standard input.