CS644 week 9: Asynchronous I/O

Last week

Link: /week8

Review

Concepts: Signal disposition, signal masking, interrupting system calls
Interfaces: signal.signal, signal.pthread_sigmask, os.kill, signal.sigtimedwait

Solutions

Signal experiments: week8/signal_experiments.py
Signal experiments, part 2: week8/postfork.py and week8/postexec.py

The problem

This week's lecture will be a brief exploration of a common problem: What if you want to do lots of I/O at once?

Maybe you are writing a web server that wants to talk to many clients at once. Or an SSH client, which must read and write from the terminal (i.e., standard input and output) as well as from a network socket connected to the remote machine.

The root of the problem is that I/O operations can block: they can hang indefinitely while waiting for input to read, or in some cases for room to write output (e.g., a full pipe). And if you need to do I/O on multiple devices at once, you can't block on any one device because you don't know which one will be ready first.

This week, we will focus on network I/O and terminal I/O rather than disk I/O. Disk I/O is slow compared to other syscalls, or to regular function calls, but it will not block forever like network I/O or terminal I/O can. And with a single disk device, doing multiple I/O operations concurrently isn't necessarily going to be faster.

We've already learned one solution to this problem: multithreading. You can do separate I/O operations on different threads, and then each thread can block without interfering with the others.

But as we have seen, multithreading can greatly complicate your program. Fortunately, Linux has alternate solutions that do not require multiple threads.

`O_NONBLOCK`

open takes an O_NONBLOCK flag that causes subsequent read and write operations to not block. Instead, if the operation cannot be completed immediately, EAGAIN or EWOULDBLOCK will be returned (which results in a BlockingIOError being raised in Python). Sockets can be made non-blocking by passing SOCK_NONBLOCK to socket (e.g., socket.socket(socket.AF_INET, socket.SOCK_STREAM | socket.SOCK_NONBLOCK)).

So, you could write something like:

for fd in fds:
    try:
        b = read(fd, 4096)
    except BlockingIOError:
        pass
    else:
        # do something with data

The problem with this solution is that you are busy polling – wasting CPU time waiting for something to happen. It would be much better to be put to sleep until I/O is ready to be done.

`epoll`

epoll is an API to do exactly that: inform the kernel of the file descriptors you are interested in, and then go to sleep until one of them is ready. It's a newer version of the old poll and select syscalls.

import select

epoll = select.epoll()
epoll.register(fd1, select.EPOLLIN | select.EPOLLERR)
epoll.register(fd2, select.EPOLLIN | select.EPOLLERR)
while True:
    events = epoll.poll()
    for fd, event_type in events:
        if event_type & select.EPOLLERR:
            # handle error
        elif fd == fd1:
            # do something with fd1
        elif fd == fd2:
            # do something with fd2

The Python epoll is a thin object-oriented wrapper around the C system calls:

select.epoll uses epoll_create
epoll.register (and epoll.unregister) uses epoll_ctl
epoll.poll uses epoll_wait

Notice that epoll does not actually do any I/O for you. It just notifies you when I/O is ready to be done.

Conveniently, epoll.poll also takes a timeout parameter, so you can wait for only a finite amount of time.

`inotify` and calling C from Python

So far in this course, we have used interfaces that already exist in the Python standard library. This covers a large amount of the Linux system call interface, but not every system call has a builtin Python wrapper.

It's easy to create our own wrappers, using the standard library's ctypes module.

As a motivating example, let's look at inotify, the Linux subsystem for monitoring filesystem events. inotify lets you register files and directories of interest and receive notifications when various events occur, such as a file being modified, created, or renamed. There are many scenarios where inotify is useful:

A network server can monitor its config file and reload it whenever it changes.
A build process can monitor a directory of source-code files and start a rebuild whenever any of them changes.

Unfortunately, inotify does not have a standard library wrapper in Python. However, we can create one ourselves. Let's start with the function to create an inotify file descriptor, inotify_init1. This is its type signature in C:

int inotify_init1(int flags);

And this is how we use it in Python:

import ctypes

# Initialize libc.
libc = ctypes.CDLL(None, use_errno=True)
# Retrieve the name.
inotify_init1 = libc.inotify_init1
# Tell Python the argument and return types.
inotify_init1.argtypes = (ctypes.c_int,)
inotify_init1.restype = ctypes.c_int

# Ready to use!
fd = inotify_init1(0)

Exercise: Wrap `inotify` in Python

Write a Python wrapper for inotify_add_watch, the function that registers file descriptors of interest:

int inotify_add_watch(int fd, const char* pathname, uint32_t mask);

Take a look at the fundamental data types table for a mapping between C and Python types.

Here are a few of the constants that you can pass as the mask parameter (taken from here):

IN_MODIFY = 0x00000002
IN_CREATE = 0x00000100
IN_DELETE = 0x00000200

IN_NONBLOCK = 0o0004000

To test that it worked, call os.read on the file descriptor returned by inotify_init1. When an event occurs, os.read returns a C struct (as an array of bytes, e.g., { x = 1; y = 2} would be bytes([1, 2])).

Extra credit: Find the definition of the structured data that os.read returns from inotify(7), and use the struct module to decode it into a Python object.

Exercise: Combine `inotify` with `epoll`

Let's use inotify together with epoll. Write an epoll loop that listens to an inotify file descriptor. Make sure to pass IN_NONBLOCK to the call to inotify_init1.

The self-pipe trick

Last week, we briefly mentioned the "self-pipe trick", one of the idioms for handling signals safely. We are now equipped to understand how to use it.

On program start-up, create a pipe.
In your signal handler, write a byte (e.g., the signal number) to the pipe. (write is signal safe.)
In your epoll loop, listen to the read end of the pipe. When it's readable, handle the signal.

Exercise: Implement the self-pipe trick

Based on the description above, augment your inotify + epoll loop to listen for the signal SIGUSR1, using the self-pipe trick.

`io_uring`

io_uring originated as an asynchronous interface for I/O, although now it is closer to an "alternative system-call interface for Linux that is inherently asynchronous" as it can do many things other than I/O. (source) Along with eBPF, it's one of the most significant new developments in the Linux kernel in the past decade.

The basis of io_uring is two ring buffers, a submission queue and a completion queue, that are shared between userspace and the kernel. Your program adds entries to the submission queue and reads off results from the completion queue. Unlike with epoll, the kernel will actually do the syscall for you, not just notify you when it's ready. Because of this, and because you can batch multiple syscalls together (including in chains of execution), io_uring programs can make many fewer syscalls. In fact, you can enable a mode to have the kernel poll for entries in the submission queue, and not have to do any syscalls at all!

The Python standard library does not expose io_uring, but there is a third-party liburing package that you can use.

Final project milestone

Add a mode to your server which uses an epoll loop instead of multithreading to serve multiple TCP connections at once.

Bonus material

Notes on epoll and io_uring

Table of contents