“Too paranoid for you?”

CS644 week 8: Signals

Last week

Link: /week7

Review

Solutions

Signal basics

A signal is an asynchronous alert sent from one process to another (or from a process to itself). There is a fixed set of named signals, and they carry no data other than their signal number. The kernel sends signals to processes for various reasons:

Other signals are meant to be sent between userspace processes:

In a process, each signal has a corresponding disposition that determines what happens when the signal is received. The possibilities are:

Though the special signals SIGKILL and SIGSTOP cannot be ignored or caught.

The signals API

The function to send a signal is called kill (though despite its name, it can send any signal, not just SIGKILL):

# https://docs.python.org/3/library/os.html#os.kill
os.kill(pid: int, sig: int)

The symbolic constants for signals (SIGTERM, SIGKILL, etc.) are defined in the signal module. The signal module also contains signal.signal, by which a process can control the disposition of signals:

# https://docs.python.org/3/library/signal.html#signal.signal
signal.signal(signalnum: int, handler)

handler may be one of:

For example, to print a message whenever SIGUSR1 is received, a program can execute:

import signal

def sighandler(_signo, _frame):
    print("Signal received")

signal.signal(signal.SIGUSR1, sighandler)

Exercise: Signal experiments

Signals in Python vs. C

A C program that has registered a signal handler must be prepared for the signal handler to interrupt the program at any time. Like preemptive multithreading, this is a form of concurrency, and like all forms of concurrency, it requires a great deal of care to use safely. Many C library functions are not signal safe, including such ubiquitous (and thread-safe) functions as printf and malloc.

The CPython interpreter is a C program, which means that it can be interrupted by a signal at any time. But it would be patently unsafe if your Python signal handler function ran inside the C signal handler – how can you ensure that your Python code doesn't result in a call to malloc? So the Python interpreter doesn't do that. Instead, when a signal is received, a flag is set, and the main interpreter loop checks the flag periodically when it is safe to invoke the Python signal handler.

So, the Python signal handler can call print and allocate memory, because the interpreter arranges for it to be called outside of the C signal handler.

Still, you should not take this as license to do whatever you want in the signal handler. You still need to worry about race conditions with the rest of your Python program. For instance, this is a bad idea:

lock = threading.Lock()

def sighandler(_signo, _frame):
    # BAD!
    with lock:
        update_data_structure()

You might think that taking a lock protects you against race conditions. Quite the opposite: what happens if the signal is received while the main program is holding the lock? The signal handler tries to take the lock, blocks, and deadlocks the whole program. The same consideration applies to data structures that use locks under the hood, like queue.Queue.

Don't do real work inside a signal handler. Instead, set a flag, and let your main loop do the real work within the normal control flow of the program.

Signal masking

Passing SIG_IGN to signal.signal causes a signal to be ignored for the entire process. It is sometimes useful to instead have signals ignored temporarily, or for a single thread. This can be achieved by signal masking. When a signal is masked, the signal is still received, but it is left pending until the signal is unmasked.

pthread_sigmask is the function to control the signal mask in Python. It can selectively block (SIG_BLOCK) or unblock (SIG_UNBLOCK) a set of signals (mask is an actual Python set, like {signal.SIGUSR1, signal.SIGUSR2}), or to replace the signal mask wholesale (SIG_SETMASK).

# https://docs.python.org/3/library/signal.html#signal.pthread_sigmask
signal.pthread_sigmask(signal.SIG_BLOCK, mask)
signal.pthread_sigmask(signal.SIG_UNBLOCK, mask)
signal.pthread_sigmask(signal.SIG_SETMASK, mask)

Signal masks, combined with signal.sigwaitinfo and signal.sigtimedwait which wait for the arrival of a masked signal, can simplify the control flow of a program and avoid asynchronous signal handlers altogether:

sigset = {signal.SIGTERM}
signal.pthread_sigmask(signal.SIG_BLOCK, sigset)

while True:
    # do work

    if signal.sigtimedwait(sigset, 0) is not None:
        break

Interrupting system calls

If a program is in the middle of a blocking system call when a signal with a registered signal handler is delivered, the kernel will terminate the system call, invoke the signal handler, and have the system call return the error code EINTR. (This behavior is the subject of a famous essay from the 1990s.)

Blocking syscalls

The term "blocking" in this context is sometimes misunderstood. Syscalls that do I/O on regular files are not blocking, though they may take a while. Only syscalls that may take an indefinite amount of time are considered to be blocking. These include sleeping, waiting on a lock, waiting for network messages, and reading from a terminal device.

Conscientious code thus has to wrap any potentially blocking syscall with a loop that detects EINTR and retries the syscall. This is annoying, so sigaction, the C equivalent to Python's signal.signal function, has a flag, SA_RESTART, which tells the kernel to automatically restart any syscalls that were interrupted by a signal.

Python does this automatically (see PEP 475).

Exercise: Signal experiments, part 2

Exercise: Determine what happens to each of (a) signal dispositions, (b) signal masks, and (c) pending signals on fork and on exec (a total of 6 scenarios).

Safe signal idioms

Many programs can happily use the default behavior of signals. Most other programs only need to catch SIGTERM and shut down gracefully. This can be accomplished in a few ways:

Signals are tricky! If you find yourself doing anything more complicated than this, consider whether you should be using a proper IPC mechanism instead.

Final project milestone

Have your server handle SIGTERM by refusing to accept any new TCP connections, but continuing to serve existing connections until they are closed. If SIGTERM is sent twice, close existing connections as well and exit. Pick any one of the safe signal idioms to implement it.

Bonus material