CS644 week 8: Signals
Last week
Link: /week7
Review
- Concepts: Concurrency vs. parallelism, preemption vs. cooperation, global interpreter lock, locks
- Interfaces:
threading.Thread,threading.Thread.start,threading.Thread.join,threading.Lock,threading.Lock.acquire,threading.Lock.release
Solutions
- Thread experiments:
week7/thread_experiments.py
Signal basics
A signal is an asynchronous alert sent from one process to another (or from a process to itself). There is a fixed set of named signals, and they carry no data other than their signal number. The kernel sends signals to processes for various reasons:
SIGCHLDis sent to a parent process when a child process exits.SIGSEGVis sent to a process when it triggers a segmentation fault via invalid memory access (normally this signal is not caught).SIGILLis sent when the process attempts to execute an illegal processor instruction.
Other signals are meant to be sent between userspace processes:
SIGTERMrequests the target process to terminate.SIGKILLforcibly kills the target process.SIGWINCHalerts a terminal program that the window has been resized.
In a process, each signal has a corresponding disposition that determines what happens when the signal is received. The possibilities are:
- Ignore the signal.
- Perform the default action, which depends on the signal.
SIGTERMterminates the process, for instance, whileSIGWINCHdoes nothing. - Catch the signal and call a signal handler function.
Though the special signals SIGKILL and SIGSTOP cannot be ignored or caught.
The signals API
The function to send a signal is called kill (though despite its name, it can send any signal, not just SIGKILL):
# https://docs.python.org/3/library/os.html#os.kill
os.kill(pid: int, sig: int)
The symbolic constants for signals (SIGTERM, SIGKILL, etc.) are defined in the signal module. The signal module also contains signal.signal, by which a process can control the disposition of signals:
# https://docs.python.org/3/library/signal.html#signal.signal
signal.signal(signalnum: int, handler)
handler may be one of:
- The constant
signal.SIG_DFL - The constant
signal.SIG_IGN - A function taking two arguments, the signal number and the current stack frame (most signal handlers ignore the second argument)
For example, to print a message whenever SIGUSR1 is received, a program can execute:
import signal
def sighandler(_signo, _frame):
print("Signal received")
signal.signal(signal.SIGUSR1, sighandler)
Exercise: Signal experiments
- What happens if a signal is received while a signal handler is executing? Does it make a difference if it's the same signal or a different one?
- Set a signal handler and then spawn a few threads. When you send a signal, what thread executes the signal handler?
- What happens if a process is sent the same signal multiple times before it can handle it?
- What happens if a process that is sleeping (i.e., has called
time.sleep) receives a signal? Does the sleep terminate early? Reset? - What happens if a signal handler raises an exception? Can it be caught?
Signals in Python vs. C
A C program that has registered a signal handler must be prepared for the signal handler to interrupt the program at any time. Like preemptive multithreading, this is a form of concurrency, and like all forms of concurrency, it requires a great deal of care to use safely. Many C library functions are not signal safe, including such ubiquitous (and thread-safe) functions as printf and malloc.
The CPython interpreter is a C program, which means that it can be interrupted by a signal at any time. But it would be patently unsafe if your Python signal handler function ran inside the C signal handler – how can you ensure that your Python code doesn't result in a call to malloc? So the Python interpreter doesn't do that. Instead, when a signal is received, a flag is set, and the main interpreter loop checks the flag periodically when it is safe to invoke the Python signal handler.
So, the Python signal handler can call print and allocate memory, because the interpreter arranges for it to be called outside of the C signal handler.
Still, you should not take this as license to do whatever you want in the signal handler. You still need to worry about race conditions with the rest of your Python program. For instance, this is a bad idea:
lock = threading.Lock()
def sighandler(_signo, _frame):
# BAD!
with lock:
update_data_structure()
You might think that taking a lock protects you against race conditions. Quite the opposite: what happens if the signal is received while the main program is holding the lock? The signal handler tries to take the lock, blocks, and deadlocks the whole program. The same consideration applies to data structures that use locks under the hood, like queue.Queue.
Don't do real work inside a signal handler. Instead, set a flag, and let your main loop do the real work within the normal control flow of the program.
Signal masking
Passing SIG_IGN to signal.signal causes a signal to be ignored for the entire process. It is sometimes useful to instead have signals ignored temporarily, or for a single thread. This can be achieved by signal masking. When a signal is masked, the signal is still received, but it is left pending until the signal is unmasked.
pthread_sigmask is the function to control the signal mask in Python. It can selectively block (SIG_BLOCK) or unblock (SIG_UNBLOCK) a set of signals (mask is an actual Python set, like {signal.SIGUSR1, signal.SIGUSR2}), or to replace the signal mask wholesale (SIG_SETMASK).
# https://docs.python.org/3/library/signal.html#signal.pthread_sigmask
signal.pthread_sigmask(signal.SIG_BLOCK, mask)
signal.pthread_sigmask(signal.SIG_UNBLOCK, mask)
signal.pthread_sigmask(signal.SIG_SETMASK, mask)
Signal masks, combined with signal.sigwaitinfo and signal.sigtimedwait which wait for the arrival of a masked signal, can simplify the control flow of a program and avoid asynchronous signal handlers altogether:
sigset = {signal.SIGTERM}
signal.pthread_sigmask(signal.SIG_BLOCK, sigset)
while True:
# do work
if signal.sigtimedwait(sigset, 0) is not None:
break
Interrupting system calls
If a program is in the middle of a blocking system call when a signal with a registered signal handler is delivered, the kernel will terminate the system call, invoke the signal handler, and have the system call return the error code EINTR. (This behavior is the subject of a famous essay from the 1990s.)
The term "blocking" in this context is sometimes misunderstood. Syscalls that do I/O on regular files are not blocking, though they may take a while. Only syscalls that may take an indefinite amount of time are considered to be blocking. These include sleeping, waiting on a lock, waiting for network messages, and reading from a terminal device.
Conscientious code thus has to wrap any potentially blocking syscall with a loop that detects EINTR and retries the syscall. This is annoying, so sigaction, the C equivalent to Python's signal.signal function, has a flag, SA_RESTART, which tells the kernel to automatically restart any syscalls that were interrupted by a signal.
Python does this automatically (see PEP 475).
Exercise: Signal experiments, part 2
Exercise: Determine what happens to each of (a) signal dispositions, (b) signal masks, and (c) pending signals on fork and on exec (a total of 6 scenarios).
Safe signal idioms
Many programs can happily use the default behavior of signals. Most other programs only need to catch SIGTERM and shut down gracefully. This can be accomplished in a few ways:
- Set a flag in the signal handler and check it in the main loop.
- Use
pthread_sigmaskandsigtimedwaitdirectly in the main loop. - Use the self-pipe trick to notify the main loop from the signal handler.
Signals are tricky! If you find yourself doing anything more complicated than this, consider whether you should be using a proper IPC mechanism instead.
Final project milestone
Have your server handle SIGTERM by refusing to accept any new TCP connections, but continuing to serve existing connections until they are closed. If SIGTERM is sent twice, close existing connections as well and exit. Pick any one of the safe signal idioms to implement it.
Bonus material
- "Thread-Specific Data and Signal Handling in Multi-Threaded Applications" – old (from 1997) and uses an obsolete implementation of threading on Linux, but still a valuable post
- "Convert SIGTERM to an exception by default?"
- "The perils of pause(2)" by Julian Squires
- "signalfd is useless" by Geoffrey Thomas