Week 6: Networking

• Slides

Networking basics

Networking – communication between two or more computers over a wired or wireless link – is fundamental to modern computing. It is also a very big topic, so big that we cannot hope to cover much of it in one week. So, we will focus on one specific topic – how to use the Linux sockets API.

For two computers to communicate successfully with each other, they must agree on a protocol. This is not so different from two different programs on the same computer, or even two different parts of the same program, which have to agree on the order of arguments, whether they are placed on the stack or in registers, etc. But network protocols have to contend with many more sources of complexity, notably, the possibility (indeed, the likelihood) that the connection between the two computers is unreliable, and the possibility that the two computers are running different operating systems or even different CPU architectures. Consequently, network protocols are complicated.

In fact, when your computer talks to another computer, there is not just one network protocol at play. A 2000s-era connection between your desktop computer and a web server might look something like:

• The desktop communicates with your home router via a wired Ethernet connection.
• Inside the Ethernet packets are IPv4 packets that go all the way across the Internet backbone to the web server.
• The IPv4 packets are themselves used to carry a stream of TCP data.
• The TCP data encodes an HTTP/1.1 request to fetch a resource from the web server.

This layering of protocols is called the protocol stack. Each layer is responsible for a specific concern:

• Ethernet handles the physical transmission of data across a fixed link between a sender and a receiver. (the link layer)
• The Internet Protocol (IP) routes data from one point to another point across an entire network, where the sender and receiver may be many hops away from each other. (the internet layer)
• The Transmission Control Protocol (TCP) handles congestion control and packet loss, and presents an abstraction of a stream of data rather than a sequence of discrete packets. (the transport layer)
• The Hypertext Transfer Protocol (HTTP) is an application-specific protocol for transferring hypertext documents such as Web pages (and many other things). (the application layer)

High-level protocols like HTTP can be simple (indeed, it is quite easy to write a basic HTTP/1.1 client and server) because the low-level protocols hide the details of how packets make their way across our gigantic, planet-scale integrated network.

Networking in Linux: TCP and UDP

That was a whirlwind tour of the network stack. For the purposes of this class, we are going to hone in on two protocols: TCP, and its sister protocol UDP. Everything lower-level than those are typically handled inside of the kernel and its device drivers, while everything higher-level is built in userspace on top of TCP and UDP. But TCP and UDP are the basic interfaces that the kernel provides to user programs.

We've already briefly mentioned TCP: it's a reliable, connection-oriented, byte-stream protocol, built on top of IP. UDP, the User Datagram Protocol, is likewise atop IP, but it is instead a unreliable, connectionless, datagram protocol.

• Reliable vs. unreliable: Does the protocol keep track of sent and received packets and retransmit any that were dropped (TCP), or are dropped packets simply lost (UDP)?
• Connection-oriented vs. connectionless: Does the protocol maintain a persistent connection between the two hosts (TCP), or not (UDP)?
• Stream vs. datagram: Do users of the protocol read and write streams of data without message boundaries (TCP), or are there discrete messages (UDP)?

TCP is ideal when you want to guarantee that data is transmitted reliable (think HTTP or SSH) and you are willing to sacrifice a little bit of performance (because of, e.g., head-of-line blocking from packet retransmission). UDP may be more appropriate if it is acceptable to sometimes drop packets (e.g., streaming video or audio), or if you want to build your own application-specific "reliable transmission" semantics.

Addresses, domain names, and ports

A host on an IP network is identified by an IP address. In IPv4, addresses are 32 bits and are written like 167.71.190.147, with each decimal number representing 4 bytes. 32 bits only allows for 4 billion unique addresses, which is not enough for the modern Internet, so IPv6 expanded addresses to 128 bits. IPv6 addresses look like 2001:db8::8a2e:370:7334.

As an Internet user, you almost always interact with human-readable domain names, like iafisher.com, rather than raw IP addresses. A network protocol called DNS lets you dynamically map a domain name to the underlying IP address.

An IP address identifies a host on a network, but a single host may offer many networked services. For instance, a web server may accept both HTTP connections from its users, and SSH connections from its administrators. TCP and UDP introduce the concept of a port, an integer that identifies a particular service at an address. There are conventional ports for different services, for instance port 22 for SSH, port 80 for HTTP, and port 443 for HTTPS, but nothing stops you from using a different port, as long as the client and server agree.

The sockets API

It's finally time to talk about the actual code interfaces we use to perform networking on Linux.

It takes quite a few steps to set up a network connection. Let's take the example of a TCP connection. On the server side:

1. getaddrinfo to create an address (e.g., IP address and port). This is the function that will do the DNS look-up.
2. socket to get a file descriptor.
3. bind to bind the file descriptor to the address.
4. listen to start listening for connections.
5. Call accept in a loop to accept a single incoming connection, as a new file descriptor.
6. Use send and recv (or write and read) on the new file descriptor.
7. shutdown to close the connection.

And on the client side:

1. getaddrinfo to create an address.
2. socket to get a file descriptor.
3. connect to establish a connection to the server.
4. send and recv, as with the server.

hello_conn_tcp.c has an example of both sides of the connection.

The first argument to socket is the "domain":

• AF_INET means IPv4.
• AF_INET6 means IPv6.

The second argument to socket is the type:

• SOCK_STREAM for reliable data streams, i.e., TCP.
• SOCK_DATAGRAM for unreliable data packets, i.e., UDP.

Because the socket API returns file descriptors, we can use regular file APIs like read and write to work with them, though not all APIs (e.g., lseek) make sense for sockets. But the socket-specific APIs send and recv expose some extra options.

For UDP, the server skips listen and accept and just calls recv in a loop, while the client skips connect.

Surprise: IPC!

It turns out that the socket API can also be used as an IPC mechanism. Pass AF_UNIX as the first argument to socket, and instead of TCP or UDP, the socket will be a Unix domain socket that does local data transfer between programs on the same computer. It's like a named pipe, except that it's duplex (data can flow in both directions), and the server can have multiple clients.

Note that this is subtly different from binding to localhost (which is a domain name that points to 127.0.0.1, the "loopback" IP address) – in that case, the client and server will write network packets to the kernel's loopback interface, while in the case of Unix domain sockets the kernel is just copying bytes and there's no networking involved at all.

Final project milestone

Let's provide a proper client interface to the database. Have the main process listen for connections (you can decide whether you want to do TCP or Unix domain sockets) and allow querying the database. You can decide what the protocol looks like; a simple one might have commands like get <key>\n and set <key> <value>\n. Write a client program that provides a nice command-line interface to send commands to the database.

Homework exercises

Note: The socket API may be exposed in a different library in your programming language, e.g. in Python it's in socket, not os.

1. (★) What function does a DNS lookup to turn a domain name into an IP address?
   Solution

   The function is getaddrinfo. This is a standard library function, not a system call: the kernel does not implement DNS.
2. (★) What's the difference between bind and listen?
   Solution

   bind associates a socket file descriptor with an address and port, while listen starts actively listening for new connections on that socket.
3. (★) What flags are used to request a TCP/IP connection?
   Solution

   AF_INET or AF_INET6 for the first argument, and SOCK_STREAM for the second argument.
4. (★★) How can a server access its client's network address?
   Solution

   When a connection is initiated, accept fills in the second argument with the client address.
5. (★★) What happens if a client calls connect before the server has called bind? What about after the server has called bind, but before it has called listen? Write a program to find out.
   Solution

   connect_bind.c shows that connect returns an error (ECONNREFUSED) if the server has not yet called bind. With slight modification, the same program shows that connect will return the same error if the server has called bind but not listen.
6. (★★) If I open a socket with SOCK_DGRAM, will recv always return a single packet at a time? What happens if the buffer is too small to fit the packet? Write a program to demonstrate what happens.
   Solution

   recv.c forks a client and a server and has the client send a few packets holding one character each. Once the server wakes up, it reads the characters one at a time, despite supplying a bigger buffer to recv. Finally the client sends a message that is too big for the server's buffer, and the message received by the server is truncated. man 2 recv says: "If a message is too long to fit in the supplied buffer, excess bytes may be discarded depending on the type of socket the message is received from." (The example uses a UDP socket.)
7. (★★) What's the difference between close and shutdown? Can you write a program that shows them behaving differently?
   Solution

   close is a local file descriptor operation, while shutdown shuts down the whole connection. close_shutdown.c shows that if you call dup and then close on one of the file descriptors, you can still use the other one. But if you call shutdown, then the connection is closed and neither file descriptor works.
8. (★★★) You can pass an open file descriptor from one process to another via a socket. What syscall allows you to do this? When might this technique be useful?
   Solution

   You can transfer an open file descriptor using the msghdr parameter of the sendmsg syscall. See Section 17.4 of Advanced Programming in the Unix Environment for details. You could use this technique to implement an "open server", a server program that implements access controls beyond what is possible in the classic Unix permissions model. For instance, it could enforce that a client can only open a file in append mode.

Further reading

• Beej's Guide to Network Programming – Goes into a lot more depth on how to use the networking APIs. Highly recommend!