SO_REUSEADDR
Dear Lazyweb,
I have a simple test application where a TCP/IP server listens for incoming connections, reads the data and closes the connection again and a client which opens connections to the server and sends a package and closes the connection as fast as it can:
The server looks like this:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.setblocking(False)
sock.bind(("", 12347))
sock.listen(1)
slist = [sock]
# use select to poll the sockets
while 1:
l = select.select(slist, [], [])
for i in l[0]:
conn, addr = i.accept()
data = “”
while 1:
tmp = conn.recv(1024)
if not tmp:
break
data += tmp
conn.shutdown(socket.SHUT_RDWR)
conn.close()
The Client:
# Open a connection, send data and close the connection as fast as possible
while 1:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(("", 12347))
sock.send("foo")
sock.shutdown(socket.SHUT_RDWR)
sock.close()
The Problem with this application: After roughly 25.000 Iterations the client quits with a friendly:
error: (99, ‘Cannot assign requested address’)
Netstat shows the problem: roughly 25.000 of these ones:
... tcp 0 0 localhost:56946 localhost:12347 TIME_WAIT tcp 0 0 localhost:47163 localhost:12347 TIME_WAIT tcp 0 0 localhost:42758 localhost:12347 TIME_WAIT ...
I’m not a TCP/IP expert but I thought SO_REUSEADDR is supposed to address this problem by allowing to reuse those as-good-as-closed connections in TIME_WAIT state, or not? So why does it fail in my test application?
July 22nd, 2009 at 11:18 am
I think you’re out of ephemeral ports. Your server will reuse port 12347 but your client will not reuse the source port until it’s completely free.
Source ports are generally chosen from the ephemeral port range (for Linux 32768 to 61000 according to Wikipedia), so about 25,000 connections seems plausible.
July 22nd, 2009 at 11:41 am
Looks like you want to be looking at SO_LINGER. See http://www.developerweb.net/forum/showthread.php?t=3752
and
http://www.developerweb.net/forum/showthread.php?t=2941
but most people think avoiding TIME_WAIT is a bad thing.
Maybe you want to use UDP?
July 22nd, 2009 at 12:58 pm
You may want to have a look at http://www.developerweb.net/forum/archive/index.php/t-2982.html - setting SO_LINGER will make sure the connection will be closed immediately and not hang around in the TIME_WAIT state.
July 22nd, 2009 at 1:41 pm
I don’t really grok network programming with C, but as far as I can see, you are not setting SO_REUSEADDR in the client, could this be part of the problem?
July 22nd, 2009 at 1:46 pm
Sven: No, setting this option on or off in the client seems to have no effect.
Maybe should simply leave the connection open and reuse it and don’t open a new one for every single message, but then of course I need to take care to find out when one message ends and another begins, oh well…
July 22nd, 2009 at 5:03 pm
You could consider using UDP, not TCP. As UDP has no connections to setup, it should be faster.
July 22nd, 2009 at 10:20 pm
SO_REUSEADDR only affects the server: it allows, among other things, multiple instances of the server to listen on the same port. This also helps when it is the server the one who closes the connection, then quits, and you want to start a new one. It does nothing to affect the client, and in fact, it is the client doing the closing.
I believe Kristof is right, and you’re running out of ephemeral ports on the client. See, when the client decides to terminate the connection, it sends a FIN. The server ACKs this packet, sends its remaining data (if any), and then sends a FIN itself (FIN+ACK if there’s no more data in the buffer). Your client will ACK the server’s FIN, and then waits a predefined amount of time in case this last ACK is lost in the wire, in which case it is resent.
Linux hardcodes this timeout to be of 60 seconds, and lacks a runtime setting to control this value. It provides, however, with another configuration parameter that can help you:
[Documentation/networking/ip-sysctl.txt]
tcp_tw_recycle - BOOLEAN
Enable fast recycling TIME-WAIT sockets. Default value is 0.
It should not be changed without advice/request of technical
experts.
So try `sysctl -w tcp_tw_recycle=1′.
July 22nd, 2009 at 10:27 pm
Err, I meant `sysctl -w net.ipv4.tcp_tw_recycle=1′.
July 23rd, 2009 at 2:19 am
To avoid connections staying in TIME_WAIT status, always close your connection at the client first before closing the connection at the server side.
November 22nd, 2009 at 10:28 pm
@wenlong That won’t help. Closing sockets will still lead to a TIME_WAIT status.
However, as mentioned earlier, SO_REUSEADDR will solve the problem.