Rare State 2
Circumstance
Server listen on port with default backlog.
Client initial a large number of connect at the same time.
Demo
sudo netstat -nap | grep -w '5555' | awk '{print $6" "$7}' | sort | uniq -c
1 LISTEN 20310/./Server
125 ESTABLISHED -
27 ESTABLISHED 20310/./Server
69 SYN_RECV -
221 ESTABLISHED 59253/./ClientSend
79 SYN_SENT 59253/./ClientSend
sudo netstat -nap | grep -w '5555' | awk '{print $6" "$7}' | sort | uniq -c
54 ESTABLISHED -
138 ESTABLISHED 20310/./Server
300 ESTABLISHED 59253/./ClientSend
1 LISTEN 20310/./Server
State
After a massive current connect (with a relatively small backlog)
[ESTABLISH (Client)| NULL (Server)]()
Fix
Solutions
Disable SYN cookies
For security reason, we don’t want to do this.Increase the backlog size of listen socket.
/proc/sys/net/ipv4/tcp_max_syn_backlog, default 2048, change to 8192
/proc/sys/net/core/somaxconn, default 128, change to 4096
sudo tail -f /var/log/messages
Backlog
man listen
backlog … it specifies the queue length for [completely established sockets]() waiting to be accepted, instead of the number of incomplete connection requests.
If the backlog argument is greater than the value in /proc/sys/net/core/somaxconn, then it is silently truncated to that value; the default value in this file is 128. …
The maximum length of the queue for [incomplete sockets]() can be set using /proc/sys/net/ipv4/tcp_max_syn_backlog.
When syncookies are enabled there is no logical maximum length and this setting is ignored. See tcp(7) for more information.
This means that current Linux versions use two distinct queues:
a SYN queue with a size specified by a system wide setting and
SYN_RECV [cona1, cona2, cona3 … ]
an accept queue with a size specified by the application.
ESTABLISHED [conb1, conb2, conb3 … ]
SYN cookies
from wikipedia:
SYN cookie is a technique used to resist SYN flood attacks. … … defines SYN cookies as [“particular choices of initial TCP sequence numbers by TCP servers.” ]()In particular, the use of SYN cookies allows a server to avoid dropping connections when the SYN queue fills up. Instead, the server behaves as if the SYN queue had been enlarged.
The server sends back the appropriate SYN+ACK response to the client but discards the SYN queue entry.
If the server then receives a subsequent ACK response from the client, the server is able to reconstruct the SYN queue entry using information encoded in the TCP sequence number.
How the cookie is generated
How TCP backlog works in Linux
TCP SYN Cookies – DDoS defence
Quick Blind TCP Connection Spoofing with SYN Cookies
Question
SynCookie can rebuild the connection , but in demo it did not.
Can not find any warn log in dmesg or /var/log/messages
The SYN_RECEIVE queue is not full
SYN_RECV < SYN_SENT
sudo netstat -nap | grep -w '5555' | awk '{print $6" "$7}' | sort | uniq -c
1 LISTEN 20310/./Server
125 ESTABLISHED -
27 ESTABLISHED 20310/./Server
69 SYN_RECV -
221 ESTABLISHED 59253/./ClientSend
79 SYN_SENT 59253/./ClientSend
sudo netstat -nap | grep -w '5555' | awk '{print $6" "$7}' | sort | uniq -c
54 ESTABLISHED -
138 ESTABLISHED 20310/./Server
300 ESTABLISHED 59253/./ClientSend
1 LISTEN 20310/./Server
Did SynCookie take effect?
Demo
SynCookie is not triggered
Why Syn Cookie did not work?
The SYN_RECEIVE queue is not full
Ack Missing
Demo
Client | Server |
---|---|
closed | closed |
closed | listen |
(SYN) | |
syn_sent | syn_received |
(SYN, ACK) | |
established | |
[(ACK)missing]() | |
established | established |
Why
1257 /*
1258 * The three way handshake has completed - we got a valid synack -
1259 * now create the new socket.
1260 */
1261 struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
1262 struct request_sock *req,
1263 struct dst_entry *dst,
1264 struct request_sock *req_unhash,
1265 bool *own_req)
1266 {
1267 struct inet_request_sock *ireq;
1268 struct inet_sock *newinet;
1269 struct tcp_sock *newtp;
1270 struct sock *newsk;
1271 #ifdef CONFIG_TCP_MD5SIG
1272 struct tcp_md5sig_key *key;
1273 #endif
1274 struct ip_options_rcu *inet_opt;
1275
1276 if (sk_acceptq_is_full(sk))
1277 goto exit_overflow;
The code after the exit_overflow label will perform some cleanup, update the ListenOverflows and ListenDrops statistics in /proc/net/netstat and then return NULL.
This will trigger the execution of the listen_overflow code in tcp_check_req:
774 listen_overflow:
775 if (!sysctl_tcp_abort_on_overflow) {
776 inet_rsk(req)->acked = 1;
777 return NULL;
778 }
This means that unless /proc/sys/net/ipv4/tcp_abort_on_overflow is set to 1 (in which case the code right after the code shown above will send a RST packet), the implementation basically does… nothing!
To summarize, if the TCP implementation in Linux receives the ACK packet of the 3-way handshake and the accept queue is full, it will basically [ignore that packet ]() .
SYN_RECV not full && SYN_RECV < SYN_SENT
sudo netstat -nap | grep -w '5555' | awk '{print $6" "$7}' | sort | uniq -c
1 LISTEN 20310/./Server
125 ESTABLISHED -
27 ESTABLISHED 20310/./Server
69 SYN_RECV -
221 ESTABLISHED 59253/./ClientSend
79 SYN_SENT 59253/./ClientSend
The reason is the following code in the tcp_v4_conn_request function (which does the processing of SYN packets) in net/ipv4/tcp_ipv4.c:
/* Accept backlog is full. If we have already queued enough
* of warm entries in syn queue, drop request. It is better than
* clogging syn queue with openreqs with exponentially increasing
* timeout.
*/
if (sk_acceptq_is_full(sk) && inet_csk_reqsk_queue_young(sk) > 1) {
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_LISTENOVERFLOWS);
goto drop;
}
What this means is that if the accept queue is full, then the kernel will impose a limit on the rate at which SYN packets are accepted. If too many SYN packets are received, some of them will be dropped. In this case, it is up to the client to retry sending the SYN packet