I have some dead connections in an application which enter a hanged state if the client machine is dead. One of the connections is shown as
->192.168.1.214:49029 (ESTABLISHED)
Is there a way to terminate these connections from the Linux command line without restarting the server?
After searching, I found a solution called tcpkill, but it will not work for me as it permanently blocks the IP.
On linux kernel >= 4.9 you can use the ss command from iproute2 with key -K
ss -K dst 192.168.1.214 dport = 49029
the kernel have to be compiled with CONFIG_INET_DIAG_DESTROY option enabled.
tcpkill could not work). However, I'll admit that I've not inspected killcx but it feels like a lot of security software would prevent that from working unless you modify your iptables to allow these spoofed packets through.
– Seth Robertson
May 21 '19 at 17:32
sudo ss -K .... on Ubuntu Bionic 18.04 LTS. I had a tmux process that was stuck at a small screen size because of a remote, but dead, but not timed-out connection. All fixed!
– nealmcb
Nov 04 '19 at 20:03
Originally from: http://rtomaszewski.blogspot.sk/2012/11/how-to-forcibly-kill-established-tcp.html
To "kill" a socket, you must send a TCP reset packet. To send it (and be accepted by the other side), you must know the actual TCP sequence number.
1) The already mentioned tcpkill method learns the SEQ number by passively sniffing on the network and waiting for valid packets of this connection to arrive. Then it uses the learned SEQ number to send RSET packets to both sides. However if the connection is idle/hanged and no data flows, it won't do anything and will wait forever.
2) Another method uses perl script called killcx (link to Sourceforge). This actively sends spoofed SYN packets and learns the SEQ number from the answer. It then sends RSET packets the same way as tcpkill.
Alternatively approach (based on what you want to achieve) is to use gdb debugger to attach to a process owning this socket/connection and issue close() syscall on its behalf - as detailed in this answer.
If you want to deal only with hanged connections (the other side is dead), there are various timeouts (TCP keepalive for example), which should automatically close such connections if configured properly on the system.
tcpkill might do it for you. In Ubuntu it is in the dsniff package.
Something like:
$ sudo tcpkill -i wlan0 host 192.168.1.214
(or some other tcpdump like expression for what connection to kill).
Do - as root netstat -tunp|grep 49029. The last column of the output should show you the PID and program name of the process responsible for that connection.
If you are lucky there is a single process for just that connection.
If you are unlucky it gets more complicated (the PID is responsible for more than just that one connection). What kind of service is this?
Why do you want to terminate that session?
tcpkill cannot close a dead (hanged) connection. It is based libpcap, it construct a packet to sent FIN packet. If the connection is already dead, it cannot get the right sequence number.
The only way is to close the process, so makes everywhere is NOT SPOF.
Adding to Marki555 answer which is a great list of options on how to do this.
I found another option - which is long - but also works for idle connections. It is to use a kernel debugger to get the TCP sequence number and then send FIN packets (I believe) using hping3. From https://blog.habets.se/2017/03/Killing-idle-TCP.html
Killing idle TCP connections
Mar 15, 2017, Categories: network,linux
Why
Let’s say you have some TCP connections to your local system that you want to kill. You could kill the process that handles the connection, but that may also kill other connections, so that’s not great. You could also put in a firewall rule that will cause the connection to be reset. But that won’t work on a connection that’s idle (also if one side is initiator then using this method the other side would not tear down its side of the connection). There’s tcpkill, but it needs to sniff the network to find the TCP sequence numbers, and again that won’t work for an idle connection.
Ideally for these long-running connections TCP keepalive would be enabled. But sometimes it’s not. (e.g. it’s not on by default for gRPC TCP connections, and they certainly can be long-running and idle).
You could also do this by attaching a debugger and calling shutdown(2) on the sockets, but having the daemon calling unexpected syscalls thus getting into an unexpected state doesn’t really make for a stable system. Also attaching a debugger hangs the daemon while you’re attached to it.
This post documents how to do this on a Debian system.
No, really. Why?
If a client connects to a dual-stack hostname it’ll (usually, see RFC3484) first try IPv6, and then IPv4 if that fails.
If a server comes up after the client tries IPv6 then it’ll fall back to IPv4, even though IPv6 would have worked at that time too.
I want to kick the IPv4 clients over to IPv6, since restarting the server (or even rebooting the server) doesn’t change anything about the race, and I don’t want to restart the clients because they’re doing long-running compute work that I don’t want to lose state on.
With IPv6 I can differentiate hosts behind NAT, for example.
How
Take the date from uname -a and add a week or so, and open the Debian archive for that day. E.g. http://snapshot.debian.org/archive/debian/2017031500T000000Z/pool/main/l/linux/. Download the -dbg version of the kernel you’re running. E.g.: linux-image-3.16.0-4-amd64-dbg_3.16.39-1+deb8u2_amd64.deb 351181890 2017-03-10 03:37:13
mkdir tmpkernel cd tmpkernel dpkg -x ../linux-image….deb . cp ./usr/lib/debug/lib/modules/*/vmlinux .
$ ss -e -t dst 10.0.64.123 State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 ::ffff:192.0.2.1:22 ::ffff:10.0.64.123:30201 uid:1003 ino:68386802 sk:ffff88000caa2800 <->
sudo apt-get install crash sudo crash -e emacs ./vmlinux
crash> struct tcp_sock.rcv_nxt,snd_una ffff88000caa2800 rcv_nxt = 2691239595 snd_una = 3825672049
hping3 -s 22 -c 1 -M 3825672049 -L 2691239595 -F -A -p 30201 10.0.64.123 hping3 -s 30201 -c 1 -L 2691239595 -M 3825672049 -F -A -p 22 -a 10.0.64.123 192.0.2.1
netstat -napW | grep 10.0.64.123 If possible you may want to check the remote end too. But if it’s the client that will eventually send traffic then it’ll be cleanly disconnected at that point.
DROP packets for incoming/outgoing IP address using:
sudo iptables -A INPUT -s 50.17.54.5 -j DROP
sudo iptables -A OUTPUT -d 50.17.54.5 -j DROP
Undo the change when finished and delete the rules you added using:
sudo iptables -D INPUT -s 50.17.54.5 -j DROP
sudo iptables -D OUTPUT -d 50.17.54.5 -j DROP
This repo https://github.com/caruccio/tcp_killer/ which is a python3 fix for https://github.com/google/tcp_killer does the job for me. And it works in both Linux and macOS.
It uses the debugger approach mentioned by Marki555 and described in this answer.
To kill port 34700 have used this command:
fuser -k 34700/tcp
Ref.: https://stackoverflow.com/questions/750604/freeing-up-a-tcp-ip-port
sscommand is much easier and more general than the others. – nealmcb Nov 04 '19 at 20:02