183

Sometimes my SSH session disconnects with a Write failed: Broken pipe message. What does it mean? And how can I keep my session open?

I know about screen, but that's not the answer I'm looking for. I think this is a sshd config option.

9 Answers9

154

It's possible that your server closes connections that are idle for too long. You can update either your client (ServerAliveInterval) or your server (ClientAliveInterval)

 ServerAliveInterval
         Sets a timeout interval in seconds after which if no data has
         been received from the server, ssh(1) will send a message through
         the encrypted channel to request a response from the server.  The
         default is 0, indicating that these messages will not be sent to
         the server.  This option applies to protocol version 2 only.

 ClientAliveInterval
         Sets a timeout interval in seconds after which if no data has
         been received from the client, sshd(8) will send a message
         through the encrypted channel to request a response from the
         client.  The default is 0, indicating that these messages will
         not be sent to the client.  This option applies to protocol
         version 2 only.

To update your server (and restart your sshd)

echo "ClientAliveInterval 60" | sudo tee -a /etc/ssh/sshd_config

Or client-side:

echo "ServerAliveInterval 60" >> ~/.ssh/config 
Gert
  • 9,994
  • 2
    NB that sometimes "*AliveInterval" isn't enough http://stackoverflow.com/questions/10665267/keep-ssh-session-alive-while-computer-sleep – rogerdpack Feb 18 '15 at 16:33
  • 3
    There is no ~/.ssh/config on my Mac, do I have to create it there or it's is somewhere else? – AGamePlayer Sep 06 '15 at 03:40
  • 7
    @AwQiruiGuo Yes, you should create the directory (~/.ssh) first. So mkdir -p ~/.ssh; chmod 700 ~/.ssh; touch ~/.ssh/config – Gert Sep 07 '15 at 06:43
  • In my case that sure is irrelevant to being idle. I once got a disconnect in two terminals simultaneously, while I was typing in one of them. And in general, the time I'm not touching the connection then get a disconnect is often smaller than the time I'm also not touching the connection, while not getting any disconnects. I think, the notable thing here is that disconnects are always in all ssh connections I had simultaneously rather than in just one of them. – Hi-Angel Sep 29 '20 at 14:10
  • 2
    so basically by modify the config at the client-side could solve the problem? What if I am an admin of some server, and I really want to shut down some ssh sessions that are idle for 1 hour? – Luk Aron Jan 28 '21 at 12:18
22

If you want to have a longer connection period, in the client add:

echo 'ServerAliveInterval 30' | tee -a ~/.ssh/config
echo 'ServerAliveCountMax 1200' | tee -a ~/.ssh/config

ServerAliveCountMax by default this is set to 3. Therefore once the ServerAliveInterval has sent 3 small packs of info to your server it will then automatically log out. Setting it to 1200 means this process will have to occur at least 1200 times. In short you should be connected at least 30*1200 seconds (10 hours).

Francisco Costa
  • 337
  • 2
  • 5
  • I tried setting it to 0, but still kept getting the broken connection error, only way to stop it was to use some larger values like mentioned in the answer – yolob 21 Dec 14 '21 at 11:03
  • 1
    Setting ServerAliveCountMax=1200 is a bad idea. It's not a cumulative counter, it's the number of failed server alive messages that can happen in a row before disconnecting. In your example your server could be down for 10 hours and your connection would still be open. See the man page http://man.openbsd.org/ssh_config – Earl Ruby Jan 18 '22 at 18:26
  • -1. This answer is completely bogus. That is simply not how ServerAliveCountMax works. – André Chalella May 11 '23 at 05:46
  • @AndréChalella Then how does it work? – Aaron Franke Oct 26 '23 at 05:07
11

An alternative solution would be to use mosh - the mobile shell. In contrast to ssh it connects via UDP and supports roaming. You can start your session at home, suspend your laptop, take it to work / friends / wherever else you have internet, unsuspend your laptop and continue to work as if nothing has happened. It is especially useful if you are on a lousy internet connection: It shows instant feedback if your keystrokes don't reach the server and continuously tries to reestablish the connection.

Installation and setup are simple: It is now included in all current Linux (plus a few non-Linux) distributions and it coordinates the session initialization and authentication via a prior ssh connection. So if you are able to connect via ssh user@server you are very likely to be able to connect with mosh just by calling mosh user@server, if the mosh packages are installed on both ends.

The main reason for connection failures is that you have to reach the server on a UDP port (default range: 60000-61000) for mosh to work. So if the server is behind a firewall you are mostly out of luck if can't punch holes in it yourself (security implications).

Perseids
  • 483
6

I spent lots of time messing with ServerAliveInterval and ClientAliveInterval variables with no joy.

Finally, after comparing with a machine that did not have the broken pipe problem, I found the directive at the bottom of /etc/ssh/sshd_config:

KeepAlive yes

Putting this at the bottom of my problem server's /etc/ssh/sshd_config did the trick.

Verified for Ubuntu 20.04.1 LTS, Ubuntu 16.04.7 LTS.

Norman
  • 61
  • Can I add this in my .ssh/config on OSX? – O.rka Mar 29 '21 at 21:11
  • 1
    @O.rka ~/.ssh/config would be the client config, for the computer you're connecting from. /etc/ssh/sshd_config would be for the remote server you're connecting to. – Kenny Evitt Mar 19 '22 at 22:23
  • 2
    On my version of ssh, it was TCPKeepAlive yes - thanks! – oemb1905 Jul 01 '22 at 00:04
  • This setting (well, TCPKeepAlive) is a bit confusing.

    The docs say: "[...] this means that connections will die if the route is down temporarily, and some people find it annoying. The default is yes (to send TCP keepalive messages), and the client will notice if the network down or the remote host dies. This is important in scripts, and many users want it too."

    So I don't see how enabling it would avoid "broken pipes" if the connection dies or oscillates, in fact I would expect the opposite and it's what I see: I'm constantly having broken pipes with TCPKeepAlive enabled.

    – TheMechanic Oct 05 '23 at 17:39
4

It usually means that your network (TCP) connection was reset. E.g. your internet provider reconnected you or something like this.

maxschlepzig
  • 57,532
4

I had the same problem but it is not as expected. If you find that on the same network another server is trying for the same IP address, you will face the same problem. To solve this you need to check if there are other servers which use your same IP address. This can be done using the arp command.

I am using Debian so here is a sample of the commands that I use to determine if another server was indeed using the same IP address

apt-get install arp-scan
arp-scan -I eth0 -l | grep 192.168.1.42
  192.168.1.42 d4:eb:9a:f2:11:a1 (Unknown)
  192.168.1.42 f4:23:a4:38:b5:76 (Unknown) (DUP: 2)

You will notice two sets of mac address using the same IP address. Avoid conflict by setting one to another IP address.

steve
  • 21,892
3

You can use SSH/SCP argument '-o' to accomplish that, no need to set your (if you're admin) server's ssh-config.

SSH options related to keeping connection alive are

  • TCPKeepAlive,
  • ServerAliveCountMax,
  • ServerAliveInterval.

Basically, it goes like this:

% ssh -o TCPKeepAlive=yes \
      -o ServerAliveCountMax=20 \
      -o ServerAliveInterval=15 \
      user@server.example

, to have the client sending "keep-alive" messages every 15 seconds until the limit of 20 messages -- which amounts to 20 x 15 seconds = 5 minutes --, at which point the connection is considered broken/dead. The client process ends.

The man pages will give you more, detailed information:

Brandt
  • 292
3

Another cause for the "Broken Pipe message" is that another machine is attempting use the same IP as your host.

A simple way to test if someone else is using that IP:

  1. Turn off your host
  2. ping the same IP to see if another machine is using that IP

To find out which machines are on your network, you can use this Unix & Linux question title: How to find what other machines are connected to the local network.

Josir
  • 130
  • 2
    This answer belongs in the list of possible causes to help the OP / other readers understand that the cause is NOT always related to sshd configuration. I often work on projects with a protected IP address (e.g., VPN) and shared users (such as deploy) and often run into this contention. – bkkflynnsta Feb 08 '22 at 04:04
2

For me this was because I had used ChrootDirectory which requires that the entire path given be owned by root (which it was not for me). Changing the permissions for the path and restarting sshd fixed the problem.

Qwertie
  • 1,264
  • what permission you used ? 701? – Rohit gupta Apr 24 '22 at 12:10
  • This! If you want your user to access a specific directory by setting

    ChrootDirectory /var/www/specificDir

    You need specificDir to be owned by root, but the files and subdirectories inside be owned by User

    – Illes Peter May 11 '22 at 07:10