How to find applications/ports that do bind() but don't do listen()?

Question

When a faulty application calls bind() with a TCP socket to a port P but does not follow with listen(), the port P is not listed among open ports, i.e. netstat or ss or ls /proc/net/tcp do not show it, but the port is occupied and no other application can use it. Is there a reasonable way to find such applications and such ports?

Answer 1

Until something more suitable is made available, here's an answer that tries in an absolutely non-industrial way to find processes that used bind(2) on a TCP socket, but then did neither listen(2) nor connect(2), and can also display what's the bound TCP address.

Requires getfattr found in a package named attr in most distributions plus kernel >= 3.7 to filter out non-TCP sockets, and a minimal installation of gdb (eg on Debian: gdb-minimal). Doesn't require a development environment. Should be run as the root user (or else it will find only the same users' information, but this won't even work accross containers). See Caveats at the end.

---

Components:

• A first shell script mimics a part of what would lsof do but only for this specific case. Searches all processes for socket FD. For sockets with the property TCP or TCPv6 (which is available as a file meta-attribute system.sockprotoname using getfattr, as found with lsof that would use getxattr(2) in such manner to at least display it's a TCP socket), check if the (sockfs pseudo-filesystem's) inode can be found in their respective network namespace's tcp or tcp6 proc file and if not displays the pid, fd and inode as candidate 3-uple. This script alone will find and list "defective" processes.

  findbadtcpprocs.sh:

  #!/bin/sh
  find /proc -mindepth 1 -maxdepth 1 -name '[1-9]*' |
      xargs -I{} find {}/fd -follow -type s 2>/dev/null |
          while read procfd; do
              type=$(getfattr --absolute --only-values -L -n system.sockprotoname $procfd | tr '\0' '\n')
              if [ "$type" = "TCP" -o "$type" = "TCPv6" ]; then
                  inode=$(stat -L -c %i $procfd)
                  pid=$(echo $procfd | cut -d/ -f3)
                  if awk '$10 == inode { exit 1 }' inode=$inode /proc/$pid/net/tcp /proc/$pid/net/tcp6; then
                      fd=$(echo $procfd | cut -d/ -f5)
                      echo $pid $fd $inode
                  fi
              fi
          done 
  

  This script can be used standalone to just find candidate processes without additional information.

• Then a gdb script which must be given the right fd information. It attaches on a candidate process and will (first allocate some memory in order to) run getsockname(2), display the bound socket (and free allocated resources) an release the process.

  getsockname.gdb:

  set $malloc=(void *(*)(long long)) malloc
  set $ntohs=(unsigned short(*)(unsigned short)) ntohs
  p $malloc(64)
  p $malloc(4)
  set *(long *)$2=64
  p (int) getsockname($fd,$1,$2)
  set logging file /dev/stdout
  set logging on
  if *((short *) $1) == 2
      set $ip=(unsigned char *) ($1+4)
      printf "%hu.%hu.%hu.%hu",$ip[0],$ip[1],$ip[2],$ip[3]
  else
      if *((short *) $1) == 10
          set $ip6=(unsigned short *) ($1+8)
          printf "[%hx:%hx:%hx:%hx:%hx:%hx:%hx:%hx]",$ntohs($ip6[0]),$ntohs($ip6[1]),$ntohs($ip6[2]),$ntohs($ip6[3]),$ntohs($ip6[4]),$ntohs($ip6[5]),$ntohs($ip6[6]),$ntohs($ip6[7])
      end
  end
  printf ":%hu\n",$ntohs(*(unsigned short *)($1+2))
  set logging off
  call (void) free($2)
  call (void) free($1)
  quit
  

• Finally a glue script uses both previous scripts for easy operation. It will avoid uselessly attaching to multiple processes (or threads) sharing the same socket.

  result.sh:

  #!/bin/sh
  oldinode=-1
  ./findbadtcpprocs.sh | sort -s -n -k 3 | while read pid fd inode; do
      printf '%d\t%d\t%d\t' $pid $fd $inode
      if [ $inode -ne $oldinode ]; then
          socketname=$(gdb -batch-silent -p $pid -ex 'set $fd'=$fd -x ./getsockname.gdb 2>/dev/null) || socketname=FAIL
          oldinode=$inode
      fi
      printf '%s\n' "$socketname"
  done
  

  Just running this will provide all:

  chmod a+rx findbadtcpprocs.sh result.sh
  ./result.sh
  

• As a bonus a simple reproducer in C source code that will create two processes using the same TCP socket, without using listen(2) on it. Usage: gcc -o badtcpbind badtcpbind.c and ./badtcpbind 5555

  badtcpbind.c:

  #include <stdio.h>
  #include <sys/types.h>
  #include <sys/socket.h>
  #include <netinet/in.h>
  #include <strings.h>
  #include <stdlib.h>
  #include <unistd.h>
  int main(int argc, char argv[])
  {
      int sockfd;
      struct sockaddr_in myaddr;
      if (argc < 2) {
          exit(2);
      }
      if ((sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
          perror("socket");
          exit(1);
      }
      bzero(&myaddr, sizeof myaddr);
      myaddr.sin_family = AF_INET;
      myaddr.sin_addr.s_addr = INADDR_ANY;
      myaddr.sin_port = htons(atoi(argv[1]));
      if (bind(sockfd, (struct sockaddr ) &myaddr, sizeof myaddr) < 0) {
          perror("bind");
          exit(1);
      }
  #if 0
       listen(sockfd,5);
  #endif
  fork();
  sleep(9999);
  
  }
  

example:

# ./badtcpbind 5555 &
[1] 330845
# ./result.sh 
108762  20  303507  0.0.0.0:0
330845  3   586443  0.0.0.0:5555
330846  3   586443  0.0.0.0:5555

(Yes for some unknown reason, a libvirtd process appears here to create a TCP socket which doesn't get used and is caught in the first line of the results).

---

Caveats:

• a language better than shell should probably be used to allow more readability and efficiency.

• certainly even more racy than lsof.

• attaching to a running process the way is done here has issues:

  • doesn't work on a statically linked binary (malloc() function or some symbol definitions aren't available then).
  • as no debug information is available, most functions are explicitly scoped and this might not run in all environments without change (tested on amd64 architecture with kernel 5.10.x, on Debian bullseye, Debian 10 and CentOS 7 userspaces).
  • likewise might not as-is work on a binary linked with an other libc than usual glibc.
  • is intrusive and might crash fragile (especially multi-threaded) applications. Checks aren't done (eg: malloc(3)'s or getsockname(2)'s failure).

• the last script considers sockfs inodes to be globally (rather than per-network-namespace) unique, which I didn't attempt to prove but keeps the script simpler.