Note: I now maintain a lsof
wrapper that combines both approaches described here and also adds information for peers of loopback TCP connections at https://github.com/stephane-chazelas/misc-scripts/blob/master/lsofc
Linux-3.3 and above.
On Linux, since kernel version 3.3 (and provided the UNIX_DIAG
feature is built in the kernel), the peer of a given unix domain socket (includes socketpairs) can be obtained using a new netlink based API.
lsof
since version 4.89 can make use of that API:
lsof +E -aUc Xorg
Will list all the Unix domain sockets that have a process whose name starts with Xorg
at either end in a format similar to:
Xorg 2777 root 56u unix 0xffff8802419a7c00 0t0 34036 @/tmp/.X11-unix/X0 type=STREAM ->INO=33273 4120,xterm,3u
If your version of lsof
is too old, there are a few more options.
The ss
utility (from iproute2
) makes use of that same API to retrieve and display information on the list of unix domain sockets on the system including peer information.
The sockets are identified by their inode number. Note that it's not related to the filesystem inode of the socket file.
For instance in:
$ ss -x
[...]
u_str ESTAB 0 0 @/tmp/.X11-unix/X0 3435997 * 3435996
it says that socket 3435997 (that was bound to the ABSTRACT socket /tmp/.X11-unix/X0
) is connected with socket 3435996. The -p
option can tell you which process(es) have that socket open. It does that by doing some readlink
s on /proc/$pid/fd/*
, so it can only do that on processes you own (unless you're root
). For instance here:
$ sudo ss -xp
[...]
u_str ESTAB 0 0 @/tmp/.X11-unix/X0 3435997 * 3435996 users:(("Xorg",pid=3080,fd=83))
[...]
$ sudo ls -l /proc/3080/fd/23
lrwx------ 1 root root 64 Mar 12 16:34 /proc/3080/fd/83 -> socket:[3435997]
To find out what process(es) has 3435996, you can look up its own entry in the output of ss -xp
:
$ ss -xp | awk '$6 == 3435996'
u_str ESTAB 0 0 * 3435996 * 3435997 users:(("xterm",pid=29215,fd=3))
You could also use this script as a wrapper around lsof
to easily show the relevant information there:
#! /usr/bin/perl
# lsof wrapper to add peer information for unix domain socket.
# Needs Linux 3.3 or above and CONFIG_UNIX_DIAG enabled.
# retrieve peer and direction information from ss
my (%peer, %dir);
open SS, '-|', 'ss', '-nexa';
while (<SS>) {
if (/\s(\d+)\s+\*\s+(\d+) ([<-]-[->])$/) {
$peer{$1} = $2;
$dir{$1} = $3;
}
}
close SS;
# Now get info about processes tied to sockets using lsof
my (%fields, %proc);
open LSOF, '-|', 'lsof', '-nPUFpcfin';
while (<LSOF>) {
if (/(.)(.*)/) {
$fields{$1} = $2;
if ($1 eq 'n') {
$proc{$fields{i}}->{"$fields{c},$fields{p}" .
($fields{n} =~ m{^([@/].*?)( type=\w+)?$} ? ",$1" : "")} = "";
}
}
}
close LSOF;
# and finally process the lsof output
open LSOF, '-|', 'lsof', @ARGV;
while (<LSOF>) {
chomp;
if (/\sunix\s+\S+\s+\S+\s+(\d+)\s/) {
my $peer = $peer{$1};
if (defined($peer)) {
$_ .= $peer ?
" ${dir{$1}} $peer\[" . (join("|", keys%{$proc{$peer}})||"?") . "]" :
"[LISTENING]";
}
}
print "$_\n";
}
close LSOF or exit(1);
For example:
$ sudo that-lsof-wrapper -ad3 -p 29215
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
xterm 29215 stephane 3u unix 0xffff8800a07da4c0 0t0 3435996 type=STREAM <-> 3435997[Xorg,3080,@/tmp/.X11-unix/X0]
Before linux-3.3
The old Linux API to retrieve unix socket information is via the /proc/net/unix
text file. It lists all the Unix domain sockets (including socketpairs). The first field in there (if not hidden to non-superusers with the kernel.kptr_restrict
sysctl parameter) as already explained by @Totor contains the kernel address of a unix_sock
structure that contains a peer
field pointing to the corresponding peer unix_sock
. It's also what lsof
outputs for the DEVICE
column on a Unix socket.
Now getting the value of that peer
field means being able to read kernel memory and know the offset of that peer
field with regards to the unix_sock
address.
Several gdb
-based and systemtap
-based solutions have already been given but they require gdb
/systemtap
and Linux kernel debug symbols for the running kernel being installed which is generally not the case on production systems.
Hardcoding the offset is not really an option as that varies with kernel version.
Now we can use a heuristic approach at determining the offset: have our tool create a dummy socketpair
(then we know the address of both peers), and search for the address of the peer around the memory at the other end to determine the offset.
Here is a proof-of-concept script that does just that using perl
(successfully tested with kernel 2.4.27 and 2.6.32 on i386 and 3.13 and 3.16 on amd64). Like above, it works as a wrapper around lsof
:
For example:
$ that-lsof-wrapper -aUc nm-applet
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nm-applet 4183 stephane 4u unix 0xffff8800a055eb40 0t0 36888 type=STREAM -> 0xffff8800a055e7c0[dbus-daemon,4190,@/tmp/dbus-AiBCXOnuP6]
nm-applet 4183 stephane 7u unix 0xffff8800a055e440 0t0 36890 type=STREAM -> 0xffff8800a055e0c0[Xorg,3080,@/tmp/.X11-unix/X0]
nm-applet 4183 stephane 8u unix 0xffff8800a05c1040 0t0 36201 type=STREAM -> 0xffff8800a05c13c0[dbus-daemon,4118,@/tmp/dbus-yxxNr1NkYC]
nm-applet 4183 stephane 11u unix 0xffff8800a055d080 0t0 36219 type=STREAM -> 0xffff8800a055d400[dbus-daemon,4118,@/tmp/dbus-yxxNr1NkYC]
nm-applet 4183 stephane 12u unix 0xffff88022e0dfb80 0t0 36221 type=STREAM -> 0xffff88022e0df800[dbus-daemon,2268,/var/run/dbus/system_bus_socket]
nm-applet 4183 stephane 13u unix 0xffff88022e0f80c0 0t0 37025 type=STREAM -> 0xffff88022e29ec00[dbus-daemon,2268,/var/run/dbus/system_bus_socket]
Here's the script:
#! /usr/bin/perl
# wrapper around lsof to add peer information for Unix
# domain sockets. needs lsof, and superuser privileges.
# Copyright Stephane Chazelas 2015, public domain.
# example: sudo this-lsof-wrapper -aUc Xorg
use Socket;
open K, "<", "/proc/kcore" or die "open kcore: $!";
read K, $h, 8192 # should be more than enough
or die "read kcore: $!";
# parse ELF header
my ($t,$o,$n) = unpack("x4Cx[C19L!]L!x[L!C8]S", $h);
$t = $t == 1 ? "L3x4Lx12" : "Lx4QQx8Qx16"; # program header ELF32 or ELF64
my @headers = unpack("x$o($t)$n",$h);
# read data from kcore at given address (obtaining file offset from ELF
# @headers)
sub readaddr {
my @h = @headers;
my ($addr, $length) = @_;
my $offset;
while (my ($t, $o, $v, $s) = splice @h, 0, 4) {
if ($addr >= $v && $addr < $v + $s) {
$offset = $o + $addr - $v;
if ($addr + $length - $v > $s) {
$length = $s - ($addr - $v);
}
last;
}
}
return undef unless defined($offset);
seek K, $offset, 0 or die "seek kcore: $!";
my $ret;
read K, $ret, $length or die "read($length) kcore \@$offset: $!";
return $ret;
}
# create a dummy socketpair to try find the offset in the
# kernel structure
socketpair(Rdr, Wtr, AF_UNIX, SOCK_STREAM, PF_UNSPEC)
or die "socketpair: $!";
$r = readlink("/proc/self/fd/" . fileno(Rdr)) or die "readlink Rdr: $!";
$r =~ /\[(\d+)/; $r = $1;
$w = readlink("/proc/self/fd/" . fileno(Wtr)) or die "readlink Wtr: $!";
$w =~ /\[(\d+)/; $w = $1;
# now $r and $w contain the socket inodes of both ends of the socketpair
die "Can't determine peer offset" unless $r && $w;
# get the inode->address mapping
open U, "<", "/proc/net/unix" or die "open unix: $!";
while (<U>) {
if (/^([0-9a-f]+):(?:\s+\S+){5}\s+(\d+)/) {
$addr{$2} = hex $1;
}
}
close U;
die "Can't determine peer offset" unless $addr{$r} && $addr{$w};
# read 2048 bytes starting at the address of Rdr and hope to find
# the address of Wtr referenced somewhere in there.
$around = readaddr $addr{$r}, 2048;
my $offset = 0;
my $ptr_size = length(pack("L!",0));
my $found;
for (unpack("L!*", $around)) {
if ($_ == $addr{$w}) {
$found = 1;
last;
}
$offset += $ptr_size;
}
die "Can't determine peer offset" unless $found;
my %peer;
# now retrieve peer for each socket
for my $inode (keys %addr) {
$peer{$addr{$inode}} = unpack("L!", readaddr($addr{$inode}+$offset,$ptr_size));
}
close K;
# Now get info about processes tied to sockets using lsof
my (%fields, %proc);
open LSOF, '-|', 'lsof', '-nPUFpcfdn';
while (<LSOF>) {
if (/(.)(.*)/) {
$fields{$1} = $2;
if ($1 eq 'n') {
$proc{hex($fields{d})}->{"$fields{c},$fields{p}" .
($fields{n} =~ m{^([@/].*?)( type=\w+)?$} ? ",$1" : "")} = "";
}
}
}
close LSOF;
# and finally process the lsof output
open LSOF, '-|', 'lsof', @ARGV;
while (<LSOF>) {
chomp;
for my $addr (/0x[0-9a-f]+/g) {
$addr = hex $addr;
my $peer = $peer{$addr};
if (defined($peer)) {
$_ .= $peer ?
sprintf(" -> 0x%x[", $peer) . join("|", keys%{$proc{$peer}}) . "]" :
"[LISTENING]";
last;
}
}
print "$_\n";
}
close LSOF or exit(1);
lsof
author. – Stéphane Chazelas Mar 17 '15 at 09:26ss
not do this? It's kind of over my head, butss -px
lists a lot of unix sockets with peer information like:users: ("nacl_helper",pid=18992,fd=6),("chrome",pid=18987,fd=6),("chrome",pid=18975,fd=5)) u_str ESTAB\t0\t0\t/run/dbus/system_bus_socket 8760\t\t* 15068
and the column headings are...State\tRecv-Q\tSend-Q\tLocal Address:Port\tPeer Address:Port
– mikeserv Mar 17 '15 at 12:31lsof -c terminology
I can seeterminolo 12731\tmikeserv\t12u\tunix\t0xffff880600e82680\t0t0\t1312426\ttype=STREAM
but if I doss -px | grep terminology
I get:u_str\tESTAB\t0\t0\t* 1312426\t*1315046\tusers:(("terminology",pid=12731,fd=12))
– mikeserv Mar 17 '15 at 12:37ss
man page even has an example to list the X server clients!ss -x src /tmp/.X11-unix/*
(doesn't work for me though) – Stéphane Chazelas Mar 17 '15 at 13:03ss
is printing only the Node wherelsof
does the device name as well.. But if I do:ss -px | grep terminology
to get that 1312426 Node again, and then after -ss -px | grep 1315046
to search for the peer end it prints:u_str\tESTAB\t0\t0\t@/tmp/.X11-unix/X0 1315046\t* 1312426
. It seems to be inline with theman
page, which gives this example usage:ss -x src /tmp/.X11-unix/*
Find all local processes connected toX
server. – mikeserv Mar 17 '15 at 13:08ss
then (and understanding even less then) but it just kind of rung a bell. – mikeserv Mar 17 '15 at 13:09ss -x src /tmp/...
does work here... I wonder what the difference would be? There's also an option for listing-m
emory usage, but that doesn't reveal much for me, though. Still, the reversegrep
thing pulled it in too. Hmmm... maybe try theman
example w/-p
as well? That seems to be a little more informative. Oh. You know - I'm probably running usermode X as well - since 1.16 (or whatever that version number was) which might make it easier for me to do introspection. Most X servers are probably still superuser. I wonder ifss
doesn't want to show you root info? – mikeserv Mar 17 '15 at 13:13ss -x src /tmp/...
gives me all non-listening sockets (the filter doesn't filter) or it doesn't give me the peer. I can parse the output ofss -x;ss -lx
though (not reliably for sockets with spaces or newlines) – Stéphane Chazelas Mar 17 '15 at 13:33-e
I also get these little arrows like<->
or<--
or-->
. The<--
and-->
(at a glance) seem to occur only alongside entries which also list filenames. Do you think that is relevant at all? Strike that - That was a pretty weak glance. But it does seem that the@
begins a pathname to a socket proper. And at least all of the pathnames appear to be fully qualified - the/
slashes might be useful. – mikeserv Mar 17 '15 at 13:43cat /usr/share/doc/iproute2/ss.html
– mikeserv Mar 17 '15 at 13:56ss -px src '/tmp/.X11*/*' | grep -v '^[^0-9]*0 *0 '
– mikeserv Mar 17 '15 at 14:18ss -px | grep mysql
I still cannot linkmysql
withmysqld
even if they both appear in the list (kernel 3.2). – Totor Mar 17 '15 at 14:53long
tolong
to find thepeer
offset but how can you be sure thatpeer
is on along
boundary? – Totor Mar 17 '15 at 16:19ss
filter issue seems to have been fixed very recently in http://thread.gmane.org/gmane.linux.network/351180 – Stéphane Chazelas Mar 17 '15 at 21:38ss -px | grep mysql
with iproute 3.16 on a 3.13.1 kernel. Still can't link the client and the server sockets. Is it only working withlsof
or doesn't it work at all? – Totor Mar 17 '15 at 23:54mysql
socket owned by you? I want to believe thatss
wouldn't reveal privileged information to you. I wonder what I have that could be comparable... – mikeserv Mar 17 '15 at 23:58chrome
's processes are run in namespaced containers - and it maintains its own database. You can get information about a running process by referencing asrc
socket file (maybe) and you cangrep
for pid. I just cross-referencedlsof
because that seemed to be the thread of things here. Anyway, it might help. Also there's this crappy doc:cat /usr/share/doc/iproute2/ss.html
. (theip
docs are far and away* better done)*. – mikeserv Mar 18 '15 at 00:02UNIX_DIAG
enabled in my kernel. Now I have, and indeed, when I usess -px
, the Peer column gives me an ID (inode number?) that is the same ID as the Local column on the other side of the socket. It works. – Totor Mar 18 '15 at 01:38<>
? Is it better now? I seem to remember it was a security thing... – mikeserv Mar 18 '15 at 12:23<>
is a dangerous feature. It's useful in that you can doperl -ne 'some processing' 'cmd|'
to do some processing on the output ofcmd
, but that becomes a problem when you doperl -ne 'some processing' -- *
and can't guarantee file names won't end in|
. Details there. Not really a concern here. – Stéphane Chazelas Mar 18 '15 at 12:30