125

How would you go about finding the DNS servers used by systemd-resolved, for troubleshooting purposes?

Generally I can use dig and test the DNS servers shown in /etc/resolv.conf. (Or windows - ipconfig /all + nslookup). But that approach doesn't work when resolv.conf just points to a local resolver daemon on a loopback address. What method is used under systemd-resolved, to show the DNS servers it uses?


(unbound has config files I could look into. dnsmasq does too, though I'm not sure if servers can be added dynamically without a config file. Even NetworkManager, now has nmcli, and I see you can query nmcli d show wlan0 to show the DNS configuration for an interface.)

sourcejedi
  • 50,249

4 Answers4

131

Very helpful for troubleshooting is also:

journalctl -u systemd-resolved -f

There you can see what systemd-resolved is really doing. In my case it was not contacting the DNS servers that were reported via systemd-resolve --status at all. If it's doing weird things like that, then somtimes a restart via sudo systemctl restart systemd-resolved is a good idea.

EDIT:

In order to get more information from resolved you need to put

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

into the override.conf of systemd-resolved via

sudo systemctl edit systemd-resolved

Restart to take effect:

sudo systemctl restart systemd-resolved

EDIT 2:

Don't forget to revert this afterwards as @bmaupin and @Aminovic have helpfully pointed out in the comments.

sudo systemctl revert systemd-resolved
sudo systemctl restart systemd-resolved
danuker
  • 317
Elmar Zander
  • 1,963
  • 9
    It might be worth noting that after running sudo systemctl edit systemd-resolved it seems sudo systemctl restart systemd-resolved needs to be run in order for the new settings to be applied. Thanks! – bmaupin Apr 23 '19 at 15:13
  • 6
    After all these years without having to deal with windows, not much anyway, you get the "If it's doing weird things like that, then somtimes a restart via.." advice and you know you are back to where you started. Sigh. – Jaap Versteegh Aug 22 '19 at 10:54
  • 3
    It might also be helpful to add the command to undo this when you're done troubleshooting: sudo systemctl revert systemd-resolved – bmaupin Aug 03 '21 at 12:55
  • Do not forget to remove the debug mode after you finish! on my side, it was filling the server's disk – Aminovic Mar 24 '22 at 08:26
  • This is way easier to to run systemctl service-log-level systemd-resolved debug. Because non persistent – ninjaconcombre Jun 28 '23 at 10:25
80

Use resolvectl status (systemd-resolve --status when using systemd version earlier than 239) to show your global and per-link DNS settings.

8

The resolv.conf file

The manual says:

The resolver is a set of routines in the C library that provide access to the Internet Domain Name System (DNS). The resolver config file contains information that is read by the resolver routines the first time they are invoked by a process.

If this file does not exist, only the name server on the local machine will be queried, and the search list contains the local domain name determined from the hostname.

systemd-resolved

On Ubuntu 20.04, systemd-resolved is a local DNS server included with systemd that acts as a stub resolver and it should automatically edit /etc/resolv.conf with the correct config.

On older versions you might need to manually symlink it with the following:

$ sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

By default, systemd-resolved configures a local DNS server listening at 127.0.0.53 on port 53 and you can run lsof -i @127.0.0.53:53 to verify. Make sure you do not have any other service like dnsmasq listening to the same address and port.

Note that 127.0.0.53 is the loopback interface but you can have another DNS server listening on port 53 at another IP address on the loopback interface, like 127.0.0.1 or 127.0.0.2.

If it does not work, try the following:

$ sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

This file contains DNS servers from DHCP leases but be aware that you will not benefit from the stub resolver caching feature.

dnsmasq with NetworkManager

The dnsmasq lightweight caching DNS server accepts DNS queries and either answers them from a small, local, cache or forwards them to a real, recursive, DNS server.

This is very good to have a little DNS service to test local websites for a web development machine.

First, verify that NetworkManager manages your current interface (wlp0s20f3 in my case):

$ nmcli dev
$ nmcli dev set wlp0s20f3 managed yes

Now you can enable the NetworkManager built-in dnsmasq resolver:

$ cat /etc/NetworkManager/NetworkManager.conf
[main]
dns=dnsmasq

You can now benefit from the flexibility of NetworkManager along dnsmasq, whose config is located in /etc/NetworkManager/dnsmasq.d:

$ cat /etc/NetworkManager/dnsmasq.d/custom.conf
# Resolve all domains ending in .dev to 127.0.0.1
address=/.dev/127.0.0.1

Do not forget to disable the system-wide dnsmasq service if present and apply your changes:

$ sudo systemctl stop dnsmasq.service && sudo systemctl disable dnsmasq.service
$ sudo systemctl restart NetworkManager.service

If for some reasons you prefer to have dnsmasq running separately, you must first prevent NetworkManager from overriding /etc/resolv.conf:

$ cat /etc/NetworkManager/NetworkManager.conf
[main]
dns=none

Now you must tell dnsmasq to get its upstream servers from somewhere other that /etc/resolv.conf:

$ cat /etc/dnsmasq.conf
listen-address=127.0.0.53
resolv-file=/run/NetworkManager/resolv.conf

This file is generated by NetworkManager and also contains DNS servers from DHCP leases. Now configure dnsmasq for resolving:

$ cat /etc/resolv.conf
# Use local dnsmasq resolver
nameserver 127.0.0.53

Configure your own local DNS settings:

$ cat /etc/dnsmasq.d/custom.conf
# Resolve all domains ending in .dev to 127.0.0.1
address=/.dev/127.0.0.1

Apply your new config:

$ sudo systemctl stop systemd-resolved.service && sudo systemctl disable systemd-resolved.service
$ sudo systemctl restart dnsmasq.service

dnsmasq with systemd-networkd

Unfortunately it is not as easy as with NetworkManager and this is due to the fact that systemd-networkd does not expose a file with DNS nameservers obtained from DHCP servers as this is managed by systemd-resolved instead.

This is also the case if you use the quite deprecated ifupdown package.

Open the man for dhclient-script:

$ man 8 dhclient-script

The HOOKS section says:

When it starts, the client script first defines a shell function, make_resolv_conf, which is later used to create the /etc/resolv.conf file. To override the default behaviour, redefine this function in the enter hook script.

So if you still want to use dnsmasq along systemd-networkd you will need to redefine make_resolv_conf to create a resolv-file for dnsmasq so that it can get its upstream servers from your DHCP settings.

The resolvconf package provides a wrapper interface around the different daemons involved in populating /etc/resolv.conf and you can use it to configure the make_resolv_conf behavior.


The same problem also happens if you need to obtain the IPv6 upstream servers from RDNSS in Router Advertisement messages (see Neighbor Discovery Protocol) as this is also managed by systemd-resolved.

Try the following command from the ndisc6 package:

$ rdisc6 wlp0s20f3

You should see various information coming from your router, including your IPv6 address prefix and DNS servers. If you do not see anything, your router may not support it.

The rdisc6 program implements the ICMPv6 Router Discovery in userland, using NETLINK_ROUTE sockets for which RDNSS support was added in Linux kernel 2.6.24. If you want to support older versions, you have no choices but to interact with the device driver at OSI Layer 2, using SOCK_RAW sockets.

emandret
  • 203
  • Your two code blocks contain exactly the same commands. I think you intended them to be different. – berndbausch Mar 06 '21 at 06:25
  • @berndbausch the snippets are not the same if you look closely, one is for stub-resolv.conf and the other for resolv.conf. – emandret Mar 06 '21 at 11:44
-1

None of the answers here brought me any closer to a working systemd-resolved. One solution to all systemd-resolved problems is to remove it from the equation.

First disable it (as root):

service systemd-resolved stop
systemctl disable systemd-resolved

But that's not enough. Systemd will re-enable it and start it right back up after a few minutes. So you must prevent that by removing systemd-resolved from your server entirely:

rm -f /lib/systemd/systemd-resolved

I've never had problems from other parts of systemd as a result of doing this, but YMMV. Instead of removing it, you could rename it.

Finally, edit /etc/resolv.conf and put some name servers into it:

nameserver 8.8.8.8
nameserver 8.8.4.4
  • 1
    I dump systemd-resolved too and all is well. I mask the service systemctl mask systemd-resolved. this stops systemd from restarting the service. No need to do that now, but next time? – Stephen Boston Aug 16 '21 at 23:50
  • If the service is being restarted by something you can't find, there probably is a need to use systemctl mask instead. When you install an updated package, it will almost certainly recreate those files you deleted. – sourcejedi Aug 17 '21 at 10:56
  • I had to look up what Poettering considers "mask" to mean: "This will link these unit files to /dev/null, making it impossible to start them. This is the stronger version of disable, since it prohibits all kinds of activation of the unit, including enablement and manual activation." – Throw Away Account Aug 20 '21 at 19:50