2

How to configure and secure a VNC server on Linux with TigerVNC? (on a screen less server or a classic machine)

Since the VNC Server (TigerVNC) configuration is the same on most Linux distributions and only the installation method differ, this question is targeting: OpenSUSE, Fedora, CentOS, RHEL, Debian, Mageia, Void Linux, Arch Linux, Manjaro and FreeBSD (in order to be useful for more people)

intika
  • 14,406

1 Answers1

10

Installing a VNC Server

On Linux (on a classic machine or a screen less server) there are multiple (opensource) possibility for a VNC server such as TightVNC, TigerVNC and TurboVNC (this is a non exhaustive list, this guide will be using the native version of TigerVNC):

  • TigerVNC Server: using native or Java code and actively maintained.
  • TurboVNC Server: only using Java, actively maintained.
  • TighVNC Server: as of 2020 current Linux's version is v1.3.10 from 2009.

We first need a desktop (E.g. XFCE or KDE):

# OpenSUSE (XFCE)
zypper in -t pattern xfce

OpenSUSE (KDE)

zypper install -t pattern kde kde_plasma

Fedora/OpenSUSE (XFCE)

dnf groupinstall -y "Xfce Desktop"

Fedora/OpenSUSE (KDE)

dnf -y group install "KDE Plasma Workspaces"

CentOS/RHEL (Gnome)

dnf -y group install "Server with GUI"

CentOS/RHEL (XFCE)

dnf --enablerepo=epel group -y install "Xfce" "base-x"

CentOS/RHEL (KDE)

dnf --enablerepo=epel group -y install "KDE Plasma Workspaces"

CentOS v8 (KDE)

dnf --enablerepo=epel,PowerTools dnf -y group install "KDE Plasma Workspaces" "base-x"

Debian (XFCE)

apt install task-xfce-desktop

Debian (KDE)

apt install task-kde-desktop

Mageia (XFCE)

dnf install task-xfce

Mageia (KDE)

dnf install task-plasma5

FreeBSD (XFCE)

pkg install xfce

FreeBSD (KDE)

pkg install x11/kde5

Void Linux (XFCE)

xbps-install -S xfce4

Void Linux (KDE)

xbps-install -S kde5

and optionally, kde5-baseapps

Arch Linux (XFCE)

pacman -S xfce4 xfce4-goodies

Arch Linux (KDE)

pacman -S plasma-desktop

or plasma for the full desktop

pacman -S plasma

Manjaro (XFCE)

pacman -S xfce4-gtk3 xfce4-goodies xfce4-terminal
network-manager-applet xfce4-notifyd-gtk3
xfce4-whiskermenu-plugin-gtk3 tumbler engrampa

Manjaro (KDE)

pacman -S plasma kio-extras

optional kde-applications

Install the TigerVNC X server:

# The package name may change depending on the used distro

CentOS

yum install tigervnc-server

Mageia/Fedora/CentOS/RHEL

dnf install tigervnc-server

ALT Linux

apt install tigervnc-server

openSUSE DNF

dnf install xorg-x11-Xvnc

openSUSE

zypper install xorg-x11-Xvnc

Debian

apt install tigervnc-standalone-server tigervnc-common

FreeBSD

pkg install tigervnc-server

Void Linux

xbps-install -S tigervnc

Arch Linux

pacman -S tigervnc

Manjaro

pacman -S tigervnc


Setup and configuration

Setup the password (the hashed version will be saved at ~/.vnc/passwd):

vncpasswd

Edit the config file (startup script, executed when the server start) ~/.vnc/xstartup as follow:

#!/bin/sh

unset SESSION_MANAGER unset DBUS_SESSION_BUS_ADDRESS

exec startxfce4

XFCE: startxfce4 or xfce4-session

#exec startxfce4 #exec xfce4-session

KDE: startkde or startplasma-x11

#exec startkde #exec startplasma-x11

Gnome: startx

#exec startx

Prepare and/or locate the VNC server's config file at:

~/.vnc/config or /etc/vnc/config

Adapt VNC server's settings with the config file... to get the full list of available options we can use Xvnc -help or man Xvnc, here is a config example (also note that on some system like Suse VNC will/may not work if the option geometry is not set):

## Supported server options to pass to vncserver upon invocation can be listed
## in this file. See the following manpages for more: vncserver(1) Xvnc(1).
## Several common ones are shown below. Uncomment and modify to your liking.
##
##
# -------------
#  Xvnc --help
# -------------
##
# Start server : vncserver
# Stop server : vncserver -kill :1
##
###############################################################################

Only allow connection from local hosts

#localhost

VNC tcp port

rfbport=5900

TCP port to listen for HTTP (default=0)

httpport=0

Directory containing files to serve via HTTP (default=)

httpd=

Protocols...

#nolisten=UDP #listen=TCP

IP settings

useipv4 #useipv6

Interface, listen on the specified network address (default=all)

#interface=127.0.0.1

Use protocol version 3.3 for backwards compatibility

protocol3.3=0

Unix socket access mode (default=384)

#rfbunixmode=384

Unix socket to listen for RFB protocol (default=)

rfbunixpath=

Name of VNC desktop

desktop=MyVNC

Geometry original peppy

geometry=1366x768

Colors

depth=24

Sharing with multiple clients

#alwaysshared nevershared

Disconnect existing clients if an incoming connection is non-shared.

If combined with NeverShared then new connections will be refused while

there is a client active

disconnectclients

Security, specify which security scheme to use (None, VncAuth, Plain,

TLSNone, TLSVnc, TLSPlain, X509None, X509Vnc, X509Plain) (default=TLSVnc,VncAuth)

securitytypes=TLSVnc,VncAuth

Path to the key of the X509 certificate in PEM format (default=)

#X509Key=

Path to the X509 certificate in PEM format (default=)

#X509Cert=

Set maximum number of clients (power of two)

#maxclients=64

Terminate after s seconds of user inactivity (default=0)

#maxidletime=0

Terminate when a client has been connected for s seconds (default0)

#maxconnectiontime=0

Terminate when no client has been connected for s seconds (default=0)

#maxdisconnectiontime=0

The number of seconds after which an idle VNC connection will be dropped

(zero means no timeout) (default=0)

#idletimeout=0

Zlib compression level (default=-1)

#zlibLevel=-1

The maximum number of updates per second sent to each client (default=60)

#framerate=60

GnuTLS priority string that controls the TLS session’s handshake algorithms.

See the GnuTLS manual for possible values. Default is NORMAL.

#GnuTLSPriority=

VNC server can be started/stopped with the following commands; After starting it you can connect to your server with any VNC client to server-ip:used-port (note that you may probably need to open the used port on the firewall)

# Start the server
vncserver

Stop the server :1

vncserver -kill :1

Forcing multiple server to stop

killall Xvnc


Security

Basic VNC setup does not use encryption for the exchanged stream, here are 4 commons way of securing a VNC connection:

  • Using an X509 certificate: the certificate's location need to be added on the config file, and you need to have the certification on the client as well (setting it's location on the client app). Generating an X509 certificate is explained further bellow.

  • Tunneling VNC through an SSH session with the local sock proxy provided by an SSH session: On the server edit /etc/ssh/sshd_config and enable/add AllowTcpForwarding yes then restart your sshd service systemctl restart sshd.service you can then use vncviewer with the option -via or establish the tunnel connection manually, then use any client to connect:
    ssh serverIP -p 22 -i /home/my/private/key -L 5900:127.0.0.1:5900 -C -N
    Then connect to 127.0.0.1:5900 with any VNC client.

  • Run vncviewer on the server and display it's x window on the client over an SSH X forwarded session: On the server edit /etc/ssh/sshd_config and enable/add X11Forwarding yes then restart your sshd service systemctl restart sshd.service.
    On an SSH session (server's SSH shell) running vncviewer :1 will display the vncviewer window on the client.

  • Tunneling VNC through a VPN connection: this is not covered here.


VNC as a systemd service

VNC server can be used as a Systemd's service with the following config file /etc/systemd/system/vncserver.service, enabling the service with systemctl enable vncserver.service will make it auto-start when the system boot (this does not apply to Void Linux or Systemd less Linux's distributions).

# /etc/systemd/system/vncserver.service

[Unit] Description=TigerVNC Server After=syslog.target network.target

[Service] Type=simple #Type=forking User=MY-USER Group=MY-USER-GROUP

#If ran with root #WorkingDirectory=/root #PIDFile=/root/.vnc/%H%i.pid

#If ran with any other user WorkingDirectory=/home/MY-USER PIDFile=/home/%u/.vnc/%H%i.pid

#Environment is required when using a custom GnuTLS version #Environment=LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/usr/lib

ExecStartPre=-/usr/bin/vncserver -kill :%i > /dev/null 2>&1 #ExecStart for forking type version #ExecStart=/usr/bin/vncserver ExecStart=/usr/bin/vncserver -fg ExecStop=/usr/bin/vncserver -kill :%i

[Install] WantedBy=multi-user.target


Generating an X509 certificate

For a simple secure setup this official wiki can be followed, otherwise this is not a light topic for a further understanding of such encryption see the documentation/links for this section. In short the documentation/links indicate that most secure keys as of 2020 are RSA with a high key size (at least 4096 bits) and EdDSA's Ed25519 implementation which is the EdDSA signature scheme using SHA-512 (SHA-2) and Curve25519 (an elliptic curve offering 128 bits of security). Ed25519 resistance is equivalent to an RSA with a 3072-bits key. (alternatively there is EdDSA's ED448 which is the EdDSA signature scheme using SHAKE256 (SHA-3) and Curve448; it is equivalent to an RSA with a ~12448-bits key, an updated OpenSSL install is required)

Generate an RSA-4096-Bits/Ed25519/ED448 private key for the CA:
For Ed25519 and ED448 an updated version of OpenSSL and GnuTLS are required (check the next section).

mkdir ~/.vnc/ssl

Generate an RSA 4096-bits key

openssl genrsa -out ~/.vnc/ssl/ca.private.rsa.4096.key.pem 4096

genrsa is superseded by genpkey (PKCS#1 vs PKCS#8 format), the following command is similar to the previous one.

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out ~/.vnc/ssl/ca.private.rsa.4096.key.pem

Or an ED25519 key (equivalent to an RSA with a 3072-bits key, updated openssl required)

openssl genpkey -algorithm ED25519 -out ~/.vnc/ssl/ca.private.eddsa.ed25519.key.pem

Or an ED448 key (equivalent to an RSA with a ~12448-bits key, updated openssl required)

openssl genpkey -algorithm ED448 -out ~/.vnc/ssl/ca.private.eddsa.ed448.key.pem

The previous step can be done in a more secure way by protecting the key with a password but TigerVNC does not support that.

Check/View the newly generated key (not required):

openssl pkey -in ~/.vnc/ssl/ca.private.rsa.4096.key.pem -text
# or
openssl pkey -in ~/.vnc/ssl/ca.private.eddsa.ed25519.key.pem -text
# or
openssl pkey -in ~/.vnc/ssl/ca.private.eddsa.ed448.key.pem -text

Generate a signing CA, make it valid for 2 years and adding the server IP (this is required, change 88.44.88.33 with your IP):

# RSA 4096-bits 
openssl req -new -x509 -days 730 -key ~/.vnc/ssl/ca.private.rsa.4096.key.pem -out ~/.vnc/ssl/ca.sign.rsa.4096.key.pem -subj '/CN=88.44.88.33' -addext "subjectAltName=IP:88.44.88.33"

#Or ED25519 openssl req -new -x509 -days 730 -key ~/.vnc/ssl/ca.private.eddsa.ed448.key.pem -out ~/.vnc/ssl/ca.sign.eddsa.ed448.key.pem -subj '/CN=88.44.88.33' -addext "subjectAltName=IP:88.44.88.33"

Or ED448

openssl req -new -x509 -days 730 -key ~/.vnc/ssl/ca.private.eddsa.ed448.key.pem -out ~/.vnc/ssl/ca.sign.eddsa.ed448.key.pem -subj '/CN=88.44.88.33' -addext "subjectAltName=IP:88.44.88.33"

Update the server configuration to use X509:

# Security, specify which security scheme to use (None, VncAuth, Plain,
# TLSNone, TLSVnc, TLSPlain, X509None, X509Vnc, X509Plain) (default=TLSVnc,VncAuth)
securitytypes=X509Vnc

Path to the key of the X509 certificate in PEM format (default=)

X509Key=/home/USER/.vnc/ssl/ca.private.eddsa.ed448.key.pem

Path to the X509 certificate in PEM format (default=)

X509Cert=/home/USER/.vnc/ssl/ca.sign.eddsa.ed448.key.pem

Connecting with TigerVNC Viewer:

  • Copy ca.sign.eddsa.ed448.key.pem to the client
  • On TigerVNC Viewer under Options > Security, on the encryption section leave only TLS with X509 certificates selected and put the path to ca.sign.eddsa.ed448.key.pem on Path to X509 CA certificate (leave CRL section empty)
  • On the authentication section select only Standard VNC

Checking the GnuTLS session's handshake algorithms

TigerVNC is using GnuTLS for the encryption, on the server/client the setting GnuTLSPriority sets the priority string that controls the TLS session's handshake algorithms (TLS1.0/TLS1.1/TLS1.2/TLS1.3/etc.); supported algorithms can be listed with gnutls-cli --list

We can for instance test the TLS v1.2 support with the following:

vncviewer GnuTLSPriority=NORMAL:-VERS-ALL:+VERS-TLS1.2 -log='*:stdout:100' 

Here is how we can enforce TLS v1.2/v1.3 on the server config file:

# GnuTLS priority string that controls the TLS session’s handshake algorithms.  
# See the GnuTLS manual for possible values. Default is NORMAL.

Only TLS v1.2

#GnuTLSPriority=NORMAL:-VERS-ALL:+VERS-TLS1.2

Only TLS v1.3

GnuTLSPriority=NORMAL:-VERS-ALL:+VERS-TLS1.3

Verifying if only TLS v1.2/v1.3 policy is working with the following

vncviewer GnuTLSPriority=NORMAL:-VERS-TLS1.2 -log='*:stdout:100' # v1.2

vncviewer GnuTLSPriority=NORMAL:-VERS-TLS1.3 -log='*:stdout:100' # v1.3

This mean use all but v1.2/v1.3 to test if the setting is enforced correctly,

and thus the connection will be refused for handshake algorithm mismatch.

Other example of gnutlspriority values (warning, this is just for the syntax)

#GnuTLSPriority=NORMAL:-VERS-SSL3.0:-VERS-TLS1.0:-VERS-TLS1.1:-VERS-TLS1.3 #NORMAL:+SECURE128:-SHA384:-SHA256:-VERS-SSL3.0:-VERS-TLS1.0:-VERS-TLS1.1 #NORMAL:+VERS-TLS1.2:+VERS-TLS1.3:+AES-128-CBC:+RSA:+SHA1:+COMP-NULL


Updating OpenSSL and GnuTLS

OpenSSL and GnuTLS are two main application/library of most Linux system, updating them manually may introduce security issues as they will not be updated automatically anymore, we can limit that side negative effect by using a custom version just for our purpose (TigerVNC).

A newer OpenSSL version is needed to generate a Ed25519/ED448 key, while GnuTLS is needed for vncviewer(client's viewer) / Xvnc (TigerVNC server) to add support for Ed25519/ED448 algorithms

Check what algorithm is currently supported:

#GnuTLS
gnutls-cli --list | grep EdDSA

#OpenSSL man -P cat genpkey | grep "Valid built-in algorithm" openssl list -public-key-algorithms | grep ED

#If EdDSA targeted algorithm is supported there is no need to install from sources

Build, install and use OpenSSL:

wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
tar -xvf openssl-1.1.1g.tar.gz
cd openssl-1.1.1g/
./config no-nextprotoneg no-weak-ssl-ciphers no-ssl3 no-shared -DOPENSSL_NO_HEARTBEATS -fstack-protector-strong enable-tls1_3
make install -j2

After install, OpenSSL can be used

for instance as follow to generate the needed key

/usr/local/bin/openssl genpkey -algorithm ED448 -out ~/.vnc/ssl/ca.private.eddsa.ed448.key.pem

Build, install and use GnuTLS:

wget https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/gnutls-3.6.14.tar.xz
tar -xvf gnutls-3.6.14.tar.xz
cd gnutls-3.6.14/
./configure --without-tpm --disable-tests --disable-full-test-suite --disable-non-suiteb-curves --disable-ssl2-support
make install -j2

This is required for Xvnc and vncviewer, after install can be used

by setting LD_LIBRARY_PATH before running vncviewer or Xvnc,

like the following example

TigerVNC Server

export LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/usr/lib vncserver

TigerVNC Viewer

env LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/usr/lib vncviewer

To check the used version we can use ldd for instance

ldd /usr/bin/vncviewer


Experiments notes

While using KDE with TigerVNC server; when stopping/starting the server several times, some X applications remain running, probably related to this systemd issue, to fix that the following script can be used to stop the server (can be used on the systemd service config file as well), and ps aux | sort | grep USER-NAME | grep -v '\[' command can be used to check if something is still running after stopping the server.

#cat /usr/bin/vncserver-stop (this is meant for KDE)

#!/bin/sh

vncserver -kill :0 vncserver -kill :1 vncserver -kill :2 vncserver -kill :3

killall Xvnc -9 killall kwin_x11 -9 killall startplasma-x11 -9 killall plasma_session -9

killall Xvnc -9 killall kwin_x11 -9 killall startplasma-x11 -9 killall plasma_session -9

After several tests with the viewer, the best speed performances were achieved with vncviewer -QualityLevel=4 -CompressLevel=2 -PreferredEncoding=Raw

If the server needs to be exposed to the internet, note that VNC Server can be easily discovered with an nmap scan (nmap -sV -sC TARGET-IP), VNC protocol need to announce itself and thus without creating a custom version of the server and client there is not much than can be done to obfuscate a VNC server in the case of a global internet exposure.


Documentations

intika
  • 14,406