Cannot do a minor upgrade on Debian 8.11 in order to then do a major upgrade

Question

I am trying to upgrade my VPS which has Debian 8.11 Jessie installed. In order to do major upgrade I first do a minor upgrade. I have done the following, which I know is the usual route:

sudo apt-get update
sudo apt-get upgrade
sudo sed -i 's/jessie/stretch/g' /etc/apt/sources.list
sudo apt-get update

Everything goes well and then I run:

sudo apt-get upgrade

And I am receiving the following errors:

The following packages have unmet dependencies:
 libmariadb3 : Breaks: libmariadbclient18 (< 1:10.3.12-2) but 10.1.37-0+deb9u1 is to be installed
 libssl1.1 : Breaks: python-httplib2 (< 0.11.3-1) but 0.9.2+dfsg-1 is to be installed
E: Broken packages

I tried removing python-httplib2 but as usual apt wants to remove vital packages on the system that would certainly cause it to permanently break. I do not know about the libmariadb3 package but I use MySQL on a production website so I assume it will want to remove all sorts to do that.

apt cache-policy

Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://ftp.us.debian.org/debian testing/non-free amd64 Packages
     release o=Debian,a=testing,n=buster,l=Debian,c=non-free,b=amd64
     origin ftp.us.debian.org
 500 http://ftp.us.debian.org/debian testing/contrib amd64 Packages
     release o=Debian,a=testing,n=buster,l=Debian,c=contrib,b=amd64
     origin ftp.us.debian.org
 500 http://ftp.us.debian.org/debian testing/main amd64 Packages
     release o=Debian,a=testing,n=buster,l=Debian,c=main,b=amd64
     origin ftp.us.debian.org
 500 http://ftp.us.debian.org/debian stretch-updates/main amd64 Packages
     release o=Debian,a=stable-updates,n=stretch-updates,l=Debian,c=main,b=amd64
     origin ftp.us.debian.org
 500 http://security.debian.org stretch/updates/main amd64 Packages
     release v=9,o=Debian,a=stable,n=stretch,l=Debian-Security,c=main,b=amd64
     origin security.debian.org
 500 http://ftp.us.debian.org/debian stretch/main amd64 Packages
     release v=9.7,o=Debian,a=stable,n=stretch,l=Debian,c=main,b=amd64
     origin ftp.us.debian.org
Pinned packages:

sources.list

deb http://ftp.us.debian.org/debian stretch main
deb http://security.debian.org/ stretch/updates main
deb http://ftp.us.debian.org/debian stretch-updates main
deb http://ftp.us.debian.org/debian testing main contrib non-free

I have removed deb http://ftp.us.debian.org/debian testing main contrib non-free from my sources and was able to remove libmariadb3 as I realise that when I upgrade I can just reinstall any packages to do with this (default-libmysqlclient-dev libmariadb-dev libmariadb-dev-compat libmariadb3) but trying to remove libssl1.1 (which is manually compiled from source) is going to do harm to my system as it wants to remove:

apt-transport-https autoconf automake bind9-host build-essential ca-certificates cloud-init cloud-initramfs-growroot cloud-utils curl dh-python dnsutils
  dpkg-dev euca2ools ghostscript git host libalgorithm-c3-perl libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libarchive-extract-perl
  libauthen-sasl-perl libbind9-90 libcgi-fast-perl libcgi-pm-perl libclass-accessor-perl libclass-c3-perl libclass-c3-xs-perl libcpan-meta-perl libcups2
  libcupsfilters1 libcupsimage2 libcurl3-gnutls libcurl4 libdata-optlist-perl libdata-section-perl libdbd-mysql-perl libdbi-perl libdns100 libdpkg-perl
  libencode-locale-perl liberror-perl libfcgi-perl libfile-fcntllock-perl libfile-listing-perl libfont-afm-perl libgs9 libgssapi-krb5-2 libgtk2.0-0 libgtk2.0-bin
  libhtml-form-perl libhtml-format-perl libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl libhtml-tree-perl libhttp-cookies-perl libhttp-daemon-perl
  libhttp-date-perl libhttp-message-perl libhttp-negotiate-perl libintl-perl libio-html-perl libio-socket-ssl-perl libio-string-perl libisccfg90 libkrb5-3
  liblog-message-perl liblog-message-simple-perl liblwp-mediatypes-perl liblwp-protocol-https-perl libmailtools-perl libmodule-build-perl
  libmodule-pluggable-perl libmodule-signature-perl libmro-compat-perl libnet-http-perl libnet-smtp-ssl-perl libnet-ssleay-perl libpackage-constants-perl
  libparams-util-perl libparse-debianchangelog-perl libperl4-corelibs-perl libpod-latex-perl libpod-readme-perl libpython3-stdlib libpython3.5-minimal
  libpython3.5-stdlib libregexp-common-perl libsoftware-license-perl libssl-dev libssl1.1 libsub-exporter-perl libsub-install-perl libsub-name-perl
  libswitch-perl libterm-readkey-perl libterm-ui-perl libtext-soundex-perl libtext-template-perl libtext-unidecode-perl libtimedate-perl libtirpc1 liburi-perl
  libwww-perl libwww-robotrules-perl libxml-libxml-perl libxml-namespacesupport-perl libxml-parser-perl libxml-sax-base-perl libxml-sax-expat-perl
  libxml-sax-perl lsof mutt mysql-client-5.5 mysql-server mysql-server-5.5 mysqltuner nfs-common openssh-client openssh-server openssh-sftp-server openssl perl
  perl-modules pinentry-gtk2 python-boto python-chardet-whl python-pip python-pip-whl python-reportbug python-requestbuilder python-requests python-requests-whl
  python-software-properties python-virtualenv python3 python3-apt python3-minimal python3-pkg-resources python3-virtualenv python3.5 python3.5-minimal rename
  reportbug rpcbind sendmail sendmail-base sendmail-bin texinfo unattended-upgrades virtualenv

Answer 1

WARNING!! Do not do what I did without first understanding the potential risks! You have been warned OK so I just thought I would 'risk it for a biscuit' and allow apt to remove what it wanted to remove. Luckily, the world did end and my system is now functioning well. To elaborate on what I did:

sudo apt remove libmariadb3
sudo dpkg -r --force-depends libssl1.1 <-- this removed the big list of packages in my question
sudo apt update
sudo apt upgrade
sudo apt dist-upgrade

I will install any packages I need as I go along.

I do not really recommend doing this way

Answer 2

What went wrong

Looks like what you currently have is mostly Debian 8 (jessie), but your MySQL/MariaDB and python-httplib2 has already been updated to the current testing (i.e. what will eventually become Debian 10, buster).

When you initially added testing to your sources.list it may have been synonymous to stretch (= Debian 9, the current stable version), but the very day Debian 9 was released, testing also jumped ahead to point to the next version, and so your "minor update" (your initial sudo apt-get update + sudo apt-get upgrade) probably updated at least some packages to buster version levels.

That's the danger of using testing instead of release codenames like stretch or buster: you would need to update your repository configuration exactly at the release day, or you'll easily get an unplanned, possibly partial upgrade unless you're extremely careful.

What to do next

The following command might be helpful in identifying all packages that are not on the version level of the current stable (assuming it works in Debian 8; I'm not sure if it was available in it):

apt list --installed | grep -v /stable

Run grep upgrade /var/log/dpkg.log to see the most recent package upgrades, and check their versions. Find all the packages that have been accidentally upgraded to the buster/testing versions, and downgrade (don't remove) them back to the correct versions.

The log messages will look like:

<timestamp> upgrade <package name> <old version> <new version>

so you can see what the previous versions were. If a package's old version was already appropriate for stretch (i.e. it was installed from testing while Debian 8 was stable) , then downgrade that package back to stretch's current level:

sudo apt-get -t=stretch <package name>

or if you need to specify an exact version number:

sudo apt-get install <package-name>=<package-version-number>