Let's Encrypt Certs Fail "openssl verify" Verification

Question

I've followed the guidance from other Stack answers- there's a gazillion related to this- in building my openssl verify command to validate my Let's Encrypt certs, shown below:

openssl verify -show_chain /etc/letsencrypt/live/mail.example.com/chain.pem /etc/letsencrypt/live/mail.example.com/cert.pem 

But it fails with the error:

CN = mail.example.com
error 20 at 0 depth lookup: unable to get local issuer certificate
error /etc/letsencrypt/live/mail.example.com/cert.pem: verification failed
/etc/letsencrypt/live/mail.example.com/chain.pem: OK
Chain:
depth=0: C = US, O = Let's Encrypt, CN = R3 (untrusted)
depth=1: C = US, O = Internet Security Research Group, CN = ISRG Root X1

Even if I substitute fullchain.pem for chain.pem this nonetheless fails. But these are all the certs Let's Encrypt distributed to me!

What am I missing here?

Answer 1

openssl verify -show_chain /etc/letsencrypt/live/mail.example.com/chain.pem /etc/letsencrypt/live/mail.example.com/cert.pem 

This command is wrong. It will try to verify all the given certificates independently from each other, i.e. not build a trust chain and verify the first. Instead the command should have been:

openssl verify -untrusted chain.pem cert.pem

With -untrusted the intermediate certificate will be given. The root certificate ISRG X1 will be taken from the trust store in modern systems, otherwise it should be given with -trusted or -CAfile.

Answer 2

Updated Answer:

I'm marking @SteffenUllich 's answer as the correct one, but I'll just expand on parts of it in relation to what's inside the bundles and the Trust Store he references in his answer.

Files Let's Encrypt Gives You:

The error appeared to indicate that I had a broken Chain of Trust and was missing at least one cert. But I couldn't know what I was missing until I understood what certs that I had.

So I reviewed the files Let's Encrypt gave me which live in the path /etc/letsencrypt/live/example.com/ and found the bundles were comprised of:

cert.pem: The server's cert
chain.pem: Let's Encrypt's "R3" cert + "ISRG Root X1" ("Intermediate Cert")
fullchain.pem: "chain.pem" + "cert.pem"
privkey.pem: Your Private key. The Public key is encoded into cert.pem

NOTE: To validate the above using a one-liner, replace mail.example.com with your own cert name in the path:

while openssl x509 -noout -text; do :; done < /etc/letsencrypt/live/mail.example.com/cert.pem
while openssl x509 -noout -text; do :; done < /etc/letsencrypt/live/mail.example.com/chain.pem
while openssl x509 -noout -text; do :; done < /etc/letsencrypt/live/mail.example.com/fullchain.pem

Trust Store:

It appeared that my initial attempt to validate the server cert lacked a "Trust Anchor" enabling the chain to be traced back to the origin.

HOWEVER: If the -CAfile is not specified, as Steffen notes openssl will parse the "Trust Store" to find a root cert to complete the Chain of Trust.

To install a "Trust Store", search for package "ca-certificates" / "ca-certificates-bundle" in your distro.

So since the CA certs are already there, nothing additional need be downloaded for the chain to be traced successfully as I erroneously stated in my original answer. Indeed, the verification is more likely to succeed if you leave openssl to just parse everything in the Trust Store until it finds something that matches.

To find the location of your "Trust Store":

openssl version -d

In Alpine Linux, this is /etc/ssl

Conclusion:

Thanks to Steffen for bringing to my attention that the command I issued succeeded for the wrong reason ;-). And openssl documentation could be a bit better to be fair...