Elisp regexps are represented as strings, which means backslashes are interesting, as they are not only special to regexps, but also when writing strings.
Emacs requires a literal \ character to be escaped in the double-quoted read syntax for strings and so, when the code is processed by the lisp reader, "\\" becomes a string object containing a single \ character; and hence that single backslash is what the regexp engine sees when it uses that string object.
So in your instance, the regexp a\|b is represented by "a\\|b" in the double-quoted read syntax for strings.
Conversely "a\|b" is the regexp a|b (because \| is not a special construct in the read syntax for strings, so all we have done there is needlessly escaped a | character; hence "a\|b" is no different to "a|b"), and a|b contains no regexp-special constructs, so it matches the three-character sequence a|b literally.
The elisp manual explains further:
`\' has two functions: it quotes the special characters (including
`\'), and it introduces additional special constructs.
Because `\' quotes special characters, `\$' is a regular
expression that matches only `$', and `\[' is a regular expression
that matches only `[', and so on.
Note that `\' also has special meaning in the read syntax of Lisp
strings (*note String Type::), and must be quoted with `\'. For
example, the regular expression that matches the `\' character is
`\\'. To write a Lisp string that contains the characters `\\',
Lisp syntax requires you to quote each `\' with another `\'.
Therefore, the read syntax for a regular expression matching `\'
is `"\\\\"'.
-- C-hig (elisp)Regexp Special
It is also worth noting that \ is not special within a character alternative (this is also true for most other regexp-special characters), and therefore [\] (aka "[\\]") matches a backslash.
As a `\' is not special inside a character alternative, it can never
remove the special meaning of `-' or `]'. So you should not quote
these characters when they have no special meaning either. This would
not clarify anything, since backslashes can legitimately precede these
characters where they _have_ special meaning, as in `[^\]' (`"[^\\]"'
for Lisp string syntax), which matches any single character except a
backslash.