2

My company has multiple servers that were setup to be identical. After using dev32 for several weeks, which stopped functioning. I go to dev35 and get similar behavior -- works for several weeks then one day it stops. dev36 is ok for now, but following this pattern I'll eventually run out of systems to use.

Any thoughts on what could be causing this behavior?

dev32 ~> which python3
python3: Command not found.
dev35 ~> which python3
python3: Command not found.
dev36 ~> which python3
/bin/python3

dev32 ~> ls -al /bin/python*
lrwxrwxrwx. 1 root root     9 Jan 17 10:18 /bin/python3 -> python3.6
-rwxr-x---. 2 root root 11328 Nov 16  2020 /bin/python3.6
-rwxr-x---. 2 root root 11328 Nov 16  2020 /bin/python3.6m
dev32 ~> echo $PATH
.:/bin:/sbin:/usr/bin:/usr/bin/X11:/usr/bsd:/usr/sbin:/home/<username>/bin:/usr/local/bin:/usr/local/etc:/usr/local/sbin:/etc:/usr/etc:/usr/gnu/bin


dev35 ~> ls -al /bin/python3*
lrwxrwxrwx. 1 root root     9 Jan 17 10:11 /bin/python3 -> python3.6
-rwxr-x---. 2 root root 11328 Nov 16  2020 /bin/python3.6
-rwxr-x---. 2 root root 11328 Nov 16  2020 /bin/python3.6m
dev35 ~> echo $PATH
.:/bin:/sbin:/usr/bin:/usr/bin/X11:/usr/bsd:/usr/sbin:/home/<username>/bin:/usr/local/bin:/usr/local/etc:/usr/local/sbin:/etc:/usr/etc:/usr/gnu/bin


dev36 ~> ls -al /bin/python*
lrwxrwxrwx. 1 root root     9 Jan 17 10:18 /bin/python3 -> python3.6
-rwxr-x---. 3 root root 11328 Nov 16  2020 /bin/python3.6
-rwxr-x---. 2 root root 11328 Nov 16  2020 /bin/python3.6m
dev36 ~> echo $PATH
.:/bin:/sbin:/usr/bin:/usr/bin/X11:/usr/bsd:/usr/sbin:/home/<username>/bin:/usr/local/bin:/usr/local/etc:/usr/local/sbin:/etc:/usr/etc:/usr/gnu/bin

Even after cd /bin, which cannot find python3

Other knowns:

  • All servers are CentOS 7.9, uname -a matches
  • Ran rpm -qa | sort > $HOSTNAME.txt on each server and compared. results are identical
  • bash v4.2.26, tcsh v6.18.01 (same across all systems)
  • Behavior confirmed by multiple users; tcsh & bash yield same results. One user's shell uses system defaults (no ~/.bashrc, ~/.tcshrc or ~/.login), I customize mine.
  • IT says they haven't changed anything beyond standard monthly patching that is applied consistently to all servers.
  • Problem persists across reboots

Please let me know if more information is needed.

JB Looney
  • 21
  • 1
    What does your ls report on the two failing systems? – Chris Davies May 17 '23 at 14:40
  • 1
    On dev32 ls /bin and echo "$PATH". Also take a look in the package manager's log. –  May 17 '23 at 14:40
  • 5
    Having . as the first element in $PATH is a potential security issue. It's better not to include it at all, but if you must then put it at the end of the list – Chris Davies May 17 '23 at 14:41
  • 2
    stupid question, does simply running python3 --version still work on all three machines? – Marcus Müller May 17 '23 at 14:53
  • 1
    The . at the end of the mode string from ls suggests there is an SELinux context - I wonder if that is a factor? See for example What does a dot after the file permission bits mean? – steeldriver May 17 '23 at 16:11
  • 2
    Is it just the which command that's misbehaving (in which case an obvious workaround is to stop using it), or do saner commands (type, command -v, …) also not find exeutables? – Gilles 'SO- stop being evil' May 17 '23 at 16:32
  • @roima & @tomasz - Added requested info to description

    @roima - Thanks for the tip on the '.' in my $PATH. I'll fix that.

    Marcus Muller - Running python3 --version on all 3x systems returns '3.6.8'

    @steeldriver - I'll read up on that next

     – JB Looney May 18 '23 at 13:48
  • Gilles - The part that gets me is using 'which' used to work, but recently stopped working, and only on a couple of systems. As these systems are supposed to be identical, and I'm getting inconsistent behavior, I'd like to track down what could possibly be different (in case there are other side effects I haven't noticed yet, and needs fixed). – JB Looney May 18 '23 at 13:54
  • Are you comparing the results for the same user? –  May 18 '23 at 14:01
  • @tomasz - Yes. Those results were all run as myself. I can try as another user, but hadn't thought to try since others reported the same behavior. – JB Looney May 18 '23 at 14:03
  • With these persmissions (-rwxr-x---) and root:root there's likely not much to relate to here. I feel this question is steering into an unclear direction. –  May 18 '23 at 14:12

0 Answers0