3

I'm using a script for Linux and for Solaris (Nexenta).

This line works on Linux, but not on solaris (but when run from shell it works):

cat "pg_hba.conf" | sed "0,/^local/{s/md5/trust/}"

The error message is:

sed: command garbled: 0,/^local/{s/md5/trust/}

After some research, I found out that the sed that bash uses in the script is different.

from shell: /usr/bin/sed

from script: /usr/sun/bin/sed

I want to make the script use /usr/bin/sed.

What I tried to do:

  1. call sed with full path. Same results. It seems that it is still uses the other sed...
  2. tried to call it via bash -l. Same results.
  3. tried to declare a different command: S=/usr/lib/sed and use $S instead. Same results.
  4. checked PATH - both cmd and script have /usr/bin in it.
  5. Tried replacing the double quotes to single. Same results.
  6. tried running the sed with -r flag. Output is :

    # /usr/xpg4/bin/sed -r /usr/xpg4/bin/sed: illegal option -- r Usage: sed [-n] script [file...] sed [-n] [-e script]...[-f script_file]...[file...]

HELP??

What I need to do is to replace the first match of "md5" with "trust" on the first line that starts with "local"). I know I can do it otherwise but I the sed issue is itching me too much!

EDIT:

I hope this makes a little order...

  1. PATH from login shell = /usr/local/ctera/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/ctera/apache-ant-1.8.2/bin
  2. PATH from script =
    /usr/sbin:/usr/bin:/usr/local/ctera/apache-ant-1.8.2/bin:/usr/local/ctera/apache-ant-1.8.2/bin
  3. PATH from script when PATH=$(command -p getconf PATH):$PATH is called = /usr/xpg4/bin:/usr/ccs/bin:/usr/bin:/opt/SUNWspro/bin:/usr/xpg4/bin:/usr/ccs/bin:/usr/bin:/opt/SUNWspro/bin:/usr/sbin:/usr/bin:/usr/local/ctera/apache-ant-1.8.2/bin:/usr/local/ctera/apache-ant-1.8.2/bin
  4. truss -f sed from login shell calls /usr/bin/sed
  5. truss -f sed from script calls /usr/sun/bin/sed
  6. truss -f /usr/bin/sed from script calls /usr/sun/bin/sed !!!
  7. After setting PATH=$(command -p getconf PATH):$PATH:
    7.1 truss -f sed from script calls /usr/xpg4/bin/sed
    7.2 truss -f /usr/bin/sed from script calls /usr/sun/bin/sed !!!

MORE INFO:

Commands output: (run both from shell prompt and from within the script)

  1. truss -ft execve /usr/bin/sed q
    as shell command:
    8604: execve("/usr/bin/sed", 0x08047D20, 0x08047D2C) argc = 2
    from script:
    8545: execve("/usr/sun/bin/sed", 0x08047768, 0x08047774) argc = 2

  2. file /usr/bin/sed
    as shell command:
    /usr/bin/sed: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), stripped
    from script:
    file: /usr/bin/sed zero size or zero entry ELF section - ELF capabilities ignored file: /usr/bin/sed: can't read ELF header /usr/bin/sed: data

  3. ls -l /usr/bin/sed
    as shell command:
    -rwxr-xr-x 1 root root 96440 May 31 2008 /usr/bin/sed
    from script:
    -rwxr-xr-x 1 root root 96440 May 31 2008 /usr/bin/sed

  4. ls -ld $(type -pa sed)
    as shell command:
    -rwxr-xr-x 1 root root 96440 May 31 2008 /bin/sed -rwxr-xr-x 1 root root 96440 May 31 2008 /usr/bin/sed
    from script:
    -rwxr-xr-x 1 root root 96440 May 31 2008 /usr/bin/sed

  5. md5sum $(type -pa sed)
    as shell command:
    385361c5111226c8eac8e25b53fed29c /bin/sed 385361c5111226c8eac8e25b53fed29c /usr/bin/sed
    from script:
    385361c5111226c8eac8e25b53fed29c /usr/bin/sed

    • The script is invoced by JAVA code.

    • uname -a
      SunOS cteraportal 5.11 NexentaOS_134f i86pc i386 i86pc Solaris

    • This might add info about the sed version on my machine

      ~# ll `find / -name sed`  
      -rwxr-xr-x 1 root root 96440 May 31  2008 /usr/bin/sed  
      -r-xr-xr-x 1 root bin  35656 Sep  7  2010 /usr/sun/bin/sed  
      -r-xr-xr-x 1 root bin  32104 Sep  7  2010 /usr/ucb/sed  
      -r-xr-xr-x 1 root bin  35636 Sep  7  2010 /usr/xpg4/bin/sed  
      
      /usr/share/doc/sed:  
      total 113  
      -rw-r--r-- 1 root root   168 Jun 21  2005 AUTHORS.gz  
      -rw-r--r-- 1 root root  2507 Jun 21  2005 BUGS.gz  
      -rw-r--r-- 1 root root  6584 Feb  3  2006 NEWS.gz  
      -rw-r--r-- 1 root root   285 Jun 21  2005 README.gz  
      -rw-r--r-- 1 root root  1071 Jan 12  2006 THANKS.gz  
      -rw-r--r-- 1 root root  4806 May 31  2008 changelog.Debian.gz  
      -rw-r--r-- 1 root root 32312 Feb  3  2006 changelog.gz  
      -rw-r--r-- 1 root root   796 May 31  2008 copyright  
      drwxr-xr-x 2 root root     3 May 30  2011 examples  
      drwxr-xr-x 2 root root     3 May 30  2011 sed-4.1.5  
      -rw-r--r-- 1 root root 56538 May 31  2008 sedfaq.txt.gz  
      
csny
  • 1,505
  • Your sed command may be syntactically incorrect. Usually when you call sed it would be followed by a flag. For example: sed 's/cat/dog/g' – ryekayo Aug 18 '14 at 15:50
  • But works when run from cmd – csny Aug 18 '14 at 15:52
  • Can you try replacing the double quotes with single quotes? I'm guessing that syntax may have something to do with this, but im not a 100%. – ryekayo Aug 18 '14 at 15:53
  • Tried that as well :) Same results... I'll edit my post with this suggestion included – csny Aug 18 '14 at 15:54
  • May I ask what the 0, is supposed to do? – ryekayo Aug 18 '14 at 15:54
  • It searches all the lines from the first until ^local is found – csny Aug 18 '14 at 15:57
  • What do you mean by from cmd? Are you saying that /usr/lib/sed '0,/^local/{s/trust/md5/}' works at the shell prompt, but not in a script? – Stéphane Chazelas Aug 18 '14 at 16:08
  • yes, that's what i meant – csny Aug 18 '14 at 16:13
  • What is the value of the PATH environment variable? It's not enough to know that /usr/bin is on it, it matters what else is there and in what order. Are you sure you have a /usr/lib/sed? That's highly unusual. And what's this about illigal option — did you copy-paste that or did you mis-type it? Always copy-paste. – Gilles 'SO- stop being evil' Aug 18 '14 at 21:22
  • I edited the post, I think it answers all the questions – csny Aug 19 '14 at 07:28
  • Can you post the exact output of truss -ft execve /usr/bin/sed q, of file /usr/bin/sed, of ls -l /usr/bin/sed. of type -a sed? , of ls -ld $(type -a sed) and of md5sum $(type -a sed)? – Stéphane Chazelas Aug 19 '14 at 09:17
  • Sorry, meant $(type -pa sed) above (assuming bash is the shell). – Stéphane Chazelas Aug 19 '14 at 09:37
  • @Stéphane Chazelas, do you want the output of the commands run as shell commands, or inside the script? – csny Aug 19 '14 at 09:49
  • At least at the shell prompt, both if you can. I can trim it to keep only the interesting parts if you wish. – Stéphane Chazelas Aug 19 '14 at 10:49
  • Got both in the edited post. This is very interesting. In the script it acts as if there is no different sed from the one run from shell, but it actually uses the Solaris version... – csny Aug 19 '14 at 11:31
  • Is your script run in a different Solaris zone? How is it invoked? Can you upload your /usr/bin/sed somewhere? What version of Nexenta is it? I don't understand how truss /usr/bin/sed can not show execve("/usr/bin/sed",...) – Stéphane Chazelas Aug 19 '14 at 15:34
  • See another edit of the post :) That's what this post about. How is it possible that a different sed is called... The same problematic sed replacement works via shell prompt. So there is a "good" sed on my system, which is the /usr/bin/sed, that just isn't called in the script. Some magic going on... – csny Aug 19 '14 at 15:56

3 Answers3

2

In any case,

sed "0,/^local/{s/md5/trust/}"

Is GNU specific (the 0 address and the missing ; before }) and won't work with any other sed implementation (and Solaris doesn't ship with GNU sed by default).

Portably/standardly:

sed '/^local/,$!s/md5/trust/'

to replace only on the lines up to (but not included) the first one starting with local. Or:

awk 'NR == 1, /^local/ {gsub(/md5/,"trust")}; {print}'

(on Solaris, you may need command -p awk).

If you want the substitution on the first line that matches /^local/:

awk '/^local/ && ! seen {gsub(/md5/, "trust"); seen = 1}; {print}'

Or:

sed -e '/^local/!b' -e 's/md5/trust/g;:1' -e 'n;b1'

To be sure to get POSIX compliant utilities in both Solaris and Linux (from a POSIX shell like bash or ksh (or /usr/xpg4/bin/sh on Solaris)), you can add:

PATH=$(command -p getconf PATH):$PATH

to the top of the script. Or add command -p in front of every command which you want the POSIX version of.

  • Thanks for the help, sed -e '/^local/!b' -e 's/md5/trust/g;:1' -e 'n;b1' works well. But I still want to understand why wouldn't the GNU compatible line work in the script, but works from shell. And why I can't make it use /usr/lib/sed... – csny Aug 18 '14 at 16:37
  • 1
    @csny, does /usr/lib/sed --version work and mention it's GNU? Does truss -f /usr/lib/sed shows it eventually executing another sed somewhere? (like /usr/lib/sed being a wrapper that calls this or that sed based on the environment)? Is /usr/lib/sed a script? – Stéphane Chazelas Aug 18 '14 at 16:39
  • Sorry, I meant /usr/bin/sed... – csny Aug 18 '14 at 16:46
  • The version is: GNU sed version 4.1.5 – csny Aug 18 '14 at 16:47
  • I added truss -f sed to the script and it does show that it isn't /usr/bin/sed, but it is /usr/xpg4/bin/sed. – csny Aug 18 '14 at 16:52
  • I edited the post, I think it answers all the questions – csny Aug 19 '14 at 07:30
  • @StéphaneChazelas - do you think maybe this guy is running a csh script accidentally - like in your which which question/answer? – mikeserv Aug 19 '14 at 15:26
2

OK, found it. That now makes sense.

the behaviour is Nexenta-specific and explained at http://lwn.net/Articles/334756/

GNU and not GNU

The default behavior of Nexenta is to prefer GNU utilities, which are installed in /usr/bin and /usr/sbin and so on. The Sun versions of these utilities are installed in /usr/sun/bin and /usr/sun/sbin. Nexenta uses a trick to be able to switch between a GNU and a SUN personality: if the environment variable SUN_PERSONALITY is set to one, the search paths /usr/sun/bin and /usr/sun/sbin take preference, even if the user executes the commands explicitly by their absolute path, e.g. /usr/bin/sed. This ensures that Solaris-based scripts work in Nexenta without modifications. Nexenta also uses this functionality in its SVR4 package commands. They can be used to install native Solaris packages in SVR4 format, calling alien to convert the package on-the-fly to a Debian package.

That's done somewhere in the libc.

So,

$ sed --version
GNU sed version 4.1.5
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE,
to the extent permitted by law.
$ SUN_PERSONALITY=1 sed --version
sed: illegal option -- version

So, your script started from Java, must have the SUN_PERSONALITY set.

You can unset that in your script if you want the GNU tools.

  • 2
    I had similar suspicions about the POSIXLY_CORRECT and or --posix switches and GNU sed's behavior in their presence, but came up dry in tests and gave up. Still - I did learn about this command: sed v ...does nothing... but sed fails if GNU sed extensions are not supported... because other seds do not implement it. ...you can specify the version... required, such as 4.0.5. The default is 4.0...v enables all GNU extensions even if POSIXLY_CORRECT is set in the environment. – mikeserv Aug 20 '14 at 03:38
  • unset SUN_PERSONALITY works, but messes other things up, as I apparently rely on SUN functionality in other stuff. So I unset and export it back to 1, and everything works. THANKS! – csny Aug 20 '14 at 08:59
  • @csny, I would use POSIX syntax in your script. Both Sun (at least after PATH=$(getconf PATH):$PATH) and Gnu should be mostly POSIX compatible. – Stéphane Chazelas Aug 20 '14 at 09:01
  • 1
    To anyone who reads this: Just wanted to point out that SUN_PERSONALITY is a Nexenta invention, not something that exists in Solaris as such. – peterh Aug 21 '14 at 11:54
  • @nolan6000, I agree it should be pointed out. I've updated my answer to make it clearer and changed the title of the question s/Solaris/Nexenta/ – Stéphane Chazelas Aug 21 '14 at 12:04
0

Here's another portable way to do it with sed:

sed 1i\\ file |
sed '1,/^local/s/md5/trust/;1d'

Just give yourself a little breathing room. That inserts a blank line at the head of the file so you can rely upon a thing or two.

You might also do:

{ echo; cat file; } |
sed '1,/^local/s/md5/trust/;1d'
mikeserv
  • 58,310
  • Thanks, I already have a workaround for this, but I still can't understand why isn't the script acting like the shell prompt, when calling sed... – csny Aug 19 '14 at 14:48
  • @csny - Did you try command -V sed at the prompt and in a script/non-interactive shell? Maybe alias | grep sed= also from the interactive shell? – mikeserv Aug 19 '14 at 14:53
  • maybe also readlink /usr/bin/sed – mikeserv Aug 19 '14 at 14:59
  • Wait... @csny - is there a shebang line in that script? At the top? Is the shell that is executing that script definitely bash? What happens when you do lsof "$0" while in the script? – mikeserv Aug 19 '14 at 15:06
  • #!/bin/bash is at the top. the shell is bash as well. There is no lsof on my Nexenta, but the output of ps -ef include this: root 4949 4498 0 08:23:39 ? 0:00 /bin/bash /usr/local/ctera/bin/ctera.sh initdb. No aliases, and no readlink... command -V sed results in /usr/bin/sed. – csny Aug 19 '14 at 15:29
  • @csny You're root in the script or you're root at the prompt? Seriously - this is weird. What do you get with printf %s\\n '#!/usr/bin/sed' '0,/^local/{s/md5/trust/}' > ~/sedscript; chmod +x ~/sedscript; ~/sedscript? Can you watch that with ps? – mikeserv Aug 19 '14 at 15:43
  • I'm always root. JAVA is run as root, and calls this script. But do you think that /usr/bin/sed has permission limitations..? I added to the bottom of the post the long list out put of all the sed that exist on my system – csny Aug 20 '14 at 06:37
  • @csny -I don't think so. I think I was on the wrong track. I think Stephane got to the bottom of this one already, have you seen his new answer yet? – mikeserv Aug 20 '14 at 07:13