0

Using below function to filter out text file and organize to new format.

Using unix redirection > command to create new files.

How can I get rid of creating all new files

(e.g. ${file}_final_1.txt,${file}only${pattern}.txt)

without deleting it afterword using rm command ?

filter(){
  cat ${file}_1.txt | grep -v ${pattern} > ${file}_final_1.txt                            
  cat ${file}_1.txt | grep ${pattern} > ${file}_only_${pattern}.txt                        
  cat ${file}_only_${pattern}.txt  | nawk -F '|' '{ print $NF}' > ${pattern}_TS.txt        
  paste ${pattern}_TS.txt ${file}_only_${pattern}.txt > ${pattern}_TS_file.txt
  cat ${pattern}_TS_file.txt | grep "|${DT}|" | grep ${pattern} | sort -r | head -1 > ${file}_f.txt
  cat ${file}_f.txt >> ${file}_final_1.txt
  dos2unix ${file}_final_1.txt ${file}_final.txt
}
Rocky
  • 3
  • (1) You can avoid creating so many files if you build longer pipes instead of writing to a file only to read from it in the next step. This won't work with paste though, where in your case you need at least one regular file as input (unless you use process substitution, but not really useful in your case where ${file}_only_${pattern}.txt is used twice). (2) Creating temporary files in a temporary directory (mktemp -d) allows you to remove them all by removing the directory (rm -r …). // This is not an answer because (1) does not "get rid of creating all" and (2) is not "without rm". – Kamil Maciorowski Aug 23 '22 at 15:03
  • 1
    I hope you're using zsh. – Kamil Maciorowski Aug 23 '22 at 15:06
  • As a starting point, copy/paste that script into http://shellcheck.net and fix the many issues it'll tell you about. – Ed Morton Aug 23 '22 at 20:45

1 Answers1

1
  1. You can chain pipes as long as you like:
  cat ${file}_1.txt | grep ${pattern} > ${file}_only_${pattern}.txt                        
  cat ${file}_only_${pattern}.txt  | nawk -F '|' '{ print $NF}' > ${pattern}_TS.txt

could be replaces with

cat ${file}_1.txt | grep ${pattern} | nawk -F '|' '{ print $NF}' > ${pattern}_TS.txt

And so on... There is a limit on the command line, of course, but it is ridiculously large. Do getconf ARG_MAX and you'll get how many characters it is.

  1. If some app does not accept input from a pipe, or it is easier to work with multiple files... Then you can always dump temp files into a temp directory. So your filter() function could follow a pattern:
filter(){
   TEMP_DIR=/tmp/dir_for_this_task_`date +%s`
   mkdir $TMP_DIR
   CUR_DIR=`pwd`
   cd $TMP_DIR

.... your code with temp files


cd $CUR_DIR
   rm -r $TMP_DIR
}

you can also use pushd/popd to remember current directory and return to it from temp.

White Owl
  • 5,129
  • 2
  • Use mktemp to create temporary files and directories 2. Double-quote your variables when you use them 3. Use $(...) instead of backticks
    • – Chris Davies Aug 23 '22 at 15:16
  • @roaima There are infinite ways to improve any script. – White Owl Aug 23 '22 at 15:21
  • 2
    There are indeed, but writing incorrect code is poor – Chris Davies Aug 23 '22 at 20:25
  • @roaima $(...) is a bashism, backticks work in every POSIX shell. – allo Aug 23 '22 at 20:43
  • 3
    @allo no, it's part of the spec for all POSIX shells, see https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02_03 and other parts of the spec. – Ed Morton Aug 23 '22 at 20:47