How do I recursively grep through compressed archives?

Question

I'm trying to find out what modules use Test::Version in cpan. So I've used minicpan to mirror it. My problem is that I need to iterate through the archives that are downloaded, and grep the files that are in the archives. Can anyone tell me how I might do this? preferably in a way that tells me which file in the archive and what line it's on.

(note: they aren't all tarballs some are zip files)

Answer 1

Ok, let's apply the unix philosophy. What are the components of this task?

• Text search: you need a tool to search text in a file, such as grep.
• Recursive: you need a tool to go looking for files in a directory tree, such as find.
• Archives: you need a tool to read them.

Most unix programs operate on files. So to operate easily on archive components, you need to access them as files, in other words you need to access them as directories.

The AVFS filesystem presents a view of the filesystem where every archive file /path/to/foo.zip is accessible as a directory ~/.avfs/path/to/foo/zip#. AVFS provides read-only access to most common archive file formats.

mountavfs
find ~/.avfs"$PWD" \( -name '*.zip' -o -name '*.tar.gz' -o -name '*.tgz' \) \
     -exec sh -c '
                  find "$0#" -name "*.pm" -exec grep "$1" {\} +
                 ' {} 'Test::Version' \;
fusermount -u ~/.avfs   # optional

Explanations:

• Mount the AVFS filesystem.
• Look for archive files in ~/.avfs$PWD, which is the AVFS view of the current directory.
• For each archive, execute the specified shell snippet (with $0 = archive name and $1 = pattern to search).
• $0# is the directory view of the archive $0.
• {\} rather than {} is needed in case the outer find substitutes {} inside -exec ; arguments (some do it, some don't).
• Optional: finally unmount the AVFS filesystem.

Or in zsh ≥4.3:

mountavfs
grep 'Test::Version' ~/.avfs$PWD/**/*.(tgz|tar.gz|zip)(e\''
     reply=($REPLY\#/**/*.pm(.N))
'\')

Explanations:

• ~/.avfs$PWD/**/*.(tgz|tar.gz|zip) matches archives in the AVFS view of the current directory and its subdirectories.
• PATTERN(e\''CODE'\') applies CODE to each match of PATTERN. The name of the matched file is in $REPLY. Setting the reply array turns the match into a list of names.
• $REPLY\# is the directory view of the archive.
• $REPLY\#/**/*.pm matches .pm files in the archive.
• The N glob qualifier makes the pattern expand to an empty list if there is no match.

Answer 2

It appears that I can do it this way

find authors/ -type f -exec zgrep "Test::Version" '{}' +  

However, this gives results like:

authors/id/J/JO/JONASBN/Module-Info-File-0.11.tar.gz:Binary file (standard input) matches

which is not very specific to where in the tarball. Hopefully someone can come up with a better answer.

Answer 3

ugrep recursively searches compressed files (gz/Z/bz2/lzma/xz/lz4/zstd) and archives (cpio/tar/pax/zip) with option -z. Options -z --zmax=2 searches compressed files and archives embedded within compressed files and archives (hence zmax=2 levels).

Answer 4

After installing p7zip-* you are able to do this:

ls | xargs -I {} 7z l {} | grep whatever | less

You don't have to use ls before the first pipe, whatever list the compressed files will work. The final less only will show the PATH of the listet life inside the compressed archive, but not the name of this.

Answer 5

Thanks for the challenge, I came up with:

#!/bin/bash
#

# tarballs to check in
find authors/ -type f | while read tarball; do

    # get list of files in tarball (not dirs ending in /):
    tar tzf $tarball | grep -v '/$' | while read file; do       

        # get contents of file and look for string
        tar -Ozxf conform.tar.gz $file | grep -q 'Text::Version' && echo "Tar ($tarball) has matching File ($file)"

    done

done

Answer 6

Use find to locate all necessary files, and that zgrep to look into compressed files:

find <folder> -type f -name "<search criteria[*gz,*bz...]>" -execdir zgrep -in "<grep expression>" '{}' ';'

Didn't test this on tarballs though

Answer 7

Maybe my answer will helpfull for someone:

#!/bin/bash

findpath=$(echo $1 | sed -r 's|(.*[^/]$)|\1/|')

# tarballs to check in
find $findpath -type f | while read tarball; do

    # get list of files in tarball (not dirs ending in /):
    if [ -n "$(file --mime-type $tarball | grep -e "application/jar")" ]; then

        jar tf $tarball | grep -v '/$' | while read file; do
            # get contents of file and look for string
            grepout=$(unzip -q -c $tarball $file | grep $3 -e "$2")

            if [ -n "$grepout" ]; then
                echo "*** $tarball has matching file ($file):"
                echo $grepout
            fi

        done

    elif tar -tf $tarball 2>/dev/null; then

        tar -tf $tarball | grep -v '/$' | while read file; do
            # get contents of file and look for string
            grepout=$(unzip -q -c $tarball $file | grep $3 -e "$2")

            if [ -n "$grepout" ]; then
                echo "*** $tarball has matching file ($file):"
                echo $grepout
            fi

        done

    else
        file=""
        grepout=$(grep $3 -e "$2" $tarball)

        if [ -n "$grepout" ]; then
            echo "*** $tarball has matching:"
            echo $grepout
        fi

    fi

done

Answer 8

Here's a dash (also tested on bash, zsh, ksh shells) script that can search inside: .zip, .bz2, .xz, .tar.*, .tgz, .tar, .gz archives:

In order to run it: run it with no parameters and it will ask for the necessary info (reads input from keyboard):

#!/bin/dash
PrintInTitle () {
    printf "\033]0;%s\007" "$1"
}
PrintJustInTitle () {
    PrintInTitle "$1">"$print_to_screen"
}
CleanUp () {
    trap - INT
    trap - TSTP
    if [ -n "$TEP" ] && [ -n "$TEF" ]; then
        rm -R -f "$output_dir/"*
    fi
    unset IFS
    PrintJustInTitle ""
    if [ "$1" = "1" ]; then
        printf "Aborted\n">"$print_to_screen"
        kill -s PIPE -- -$$ 2>/dev/null
    fi
}
StoreArchiveFilePath () {
    eval k=$(($k + 1))
    eval archive_files_$k="$archive_file"
}
PrintMatch () {
    printf '\n%s\n\n' "$search_path$current_archive_file/$inside_current_archive_file"|grep --color -F "$search_path$current_archive_file/$inside_current_archive_file"
    for i in $(seq 1 $search_strings_0); do
        eval current_search_string="$search_strings_$i"
        printf '\n%s\n\n' "$current_search_string:"|grep --color -F "$current_search_string:"
        cat "$inside_current_archive_file"|grep -i -n -F "$current_search_string" 2>/dev/null
    done
}
GetCurrentContent () {
    cd "$TEP" && {
        eval current_archive_file="$$2"
        case "$current_archive_file" in
            '.zip' )
                unzip "$full_search_path/""$current_archive_file" -d "$TEF" >/dev/null 2>/dev/null
            ;;
            '.bz2' )
                bzip2 "$full_search_path/""$current_archive_file" -d "$TEF" >/dev/null 2>/dev/null
            ;;
            '.xz' )
                xz "$full_search_path/""$current_archive_file" -d "$TEF" >/dev/null 2>/dev/null
            ;;
            '.tar.'* | '.tgz' | '.tar' )
                tar -xvf "$full_search_path/""$current_archive_file" -C "$TEF" >/dev/null 2>/dev/null
            ;;
            *'.gz' )
                cp "$full_search_path/""$current_archive_file" "./$TEF"; gzip -d "./$TEF/""$current_archive_file" >/dev/null 2>/dev/null
            ;;
        esac
    cd "$output_dir"
    for inside_current_archive_file in $(for t in $(seq 1 $inside_archive_file_path_filters_0); do eval current_inside_archive_file_path_filter=\"\$inside_archive_file_path_filters_$t\"; eval find . -type f -path "$current_inside_archive_file_path_filter"|sort --numeric-sort; done;); do
        gcc_found="true"
        for i in $(seq 1 $search_strings_0); do
            eval current_search_string=\"\$search_strings_$i\"
            gcc_stored_content="$(cat "$inside_current_archive_file"|grep -i -n -F "$current_search_string" 2>/dev/null;)";
            if [ -z "$gcc_stored_content" ]; then
                gcc_found="false"
                break
            fi
        done
        if [ "$gcc_found" = "true" ]; then
            if [ "$1" = "StoreArchiveFilePath" ]; then
                StoreArchiveFilePath
                break
            elif [ "$1" = "PrintMatch" ]; then
                PrintMatch
            fi
        fi
    done
    if [ -n "$TEP" ] && [ -n "$TEF" ]; then rm -R -f "$output_dir/"*; fi
    cd "$full_search_path"
}

}
set +f #Enable globbing (POSIX compliant)
setopt no_nomatch 2>/dev/null #Enable globbing (zsh)
IFS='
'
print_to_screen='/dev/tty'
initial_dir="$PWD"
case "$(uname -s)" in
    "Linux" )
        TEP='/dev/shm' #TEMPORARY_EXTRACT_PATH
    ;;
    "Darwin" | "BSD" | * )
        TEP="$HOME" #TEMPORARY_EXTRACT_PATH
    ;;
esac
TEF='TEMP_EXTRACT_FOLDER' #TEMP_EXTRACT_FOLDER
output_dir=""
error="false"
{
    cd "$TEP" && {
        if [ ! -e "$TEF" ]; then
            printf '%s\n' "The specified temporary directory: "$TEF" - does not exist in the specified location: "$TEP" - do you want to create it? [ Yes / No ] (default=Enter=No): ">"$print_to_screen"
            read answer
            if [ "$answer" = "Yes" ] || [ "$answer" = "yes" ] || [ "$answer" = "Y" ] || [ "$answer" = "y" ]; then
                mkdir "$TEF" || error="true"
            fi
        fi
        cd "$TEF" && output_dir="$PWD" || error="true"
    } || error="true"
} 2>/dev/null
if [ "$error" = "true" ]; then
    printf '%s\n' "Error: Could not access temporary folder "$TEF" in the extract location: "$TEP"!">&2
    read temp
    exit 1
fi
trap 'CleanUp 1' INT
trap 'CleanUp 1' TSTP
cd "$HOME"
printf '%s\n' "Search Path (blank=default=current folder=$PWD): "
read search_path
if [ -z "$search_path" ]; then
    search_path="."
fi
printf '\n%s\n' "Inside archive file path filters: (what file path to lookup inside the archive) (concatenated internaly with logical OR) (default=Enter=''): ">"$print_to_screen"
i=0
while [ "1" = "1" ]; do
    printf '%s' ">> inside archive path filter: >> "
    IFS= read -r current_inside_archive_file_path_filter
    unset IFS
    if [ -z "$current_inside_archive_file_path_filter" ]; then
        break
    fi
    i=$(($i + 1))
    eval inside_archive_file_path_filters_$i="$current_inside_archive_file_path_filter"
done
inside_archive_file_path_filters_0=$i
unset IFS #Reset IFS
if [ "$inside_archive_file_path_filters_0" = "0" ]; then
    inside_archive_file_path_filters_1="'"''"'"
    inside_archive_file_path_filters_0="1"
fi
printf '\n%s\n' "Search strings (concatenated internaly with logical AND):">"$print_to_screen";
i=0
while [ "1" = "1" ]; do
    printf '%s' ">> add search string: >> "
    IFS= read -r current_search_string
    unset IFS
    if [ -z "$current_search_string" ]; then break; fi;
    i=$(($i + 1))
    eval search_strings_$i="$current_search_string"
done
search_strings_0=$i
if [ "$search_strings_0" = "0" ]; then
    search_strings_1=""
    search_strings_0=1
fi
i=0
PrintJustInTitle "Loading list of archive files to analyze..."
IFS='
'
cd "$search_path" && {
    full_search_path="$PWD"
    { find . ( -type f -path '.zip' -o -path '.bz2' -o -path '.xz' -o -path '.tar.' -o -path '.tgz' -o -path '.tar' -o -path '.gz' ) -exec printf "%s\n" "{}" ;|sort --numeric-sort;

        printf '%s\n' "..."; index=defined; while [ -n "$index" ]; do read index; printf '%s\n' $index; done; }|{
        j=0; k=0
        while read -r line; do
            j=$(($j + 1))
            if [ "$line" = "..." ]; then
                break
            else
                PrintJustInTitle "Analyzing archive file $j..."
                archive_file="$line"
            fi
            GetCurrentContent StoreArchiveFilePath archive_file
        done
        archive_files_0="$k"
        PrintJustInTitle ""
    if [ ! "$k" = "0" ]; then
        index="defined"
        count="$archive_files_0"
        while [ -n "$index" ]; do
            for i in $(seq 1 $count); do
                eval current_archive_file=\"\$archive_files_$i\"
                printf '\033[0;31m%s\033[0m\n' "$i = ""$current_archive_file"
            done
            printf '\n%s\n' "Print results (one at a time - blank = exit)?: [ 1 - $count ]: "
            read index
            if [ -z "$index" ]; then
                break
            fi
            printf '%s\n' '---------------------------------------------------------------'
            GetCurrentContent PrintMatch archive_files_$index
            printf '%s\n\n' '---------------------------------------------------------------'
        done
    fi
}

}
PrintJustInTitle ""
CleanUp

It consumes a lot of resources but the output is detailed.

Answer 9

In zsh and with bsdtar + GNU tar + GNU grep, that could be:

set -o extendedglob
for f (**/*.(#i)(zip|tar|(t|tar.)(xz|gz|bz2))(N.))
  bsdtar cf - @$f | ARCHIVE=$f tar -xf - --to-command='
    if [ "$TAR_FILETYPE" = f ]; then
      grep -H --label="$ARCHIVE[$TAR_FILENAME]" Test::Version
    fi
    true'

Where

• the zsh glob looks for regular (. glob qualifier) unhidden files whose name ends in .zip, .tar, .tar.gz, .tgz... (case insensitively).
• bsdtar converts the file to a ustar archive format that GNU tar supports
• we use GNU tar's --to-command to pipe the contents of each file to grep
• grep finds the matches, labelling them with file.gz[file/in/archive].
• we terminate the --to-command script with true to avoid the warnings by tar when that command returns with a non-zero exit status:

To restrict the search to archive members whose name ends in .pm, you can change the script for --to-command to:

    case $TAR_FILETYPE$TAR_FILENAME in
      (f*.pm) grep -H --label="$ARCHIVE[$TAR_FILENAME]" Test::Version
    esac
    true