3

I'm trying to zip streaming text, and unzip it later. A minimal example is:

echo "Hey, could you zip this for me?" | zip hello.zip - ; unzip -p hello.zip

This fails, with the following output:

warning [hello.zip]:  76 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [hello.zip]:  reported length of central directory is
  -76 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
  zipfile?).  Compensating...
   skipping: -                       need PK compat. v4.5 (can do v2.1)

note:  didn't find end-of-central-dir signature at end of central dir.
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)

I'm guessing this has something to do with that note about binary mode, but I can't figure out what, or find any help in the relevant man pages / Google / StackExchange. I believe this question is different from "Extra-bytes error when unzipping a file." The error messages are the same, but that question is about fixing an existing 3rd party file archive, whereas this is about working with an archive you just created yourself.

Version detail:

  • OS X 10.10.4
  • Zip 3.0 (July 5th 2008), by Info-ZIP
  • UnZip 5.52 of 28 February 2005, by Info-ZIP
duozmo
  • 133
  • 1
  • 5
  • I just ran echo "Hey, could you zip this for me?" | zip hello.zip - ; unzip -p hello.zip on Debian-6 Gnu/Linux. It worked fine. – ctrl-alt-delor Aug 03 '15 at 22:32
  • Has this minimal example been tested? Does it produce the exact output shown? – ctrl-alt-delor Aug 03 '15 at 22:42
  • @richard I saw that question, but don't think it's a dupe. FTP wasn't involved and I'm not unzipping a file. Using --fixfix doesn't seem like the solution here. Output is from copy & paste. – duozmo Aug 03 '15 at 22:51
  • Your zip may be broken. Can any MacOS users replicate this? – ctrl-alt-delor Aug 03 '15 at 22:53
  • What does echo "Hey, could you zip this for me?" | zip -T hello.zip - produce? – ctrl-alt-delor Aug 03 '15 at 22:57
  • @richard That test outputs: zip warning: Found UnZip version 5.52. Need UnZip 6.00 or later to test this Zip64 archive. That's a good lead, I will look into it and report back. – duozmo Aug 03 '15 at 23:04
  • I replicated this on OSX 10.6 with the default unzip binary. Using gunzip -c (gunzip/gzip v1.6) yields Hey, could you zip this for me? PK-?SG?$?< ?-PK,-/SPK?PK/S gzip: hello.zip: unexpected end of file – Kent Aug 04 '15 at 01:35

1 Answers1

1

As per the comments I have confirmed this at the bottom of this answer.

I think the issue is that a streaming zip will default to Zip64 format but that unzip 6 or later is needed to read Zip64. The manual says -fz- can be used if the input is known to be smaller than 4 GB to prevent the use of Zip64, but that isn't clearly documented anywhere else in that manual.

Forcing the <4GB zip binary format, as in the zip manual and hinted at in the "(please check that you..." part of the warning from unzip [see below], basically works with the extra output of the zip step:

Daniels-Mini:~ dlamblin$ echo "Hey, could you zip this for me?" | zip -fz- hello.zip - ; unzip -p hello.zip
updating: - (stored 0%)
Hey, could you zip this for me?

It's fun to point out that while zip can be piped like a filter echo a|zip|cat unzip cannot work on stdin, so there's already a mismatch there. Oddly echo a|zip a.zip - and echo a|zip>b.zip produce different files of the same size. That confused me.

For unzipping streams as a filter there's funzip. Sadly the compatible version of funzip is also not installed by default on Mac OS X 10.6.

Daniels-Mini:~ dlamblin$ echo "Hey, could you zip this for me?" | zip |funzip
  adding: - (deflated -5%)
Hey, could you zip this for me?
funzip error: invalid compressed data--length error
Daniels-Mini:~ dlamblin$ echo "Hey, could you zip this for me?" | zip -fz-|funzip
  adding: - (deflated -5%)
Hey, could you zip this for me?

The confirmation of the issue follows:

Last login: Mon Aug  3 22:54:02 on ttys010
Daniels-Mini:~ dlamblin$ echo "Hey, could you zip this for me?" | zip hello.zip - ; unzip -p hello.zip
  adding: - (stored 0%)
warning [hello.zip]:  76 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [hello.zip]:  reported length of central directory is
  -76 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
  zipfile?).  Compensating...

note:  didn't find end-of-central-dir signature at end of central dir.
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)
Daniels-Mini:~ dlamblin$ zip --version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.

Compiled with gcc 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39) for Unix (Mac OS X) on Apr 30 2015.

Zip special compilation options:
    USE_EF_UT_TIME       (store Universal Time)
    SYMLINK_SUPPORT      (symbolic links supported)
    LARGE_FILE_SUPPORT   (can read and write large files on file system)
    ZIP64_SUPPORT        (use Zip64 to store large files in archives)
    STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field)
    UIDGID_16BIT         (old Unix 16-bit UID/GID extra field also used)
    [encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3)

Encryption notice:
    The encryption code of this program is not copyrighted and is
    put in the public domain.  It was originally written in Europe
    and, to the best of our knowledge, can be freely distributed
    in both source and object forms from any country, including
    the USA under License Exception TSU of the U.S. Export
    Administration Regulations (section 740.13(e)) of 6 June 2002.

Zip environment options:
             ZIP:  [none]
          ZIPOPT:  [none]
Daniels-Mini:~ dlamblin$ unzip --version
caution:  both -n and -o specified; ignoring -o
UnZip 5.52 of 28 February 2005, by Info-ZIP.  Maintained by C. Spieler.  Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Usage: unzip [-Z] [-opts[modifiers]] file[.zip] [list] [-x xlist] [-d exdir]
  Default action is to extract files in list, except those in xlist, to exdir;
  file[.zip] may be a wildcard.  -Z => ZipInfo mode ("unzip -Z" for usage).

  -p  extract files to pipe, no messages     -l  list files (short format)
  -f  freshen existing files, create none    -t  test compressed archive data
  -u  update files, create if necessary      -z  display archive comment
  -x  exclude files that follow (in xlist)   -d  extract files into exdir

modifiers:                                   -q  quiet mode (-qq => quieter)
  -n  never overwrite existing files         -a  auto-convert any text files
  -o  overwrite files WITHOUT prompting      -aa treat ALL files as text
  -j  junk paths (do not make directories)   -v  be verbose/print version info
  -C  match filenames case-insensitively     -L  make (some) names lowercase
  -X  restore UID/GID info                   -V  retain VMS version numbers
  -K  keep setuid/setgid/tacky permissions   -M  pipe through "more" pager
Examples (see unzip.txt for more info):
  unzip data1 -x joe   => extract all files except joe from zipfile data1.zip
  unzip -p foo | more  => send contents of foo.zip via pipe into program more
  unzip -fo foo ReadMe => quietly replace existing ReadMe if archive file newer
dlamblin
  • 403
  • Can confirm: it's a mismatch of having Zip64 on by default in zip and unavailable in unzip. I even tried UnZip 6, but it too failed. Admittedly, zip's man page does caution, "some utilities may still not support streamed archives yet." The solution is, as you say, calling zip with -fz-. – duozmo Aug 04 '15 at 18:09