Can gzip -t detect 100% of truncated-download errors?

Question

Scenario: a single 1g CSV.gz is being written to an FTP folder. At the same time, my client machine connects over sFTP to that folder and attempts to pull it down.

Q: After fetching that file, resulting in whatever apparent length I get client-side, can gzip -t detect and fail the partial file, regardless of where the truncation hit?

I figure that unzipping or -t'esting will error out on 99% of the possible truncation points when a fragment ends abruptly, but does the gz structure have clean cleaving points where gzip will accidentally report success?

mitigations that are not on table (because if one of these were in play, I wouldn't need to ask the above.)

1. Getting the file length or md5 through another network request.
   1. polling the file length through FTP isn't great, as the server may be sporadically writing chunks to the zip stream. Until the batch job has closed the file handle, it would be lethal to my analysis to mistake that for a complete data set.
   2. being given the final file length or hash by the batch job removes the need for this Q, but that puts an implementation burden on a team that (for this Q's purposes), may as well not exist.
2. we can't avoid the racing by scheduling reads/writes for different times of day.
3. the server is not using atomic move operations.
4. I don't know the CSV row/column counts; it'll change per snapshot and per integration made. May as well say the file being gzipped is a opaque binary blob for this Q.
5. there are no client=>sFTP network errors in play. (those are caught and handled; my concern is reading a file that's still being sporadically written during the server's batch job.)
6. Using a RESTful API instead of sFTP.

didn't find an existing SO

Several SO touch on handling truncations, but are in a lossy-acceptable context, as compared to needing to reliably fail the whole workflow on any problem. (I'm computing in a medical-data context, so I'd rather have the server halt and catch fire than propagate incorrect statistics.)

• gzip: unexpected end of file with - how to read file anyway is the reverse -- they want to suppress EOF errors as not a problem for their use case
• Why I get unexpected end of file in my script when using gzip? is just posix stream ends intentionally inserted by head and doesn't cover "is it possible for a false-positive to slip in?"
• zcat / gzip error while piping out is very close, but doesn't ask "am I guaranteed to get this error?"
• Merge possibly truncated gzipped log files is also close, as it deals with partial files from terminated batch jobs, but is still about throwing out a few unreadable rows, not about guaranteeing errors.

Answer 1

Files in gzip format contain the length of the compressed data and the length of uncompressed data. However, this is an ancient format and the length fields only have 32 bits, so nowadays they're interpreted as being the length modulo 2^32 (i.e. 4 GiB). Before decompression, gzip checks that the checksum of the compressed data is correct. After decompression, gzip checks that the checksum of the decompressed data is correct, and that the size of the decompressed data is correct modulo 2^32.

As a consequence, gzip is guaranteed to detect a truncated input if the size of the compressed data (or the size of the decompressed data) is less than 4 GiB. However, for arbitrary-sized files, I don't see any reason why those checks would be enough. If the input is not deliberately crafted and its length is uniformly distributed modulo 4 GiB, there's only a 1/2^64 chance that both the compressed length and the checksum match, and additionally an error will be detected if the file is truncated in the middle of a multi-byte sequence or if the length of the uncompressed data doesn't match. (That doesn't necessarily reduce the chance to 1/2^96 because the compressed length modulo 2^32 and the uncompressed length modulo 2^32 are correlated.) So there's only a tiny chance of an undetected error, but it's nonzero, and I'm sure it could be crafted deliberately.

Note that this analysis only applies if the gzipped file consists of a single stream. gunzip can uncompress files that consists of multiple concatenated streams, and there's no way to detect if the file contains a valid sequence of streams but more streams were intended. However, your production chain probably doesn't generate multi-stream files: gzip doesn't do it by itself, you'd have to concatenate the output of multiple runs manually, or use some other tool (pkzip?).

the server is not using atomic move operations.

Unfortunately I don't think there's a completely reliable way to detect errors without either that or an external piece of metadata (length or cryptographic checksum) which is calculated after the server has finished writing.