Unusable zip arhive of huge volume

[Shmuel (Seymour J.) Metz]

When I zip a volume containg more than 4 MiB compressed, Arctool fails with the message

Z0001: Unable to open file
K:\M.zip
The data is invalid

An unzip test shows

Thu 10-18-18 4:01:16{9}[h:\] unzip -t K:\M
Archive: K:/M.zip

End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.

[Silvan Scherrer]

4 MiB ??? or 4 GiB???

and to what filesystem do you zip it?

[Shmuel (Seymour J.) Metz]

Sorry, 4 GiB. The archive volume was JFS.

[Silvan Scherrer]

could you test if the zip file is valid? means if it can be opened on another operating system? as it might be a unzip issue and not zip. please try to find the real cause first.

[Shmuel (Seymour J.) Metz]

The unzip fails with the same error on x86 Linux, both 32-bit and 64-bit.

[Silvan Scherrer]

ok so it seems like a zip issue. I might have a look when time permits. But most likely not very soon.

[Shmuel (Seymour J.) Metz]

Is there any way to generate diagnostic output from zip that would help you track it down?

[Silvan Scherrer]

you could try the split option. eventually this works.

How to diagnostic? I would need to look at the code first. But as said. I do when time permits, or someone sponsors this fix immediately.

[Shmuel (Seymour J.) Metz]

Doesn't split generate the same archive name on multiple volumes rather than multiple archive names on a single volume?

Is the a command line 7z utility for ArcaOS and, if so, what is the package name?

[Silvan Scherrer]

split just splits the zip file after a certain amount of bytes. It's well documented in the help screen iirc.
7z??? is something other and has nothing to do with this ticket. And I have no idea at all.

And as said, if you need a fix either wait until we have time do look at it or sponsor the fix. We can't work for free sorry.

[ak120]

Please provide the output of

zip -v

and

unzip -v

which are involved in your operations. Please read the file ziplimit.txt for further information.

[Shmuel (Seymour J.) Metz]

I'm away from my computer, but I'll check the versions when I get home. Meanwhile, the "-s 3G" option didn't help; should I try it with a smaller size?

BTW, in my previous reply I was confusing split with spanning in my previous reply.

[Silvan Scherrer]

start with 1GB and then go up until it blows

[Shmuel (Seymour J.) Metz]

1. Some error messages only appear with the -v (verbose) option

2. with -s 1G, Arctool will still not handle the zip file or the concatenated file.

3. unzip -t recognizes the zip fil as split

4. If I concatenate Z0a Z02 , ..., ZIP, unzip -t gives warnings but no fatal errors

[Shmuel (Seymour J.) Metz]

unzip -v output

[Shmuel (Seymour J.) Metz]

zip -v output

[Silvan Scherrer]

what Arctool does is out of this scope, as this is not maintained by us. So with split all runs?

[Shmuel (Seymour J.) Metz]

If I use -s (split) and concatenate the segments in sequence then unzip gives warnings but successfully extracts a file.

JFS supports large files, and Info-zip has support for 64-bit file addressing on 32-bit systems, so it may be easier to support large archives than to fully support split.

I ran into a couple of small problems and wonder whether I should open separate tickets on them:

1. Some error messages are only written with -v (verbose)

2. Some error messages, e.g., permission, are written without any context, so -v is needed to determine the relevant file

3. The Disk full message does not distinguish between the volume for the new archive and the -b (work) volume

[Silvan Scherrer]

please do what I asked for and try to find out which split value is the highest working one.
We have 64-bit addressing enabled, thatswhy I ask about the highest working split value, to see why it might be broken.
For the other issues you need to open individual tickets.

But as I told already: don't expect fast fixes, as zip is right now not top priority.
If you want a fast fix, we are more than glad to send you a price quote.

[Shmuel (Seymour J.) Metz]

Script used to generate zip commands

[Shmuel (Seymour J.) Metz]

Script used to concatenate z?? file writte by zip -s

[Shmuel (Seymour J.) Metz]

The zip command treats -s 8G as -s 4G; I get the same warnings as with 1G, the same error messages:

testing: PDF/honeywell/Multics/AG90-03_PgmgIntro_Dec81.pdf

error: invalid compressed data to inflate

and I am able to extract files other than the one with the above error message. I've attached the script used to build the zip commands and the script used to concatenate the pieces from zip -s; would it help to attach logs from the zip, the unzip -t or the unzip to extract?

Meanwhile I'll take a look at gzip. Thanks.

[Shmuel (Seymour J.) Metz]

I spoke too soon. It is true that I can unzip *some* files, but it turns out that I can't unzip all of them. The attached files are the logs from generating and attempt to use a large zip archive with -s 1G.

[Shmuel (Seymour J.) Metz]

Sorry: the attachments are too large to upload. I can try to trim them, or can provide them through alternate channels.

I zipped them in pairs and that shrank them enough to upload.

[Shmuel (Seymour J.) Metz]

Logs of zip -s 1G and unzip -t of zip (last) segment

[Shmuel (Seymour J.) Metz]

Logs of unzip -l and unzip of concatenated split files

[Silvan Scherrer]

thanks for the info. When time permits or someone sponsors the fix we will look at it.

[Shmuel (Seymour J.) Metz]

Please see discussion at ​https://www.os2world.com/forum/index.php/topic,1941.0.html

[Silvan Scherrer]

this discussion is completely useless, at least for the issues you have. Because all of that I decided to put aside zip for now and might look at it next year again. Why should I invest time, which no one pays and then some individuals (not referring to the reporter of this ticket) write such bullshit? I really have a lot better things to do with my time.

So I repeat again, if one sponsors it, I will look at it, but if no one steps up, I will not look at those issues for longer.

[Shmuel (Seymour J.) Metz]

In case someone else runs into this, /pub/os2/util/archiver/p7zip-9.20.1-os2.zip in ​http://hobbes.nmsu.edu/h-browse.php?dir=/pub/os2/util/archiver provides a viable workaround; the newer ​http://hobbes.nmsu.edu/download/pub/os2/util/archiver/p7zip-16.02-os2-1.zip didn't work.

[Shmuel (Seymour J.) Metz]

It turns out that the problem only occurs with the work directory (-b) option. Removing it is a viable bypass, although it does more than double the elapsed time when the archive goes to a USB thumb drive.

[Silvan Scherrer]

USB is slow. But that's not a zip issue.

[Shmuel (Seymour J.) Metz]

@diver Of course not, but it is an issue with the acceptability of the bypass, which is why I mentioned it.

[Silvan Scherrer]

it it true that all works as it should when you don't use the -b option? If so please open a new ticket with a valid description about the -b option and close this ticket.

[Shmuel (Seymour J.) Metz]

I created ticket 201 explicitly referencing -b. Should I also open a ticket for the split (-s) issue, or is that not supported?

[Silvan Scherrer]

Every issue needs a own ticket. Please verify first in other platforms as well. As it might be a crossbug.
Marking this one as solved in favour of the new tickets