Zip text files with minizip

[enzoevers]

No description provided.

[enzoevers]

All unit tests in release and debug builds pass on the Zybo board as well.

[enzoevers]

The generated .zip file from test_ZippingAndUnzippingRawTextGivesIdenticalOutput_MultiFileAndSubDir can be extracted using 7zip on windows. However, while the text files are extracted correctly and can be read, there are messages about failing to validating the CRC.

Extracting with the built-in extracter from the windows file explorer cannot be done for the files. While the folder are extracted and created, no files are in them. The message says: 0x80004005: Unspecified error.

• Look into the CRC setup and make sure that the generated .zip files can be extracted normally with the built-in windows extracter as well.
  • Fixed by not writing files with raw mode to in this push: https://github.com/enzoevers/CompressionOffloading/compare/ed513b9776d0a57bb7db2446f07628e0da8b70d9..ec13a496d279a57e97bc55bd0416580619d695f5

[enzoevers]

On Zybo it failed with:

BasicLinuxZybo:~$ ./Test_Release ./Files/ $(pwd)/ true
Unity test run 1 of 1
................................................................................                           .........................Expected (sizeToCreate): 6000000000
Actual (fileSize): 3802181632
D:\a\CompressionOffloading\CompressionOffloading\CoDeLib\Test\src\TestFileUtils.                           c:1106:TEST(TestFileUtils, test_GetFileSizeInBytes_ReturnsCorrectSizeOfFile_Larg                           eFile):FAIL: File size does not match expected size.
.....Could not open SomePath/someZipPath.zip for unzipping
......................D:\a\CompressionOffloading\CompressionOffloading\CoDeLib\Test\src\TestZipMinizipUnZipMinizip.c:85:TEST(TestZipMinizipUnZipMinizip, test_ZippingAndUnzippingRawTextGivesIdenticalOutput_CanZipAndUnZipFileOf6GB):FAIL: Expected 0 Was 1
..............

-----------------------
146 Tests 2 Failures 0 Ignored
FAIL

It turns out that the SD card of the Zybo just didn't have enough free space. Need to check with a bigger SD card.

[enzoevers]

Some debugging resulted in

Zip - Entry - Write - 1024 (max 1024)
Zip - Entry - Write - -116 (max 1024)
Failed to write to zip with zipWriteInFileInZip(...): : File too large
Zip - Entry - Write Close (crc 0xdfffdc96 cs -1 ucs -1)
Zip - Entry - Write cd (ucs 2147482624 cs 2147482624 crc 0xdfffdc96)
Zip - Close
Zip - Write cd (disk 0 entries 1 offset -114 size 53)

Here -116 is MZ_WRITE_ERROR and log(2147482624)/log(2) ~= 31. So somehow the the file is still capped at an unsigned 32-bit size. However, the unit test can generate 6GB files. So some (compile) flags in minizip-ng are not set properly.

This seems top be this errno EFBIG File too large (POSIX.1-2001). https://man7.org/linux/man-pages/man3/errno.3.html

---

To get the zip - .... logs I changed the following in External\minizip-ng\mz_zip.c.

#if 0
#  define mz_zip_print printf
#else
- #  define mz_zip_print(fmt, ...)
+ #  define mz_zip_print(fmt, ...) printf(fmt, ##__VA_ARGS__)
#endif

[enzoevers]

Making the following change in External\minizip-ng\CMakeLists.txt, and adding -DUSE_FILE_OFFSET_BITS64=ON to the minizip-ng build script at the config step, allowed for test_ZippingAndUnzippingRawTextGivesIdenticalOutput_CanZipAndUnZipFileOf3GB to pass. Without this it would fail.

project(minizip-ng
    VERSION ${VERSION}
    LANGUAGES C
    DESCRIPTION "Zip manipulation library"
	HOMEPAGE_URL https://github.com/zlib-ng/minizip-ng/)

+ if(USE_FILE_OFFSET_BITS64)
+     add_compile_definitions(_FILE_OFFSET_BITS=64)
+ endif()

Now on the zybo it fails with:

TEST(TestZipMinizipUnZipMinizip, test_ZippingAndUnzippingRawTextGivesIdenticalOutput_CanZipAndUnZipFileOf6GB)File sizes not equal:
    fileSize1: 6000000000
    fileSize2: 3503849472
C:\Users\enzoe\Documents\git\personal\CompressionOffloading\CoDeLib\Test\src\TestZipMinizipUnZipMinizip.c:110::FAIL: Expected TRUE Was FALSE

This most likely has to do with the fact that the SD card only has a 16GB rootfs. of which effectively ~14.5GB is available. Meaning 6.000.000.000 Bytes (~5.6GB) * 2 = 11.2GB (times 2 because there is a generated and the zipped 6GB), and 3.503.849.472 (~3.3GB), adds up to ~14.5GB. So this makes sense.

Going to increase the rootfs size and try again.

EDIT: it indeed passed with a larger rootfs.

[enzoevers]

Created a fork of minizip-ng: https://github.com/enzoevers/minizip-ng

This includes the option to add _FILE_OFFSET_BITS=64.

[enzoevers]

Test release tests from sha256:c14446af103e5a1bbb54e1518c21d6ee079d511ae65dc2308d2d6510d7ba6587 on the Zybo board.