BiospexZipCreator

Purpose

Compresses images and manifests from S3 into ZIP files. It supports creating both full ZIP archives and partial ZIP segments for large datasets.

Workflow

1. Trigger: Receives messages from SQS (typically via ZooniverseZipTriggerService).
2. Analysis: Lists objects in the specified S3 processDir.
3. Slicing: Filters and slices the file list based on startIndex and endIndex.
4. Manifest: Ensures the manifest CSV is included in every ZIP part.
5. Compression:
   • Files < 2GB: Processed in Lambda's /tmp space.
   • Files > 2GB: Processed using Amazon EFS mount (/mnt/efs).
6. Upload: Saves the resulting ZIP to export/ in the S3 bucket.
7. Callback: Sends a status update (zip-ready or partial-zip-ready) back to the Laravel app via SQS.

Inputs/Outputs

• Inputs (JSON):
  • processDir: The directory in S3 containing files to zip.
  • s3Bucket: Target S3 bucket.
  • updatesQueueUrl: SQS URL for reporting status.
  • queueId: Reference ID for the Laravel export queue.
  • startIndex/endIndex: (Optional) Indices for partial zipping.
• Outputs:
  • S3: export/{processDir}.zip or export/{processDir}_part{index}.zip.
  • SQS: Status notification to updatesQueueUrl.

Related Components

• Laravel Command: App\Console\Commands\SqsListenerExportUpdate (Listens for zip-ready status).
• Laravel Service: App\Services\Actor\Zooniverse\ZooniverseZipTriggerService (Triggers this Lambda).
• Related Lambda: BiospexZipMerger (Consolidates partial ZIPs created by this function).

Deployment

Use the deploy.sh script for interactive deployment to AWS (Region: us-east-2).