Add celery and celery-beat to delete partial files every 48 hours by paigewilliams · Pull Request #234 · Ecotrust/TEKDB

paigewilliams · 2026-03-10T21:09:42Z

Adds celery, celery-beat and django-celery-results, django-celery-beat to delete partial uploads every 48 hours, with the goal of not bloating servers with partial uploads. Deletion logic exists in the delete_expired_chunks task. Uses redis as the message broker. django-celery-results displays the task results in the admin interface. This should help with transparency and observability. django-celery-beat also displays the intervals at which tasks run. The thinking here is that it allows tribes to have more control over background tasks, and if they want them or not.

For running with docker, adds a celery worker, celery-beat and redis container. For running with Vagrant it runs daemonized. The only step to add in the install instructions is sudo bash /usr/local/apps/TEKDB/TEKDB/scripts/provision_celery.sh to run after database migrations.

Screenshots

Copilot

Pull request overview

This PR introduces Celery (worker + beat) into the TEKDB Django project to periodically delete expired resumable-upload chunk files (intended every 48 hours), using Redis as the broker and django-celery-results / django-celery-beat for admin visibility and scheduling control.

Changes:

Add Celery app/task (delete_expired_chunks) and schedule it via Django settings; add a basic test for the cleanup behavior.
Add Docker compose services for redis, celery, and celery-beat, plus a web healthcheck used for startup ordering.
Add Vagrant/systemd provisioning artifacts to run Redis + Celery worker/beat as services.

Reviewed changes

Copilot reviewed 14 out of 16 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
docker/docker-compose.yaml	Adds redis/celery/celery-beat services to dev compose and wires broker env.
docker/docker-compose.prod.local.yaml	Adds redis/celery/celery-beat services to prod-local compose.
docker/common.yaml	Defines redis/celery/celery-beat service templates and adds a web healthcheck.
deployment/celery-worker.service	New systemd unit for celery worker in Vagrant.
deployment/celery-beat.service	New systemd unit for celery beat (DatabaseScheduler) in Vagrant.
TEKDB/scripts/provision_celery.sh	Provision script to install Redis and enable/start celery systemd units.
TEKDB/requirements.txt	Adds celery, redis, django-celery-results, django-celery-beat dependencies.
TEKDB/media/init.py	Adds placeholder file under media directory.
TEKDB/TEKDB/tests/test_tasks.py	Adds tests for chunk cleanup behavior.
TEKDB/TEKDB/tekdb_filebrowser.py	Minor lambda formatting adjustment (parentheses).
TEKDB/TEKDB/tasks.py	Adds `delete_expired_chunks` Celery task and helper formatting function.
TEKDB/TEKDB/settings.py	Registers celery results/beat apps and adds Celery configuration + beat schedule.
TEKDB/TEKDB/celery.py	Adds Celery app initialization and autodiscovery.
TEKDB/TEKDB/admin.py	Adjusts django-celery-results admin registration (unregister GroupResult).
TEKDB/TEKDB/init.py	Imports Celery app on Django startup.
TEKDB/TEKDB/management/commands/delete_expired_chunks.py	Adds a management command module file (currently empty).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-10T21:17:26Z

TEKDB/TEKDB/tasks.py

+
+

bytes_to_readable can return None for very large values because the loop has no fallback return after the last unit. Ensure the function always returns a string (e.g., add a final return for the largest unit).

Suggested change

# Fallback: for extremely large values, use the largest unit.

return f"{num_bytes:.2f} P{suffix}"

TEKDB/TEKDB/settings.py

deployment/celery-worker.service

deployment/celery-beat.service

Copilot · 2026-03-10T21:17:27Z

docker/docker-compose.prod.local.yaml

+  redis:
+    extends:
+      file: common.yaml
+      service: redis
+  celery:
+    extends:
+      file: common.yaml
+      service: celery
+  celery-beat:
+    extends:
+      file: common.yaml
+      service: celery-beat


PR description indicates celery/beat should be part of the Docker deployment to run the 48h cleanup, but only docker-compose.yaml (dev) and docker-compose.prod.local.yaml include the redis/celery services. The actual deployed compose file (docker/docker-compose.prod.yaml, used in .github/workflows/deploy-ec2-staging.yml) currently does not include redis/celery/celery-beat, so the cleanup task won’t run in that environment.

docker/common.yaml

… exit from healthcheck

paigewilliams · 2026-03-11T17:10:53Z

TEKDB/TEKDB/tekdb_filebrowser.py

            path,
-            filter_func=lambda fo: not fo.has_media_record()
-            and fo.filename not in self.files_folders_to_ignore(),
+            filter_func=lambda fo: (


changes to this file are just ruff formatting fixes

paigewilliams · 2026-03-11T17:12:39Z

deployment/celery-beat.service

@@ -0,0 +1,18 @@
+[Unit]


pulled much of this from these docs: https://docs.celeryq.dev/en/stable/userguide/daemonizing.html#daemonization

paigewilliams · 2026-03-11T17:29:36Z

TEKDB/Dockerfile

 RUN chmod +x /usr/local/bin/entrypoint.sh

+# Create a non-root user and give it ownership of the app directory
+RUN addgroup --system appgroup && adduser --system --ingroup appgroup appuser \


Added to resolve warning of SecurityWarning: You're running the worker with superuser privileges: this is absolutely not recommended! in the celery container.

pollardld

🎉 🎉 🎉

pollardld · 2026-03-11T19:06:12Z

docker/docker-compose.prod.local.yaml

    volumes:
      - static_volume:/vol/static/static:ro
      - media_volume:/vol/static/media:ro
+  redis:


I like that it is possible, and so easy, to extend into other yaml files.

agreed! makes it super easy to have different configs for different environments, but keep things DRY.

rhodges

Not only is it a great new feature, but this makes an easy job of including several very powerful tools that will be welcome when adding features in the future! I am curious how this will impact hardware usage, but in past experience Redis and Celery have proven themselves to be quite performant.

rhodges · 2026-03-11T23:23:47Z

docker/common.yaml

      context: ../TEKDB/
      dockerfile: ../TEKDB/Dockerfile
    restart: unless-stopped
-    depends_on:


Why was this dependency removed?

ah, I see the inheritance now - cool!

rhodges · 2026-03-11T23:31:36Z

TEKDB/TEKDB/tasks.py

+            file_path = os.path.join(root, filename)
+            try:
+                mtime = os.path.getmtime(file_path)
+                if mtime >= cutoff_timestamp:


This is the one piece of logic I most wanted to see in this review, and not only is it here, it's clear, readable, and well-written!

paigewilliams added 17 commits March 4, 2026 13:38

add logs module to track expired chunk deletion

b3e3dc1

management command to delete expired chunks

b943f3b

ruff format

6ccc0ca

wip: add celery for deleting expired chunks

61aa1d9

Merge branch 'develop' into delete-partial-files

e5234f8

sequence containers, update env vars

8cecf62

schedule task to delete remaining chunks

7e4b7b9

move delete_expired_chunks tasks to TEKDB

6a9f661

Merge branch 'develop' into delete-partial-files

0241a46

move env vars out of common.yaml

f14f7a0

add redis, celery, and celery beat to docker-compose.prod.local

2c95eec

remove debug task

d2fefb8

wip: add tests for delete_expired_chunks

354fa94

rename tests

bcdd5fb

add scripts and services for celery in vagrant

ed2c2b4

add CELERY_RESULT_EXTENDED setting

17a3ffb

remove celery and celery beat restart commands from update.sh

4ffb47f

paigewilliams requested review from Copilot, pollardld and rhodges March 10, 2026 21:09

paigewilliams self-assigned this Mar 10, 2026

Copilot started reviewing on behalf of paigewilliams March 10, 2026 21:10 View session

Copilot AI reviewed Mar 10, 2026

View reviewed changes

paigewilliams added 4 commits March 10, 2026 15:54

set redis service name as redis-server in celery service files

c24318b

add redis, celery and celery-beat to docker-compose.prod.yaml; remove…

bd06fda

… exit from healthcheck

delete delete_expired_chunks.py

ad6f113

remove unnecessary env vars from celery and celery beat containers

9633ef5

paigewilliams commented Mar 11, 2026

View reviewed changes

run as non-root user in docker to resolve warning in celery worker

ce84241

paigewilliams commented Mar 11, 2026

View reviewed changes

add test for failed to delete chunk case

4f1de26

pollardld approved these changes Mar 11, 2026

View reviewed changes

rhodges approved these changes Mar 11, 2026

View reviewed changes

paigewilliams merged commit efb9681 into develop Mar 12, 2026
2 checks passed


	# Fallback: for extremely large values, use the largest unit.
	return f"{num_bytes:.2f} P{suffix}"

Conversation

paigewilliams commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Screenshots

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pollardld left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhodges left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

paigewilliams commented Mar 10, 2026 •

edited

Loading