Celery Task Management

Instructions

Design and operate Celery distributed task queues for reliable background processing.

Broker and result backend:

• Use Redis as the message broker (broker_url = 'redis://redis:6379/0')
• Configure a result backend only when tasks need to return values — omit for fire-and-forget workloads
• Set broker_connection_retry_on_startup = True for resilient container startups

Worker configuration:

• Set concurrency based on workload type: CPU-bound tasks use process pool (match CPU count), I/O-bound tasks use gevent or eventlet pool (higher concurrency)
• Tune worker_prefetch_multiplier — set to 1 for long-running tasks to prevent worker starvation
• Use task_routes to direct specific tasks to dedicated queues (e.g., {'email.*': {'queue': 'email'}})

Task design principles:

• Make every task idempotent — safe to retry without side effects
• Keep payloads small — pass IDs and fetch data inside the task, not serialized objects
• Set task_serializer = 'json' and accept_content = ['json'] for safe serialization
• Use task_time_limit and task_soft_time_limit to prevent runaway tasks

Retry strategies:

• Use autoretry_for with specific exception types (e.g., ConnectionError, TimeoutError)
• Set max_retries (typically 3-5) and retry_backoff = True for exponential backoff
• Configure retry_backoff_max to cap delay (e.g., 600 seconds)
• Use retry_jitter = True to prevent thundering herd on recovery

Monitoring:

• Deploy Flower (celery -A proj flower) for real-time web dashboard of workers, tasks, and queues
• Use celery inspect active, reserved, stats for CLI-based diagnostics
• Monitor queue depth in Redis with redis-cli llen

Dify worker pattern:

• Run the worker container from the same image as the API container with MODE=worker
• The entrypoint script switches to celery worker mode based on the MODE environment variable
• Scale workers independently of API replicas

Celery beat for periodic tasks:

• Run a single beat instance (celery -A proj beat) — never scale beat horizontally
• Define schedules in beat_schedule dict or use django-celery-beat for database-backed schedules