> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sluice.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Celery Failure Modes

> The five most common failure patterns in Celery deployments and how Sluice helps detect them.

These are the most common failure patterns in Celery deployments. Sluice helps you detect and diagnose each one.

## 1. Silent task stalls

A task enters the `active` state and never completes — no success, no failure, no error. The worker appears healthy but the task hangs forever.

**Common causes:**

* No `task_time_limit` configured (Celery default: no timeout)
* Task blocked on an external service that's unresponsive (database, API, DNS)
* Deadlock in task code

**What Sluice shows:** The job stays in `active` state with increasing duration. Filter by `state=active` and sort by `startedAt` ascending to find stalled jobs.

**Fix:** Set `task_time_limit` and `task_soft_time_limit` in your Celery config:

```python theme={null}
task_soft_time_limit = 300  # Raise SoftTimeLimitExceeded after 5 min
task_time_limit = 600       # Hard-kill task after 10 min
```

## 2. Worker OOM kills

The Linux OOM killer terminates a worker process. The task that was running is lost — no failure event, no traceback, just silence.

**What Sluice shows:** The job's state is `active`, the worker transitions to `offline`, and the job never reaches `completed` or `failed`. A gap between `worker-offline` and the job's last state change is a strong OOM signal.

**Fix:**

* Set `worker_max_memory_per_child` to recycle workers before they hit system limits
* Use `task_time_limit` to kill tasks that grow unbounded
* Profile your tasks for memory usage

## 3. Visibility timeout duplicates

When a task takes longer than Redis's `visibility_timeout` (default: 1 hour), the broker assumes the worker died and redelivers the message. The original worker is still running the task — resulting in duplicate execution.

**What Sluice shows:** Two `active` events for the same `externalId`. If you see the same task running concurrently on different workers, it's a visibility timeout issue.

**Fix:**

```python theme={null}
app.conf.broker_transport_options = {
    'visibility_timeout': 43200,  # 12 hours
}
```

Set the visibility timeout higher than your longest-running task. For tasks with very long execution times, consider using `acks_late=True` with careful retry configuration.

## 4. Prefetch blindness

Workers prefetch tasks from the broker into memory. These tasks are invisible — the broker reports them consumed, but they haven't started executing. If the worker dies, all prefetched tasks are lost.

**What Sluice shows:** Jobs transition from `queued` to a long pause before `active`, or the gap between `task-received` and `task-started` is unusually large.

**Fix:**

```python theme={null}
worker_prefetch_multiplier = 1  # Default is 4
```

For long-running tasks, set `worker_prefetch_multiplier = 1` so workers only grab one task at a time.

## 5. Broker disconnects

When the Redis broker goes down temporarily, workers lose their connection. Celery automatically reconnects, but event consumers may not resume correctly — leading to a gap in monitoring data.

**What Sluice shows (agent path):** The agent logs reconnection attempts and resumes event subscription after reconnecting. Events during the disconnect window are lost (Redis PUB/SUB limitation), but the agent reconciles state by scanning keys on reconnection.

**What Sluice shows (SDK path):** The SDK's event forwarding continues normally after the worker reconnects to Redis. The worker handles reconnection — the SDK captures events from the worker's perspective.

**Fix:** The agent handles this automatically with exponential backoff. For prolonged Redis outages, ensure your Celery `broker_connection_retry` setting is `True` (default).