Queues & jobs

Why a failed job does not go straight to failed_jobs

The mistake

Two things people get wrong about failed jobs. The first: they read $tries = 3 as three retries, so four runs in total. The second: they think a job that throws lands in failed_jobs right away.

Neither is true. $tries is the total number of attempts, so three attempts, two retries. And a job that throws with attempts left is not failed. It is put back on the queue with a delay and tried again. It only reaches failed_jobs once the attempts run out.

The machine

Drive it

Time only moves when you move it. After a failure or a kill, the main button becomes Advance, which jumps the clock to the next event.

• Dispatch a job, reserve it, then fail it. The status goes to delayed, not failed. The attempt counter reads one of three, and a backoff timer starts.
• Advance time, reserve, and fail again, twice more. Only on the third failed attempt, with no tries left, does it move to the failed state.
• Reserve, then kill the worker. The worker is marked gone, but the job stays reserved. Advance time past retry_after and it returns to the queue, never having been marked failed.

The mechanism

When a worker picks up a job, it reserves it and counts the attempt. So attempts() is one during the first run, not zero. That is why $tries is a count of attempts and not of retries.

If the job throws, the worker checks the count. With attempts left, it releases the job with a delay. That delay is the backoff, and the job becomes available again once it passes. Every throw repeats this until the attempts are used up. Only then does the job move to failed_jobs and failed() runs.

Killing the worker is different. The reserved record just sits there. Laravel does not know the worker died. After retry_after seconds, the connection treats the reservation as stale, and the job becomes available again. Another worker reserves it and counts the next attempt. This is why retry_after must be longer than your slowest job. Otherwise a job that is still running is reserved a second time.

In your code

class ProcessPodcast implements ShouldQueue
{
    public int $tries = 3;     // total attempts, not retries
    public int $backoff = 10;  // seconds to wait before each retry

    public function handle(): void
    {
        // ... work that might throw
    }
}

retry_after lives on the connection, in config/queue.php. Keep it larger than your worker --timeout, so a slow job is never reserved twice.

The fine print

• One worker and one job. Real queues run many workers over many jobs at once.
• Queue priorities and --queue=high,low ordering.
• Batches, chains, and Bus::batch, which track a group of jobs together.
• backoff as an array for a growing delay per attempt, and retryUntil for a time based limit instead of a count.

Further reading

• Laravel: retrying jobs and backoff covers $tries, retryUntil and per attempt backoff straight from the source.
• Laravel: dealing with failed jobs is what happens once the attempts run out, and how to retry from the failed_jobs table.