Queues & jobs

Why a failed job does not go straight to failed_jobs

The mistake

Two things people get wrong about failed jobs. The first: they read $tries = 3 as three retries, so four runs in total. The second: they think a job that throws lands in failed_jobs right away.

Neither is true. $tries is the total number of attempts, so three attempts, two retries. And a job that throws with attempts left is not failed. It is put back on the queue with a delay and tried again. It only reaches failed_jobs once the attempts run out.

The machine

Drive it

Time only moves when you move it. After a failure or a kill, the main button becomes Advance, which jumps the clock to the next event.

• Dispatch a job, reserve it, then fail it. The status goes to delayed, not failed. The attempt counter reads one of three, and a backoff timer starts.
• Advance time, reserve, and fail again, twice more. Only on the third failed attempt, with no tries left, does it move to the failed state.
• Reserve, then kill the worker. The worker is marked gone, but the job stays reserved. Advance time past retry_after and it returns to the queue, never having been marked failed.

The mechanism

When a worker picks up a job, it reserves it and counts the attempt. So attempts() is one during the first run, not zero. That is why $tries is a count of attempts and not of retries.

If the job throws, the worker checks the count. With attempts left, it releases the job with a delay, the backoff, and the job becomes available again once that delay passes. Every throw repeats this until the attempts are used up. Only then does the job move to failed_jobs and failed() runs.

Killing the worker is different. The reserved record just sits there. Laravel does not know the worker died. After retry_after seconds, the connection treats the reservation as stale and the job becomes available again for another worker, which reserves it and counts the next attempt. This is why retry_after must be longer than your slowest job, or a job still running gets picked up a second time.

In your code

class ProcessPodcast implements ShouldQueue
{
    public int $tries = 3;     // total attempts, not retries
    public int $backoff = 10;  // seconds to wait before each retry

    public function handle(): void
    {
        // ... work that might throw
    }
}

retry_after lives on the connection, in config/queue.php. Keep it larger than your worker --timeout, so a slow job is never reserved twice.

The fine print

• One worker and one job. Real queues run many workers over many jobs at once.
• Queue priorities and --queue=high,low ordering.
• Batches, chains, and Bus::batch, which track a group of jobs together.
• backoff as an array for a growing delay per attempt, and retryUntil for a time based limit instead of a count.

Further reading

• Laravel: retrying jobs and backoff covers $tries, retryUntil and per attempt backoff straight from the source.
• Laravel: dealing with failed jobs is what happens once the attempts run out, and how to retry from the failed_jobs table.