# Retries and timeouts

> How Runnel retries failed attempts, how to tune backoff and timeouts, and how to stop a run from retrying when it never will succeed.

Source: https://galley.themes.dev/docs/guides/retries/

Background work fails for dull reasons: a provider returns a 503, a connection resets, a deploy restarts the worker mid-run. Runnel retries these for you. A retry is a new **attempt** of the same run, with the same id and payload, so it shows up as one run in the dashboard and counts once on your bill.

## The defaults

Without a `retries` option, a job gets three attempts in total, with exponential backoff starting at five seconds and capped at five minutes, plus up to 20% jitter. Each attempt may run for 15 minutes.

| Attempt | Waits before it (roughly) |
|---|---|
| 1 | none |
| 2 | 5 s |
| 3 | 10 s |

After the last attempt fails, the run's status becomes `failed` and Runnel sends a `run.failed` [webhook](/docs/reference/webhooks/).

## Tuning retries

Set `retries` on the job. Durations are strings such as `'500ms'`, `'10s'`, `'5m'` or `'1h'`.

```ts title="src/jobs/sync-supplier.ts"
export const syncSupplier = runnel.job({
  name: 'sync-supplier',
  retries: {
    attempts: 8,
    backoff: 'exponential', // or 'fixed'
    initialDelay: '30s',
    maxDelay: '30m',
    jitter: 0.2,
  },
  timeout: '5m',
  run: async ({ supplierId }: { supplierId: string }, ctx) => {
    const res = await fetch(`https://suppliers.example/${supplierId}/feed`, { signal: ctx.signal });
    if (!res.ok) throw new Error(`feed returned ${res.status}`);
    await importFeed(supplierId, await res.json());
  },
});
```

With eight attempts from 30 seconds doubling to a 30-minute cap, the last attempt starts about two hours after the first. That is a sensible window for a supplier that goes down for maintenance.

> **Tip** Pick the window first ("keep trying for about two hours"), then work out attempts and delays. `runnel dev` prints the schedule for each job when it starts.

## Errors that should not be retried

Some failures will be the same on every attempt: a deleted record, a malformed payload, a 4xx from an API that will not change its mind. Throw `runnel.fatal` and the run fails at once, without using up its remaining attempts.

```ts
const invoice = await db.invoices.find(payload.invoiceId);
if (!invoice) throw runnel.fatal(`invoice ${payload.invoiceId} no longer exists`);
```

To retry after a specific delay, for example when an API sends `Retry-After`, throw `runnel.retryAfter`:

```ts
if (res.status === 429) throw runnel.retryAfter(res.headers.get('retry-after') + 's');
```

In Python these are `raise FatalError("…")` and `raise RetryAfter("30s")`; in Go, return `runnel.Fatal(err)` or `runnel.RetryAfter(30*time.Second)`.

## Timeouts

A timeout applies to one attempt, not the whole run. When it is reached, `ctx.signal` is aborted and the worker gives the attempt five more seconds to stop before it is counted as failed and retried.

Pass `ctx.signal` to anything that waits: `fetch`, database queries, child processes. Code that ignores it keeps running in the background after the attempt has been written off, which is how duplicate emails happen.

> **Warning** A timeout does not stop code that is blocking the event loop. A long synchronous loop runs to the end, whatever the timeout says.

## Making attempts safe to repeat

Because an attempt can fail halfway, the next one may repeat work the first one already did. Two habits cover most cases:

- Use the run id as an idempotency key with the services you call. Most payment and email providers accept one, and `ctx.run.id` is the same on every attempt.
- Check before you write. "Mark the invoice as sent" should be "mark it as sent unless it already is".

For work with several side effects in a row, use a [workflow](/docs/guides/workflows/). Each finished step is stored, so a retry picks up at the step that failed instead of starting from the top.

## Retrying by hand

A run that has used all its attempts can be retried from the dashboard, the CLI or the API. A manual retry starts a fresh set of attempts on the same run.

```bash
runnel runs list --job sync-supplier --status failed --limit 5
runnel runs retry run_01JB7X2K9Q4MZ8
```

For workflows, `runnel runs retry <run-id> --from-step <name>` replays from a chosen step. See the [CLI reference](/docs/reference/cli/).