Retries and timeouts

Background work fails for dull reasons: a provider returns a 503, a connection resets, a deploy restarts the worker mid-run. Runnel retries these for you. A retry is a new attempt of the same run, with the same id and payload, so it shows up as one run in the dashboard and counts once on your bill.

The defaults

Without a retries option, a job gets three attempts in total, with exponential backoff starting at five seconds and capped at five minutes, plus up to 20% jitter. Each attempt may run for 15 minutes.

After the last attempt fails, the run’s status becomes failed and Runnel sends a run.failed webhook.

Tuning retries

Set retries on the job. Durations are strings such as '500ms', '10s', '5m' or '1h'.

export const syncSupplier = runnel.job({
  name: 'sync-supplier',
  retries: {
    attempts: 8,
    backoff: 'exponential', // or 'fixed'
    initialDelay: '30s',
    maxDelay: '30m',
    jitter: 0.2,
  },
  timeout: '5m',
  run: async ({ supplierId }: { supplierId: string }, ctx) => {
    const res = await fetch(`https://suppliers.example/${supplierId}/feed`, { signal: ctx.signal });
    if (!res.ok) throw new Error(`feed returned ${res.status}`);
    await importFeed(supplierId, await res.json());
  },
});

With eight attempts from 30 seconds doubling to a 30-minute cap, the last attempt starts about two hours after the first. That is a sensible window for a supplier that goes down for maintenance.

Tip Pick the window first (“keep trying for about two hours”), then work out attempts and delays. runnel dev prints the schedule for each job when it starts.

Errors that should not be retried

Some failures will be the same on every attempt: a deleted record, a malformed payload, a 4xx from an API that will not change its mind. Throw runnel.fatal and the run fails at once, without using up its remaining attempts.

const invoice = await db.invoices.find(payload.invoiceId);
if (!invoice) throw runnel.fatal(`invoice ${payload.invoiceId} no longer exists`);

To retry after a specific delay, for example when an API sends Retry-After, throw runnel.retryAfter:

if (res.status === 429) throw runnel.retryAfter(res.headers.get('retry-after') + 's');

In Python these are raise FatalError("…") and raise RetryAfter("30s"); in Go, return runnel.Fatal(err) or runnel.RetryAfter(30*time.Second).

Timeouts

A timeout applies to one attempt, not the whole run. When it is reached, ctx.signal is aborted and the worker gives the attempt five more seconds to stop before it is counted as failed and retried.

Pass ctx.signal to anything that waits: fetch, database queries, child processes. Code that ignores it keeps running in the background after the attempt has been written off, which is how duplicate emails happen.

Warning A timeout does not stop code that is blocking the event loop. A long synchronous loop runs to the end, whatever the timeout says.

Making attempts safe to repeat

Because an attempt can fail halfway, the next one may repeat work the first one already did. Two habits cover most cases:

• Use the run id as an idempotency key with the services you call. Most payment and email providers accept one, and ctx.run.id is the same on every attempt.
• Check before you write. “Mark the invoice as sent” should be “mark it as sent unless it already is”.

For work with several side effects in a row, use a workflow. Each finished step is stored, so a retry picks up at the step that failed instead of starting from the top.

Retrying by hand

A run that has used all its attempts can be retried from the dashboard, the CLI or the API. A manual retry starts a fresh set of attempts on the same run.

runnel runs list --job sync-supplier --status failed --limit 5
runnel runs retry run_01JB7X2K9Q4MZ8

For workflows, runnel runs retry <run-id> --from-step <name> replays from a chosen step. See the CLI reference.