Copperleaf's Go order pipeline held at 40 times normal volume on Black Friday

The problem

Copperleaf hosts online shops for 4,800 independent businesses: bookshops, bike shops, specialist food. Every order sets off a chain of work. The payment is settled, stock is updated across sales channels, the order goes to a warehouse or a shipping service, and the customer gets emails along the way.

Copperleaf had built its own queue on Postgres with SELECT … FOR UPDATE SKIP LOCKED, about 1,900 lines of Go. It was fast on a normal day. It had no fairness, though, so one merchant’s flash sale could fill the queue and every other shop’s orders waited behind it. On Black Friday 2024 the slowest 1% of orders took 19 minutes to reach their warehouse, and the team spent the weekend tuning worker counts by hand.

What they built

Copperleaf replaced the queue with the Runnel Go SDK. Each order runs as a workflow with steps for payment, stock, fulfilment and email. The key change is a concurrency key per merchant: no merchant can use more than 25 workers at once, however many orders arrive, so a sale in one shop no longer delays another.

fulfil := runnel.NewWorkflow("fulfil-order",
	func(ctx runnel.Context, o Order) error {
		if _, err := runnel.Step(ctx, "settle-payment", func() (Payment, error) { return payments.Settle(o) }); err != nil {
			return err
		}
		if _, err := runnel.Step(ctx, "reserve-stock", func() (bool, error) { return stock.Reserve(o) }); err != nil {
			return err
		}
		_, err := runnel.Step(ctx, "send-to-warehouse", func() (string, error) { return warehouse.Submit(o) })
		return err
	},
	runnel.WithRetries(6, runnel.Exponential),
	runnel.WithConcurrencyKey(25, func(o Order) string { return o.MerchantID }),
)

Workers run on Cloud Run and scale on Runnel’s queue-depth metric. Warehouse partners with their own rate limits get a second, global concurrency limit on the send-to-warehouse step’s job.

Copperleaf also uses run.failed webhooks to open a ticket in the merchant’s dashboard when an order cannot be sent after six attempts, usually because a product’s warehouse code is wrong. The merchant can fix the code and retry the order themselves.

Results

Black Friday 2025 peaked at 40 times Copperleaf’s normal order rate and the queue never built a backlog. The slowest 1% of orders reached their warehouse within 38 seconds, against 19 minutes the year before. No one tuned anything during the weekend.

The team deleted 1,900 lines of queue code and the on-call runbook that went with it. Merchants now fix most failed orders themselves from the ticket the webhook opens. Support handles about a third as many order tickets as it did before the move.

Copperleaf runs about 31 million runs a month on Runnel Cloud and was one of the design partners for concurrency keys, which shipped to everyone in Runnel 2.2.