Course resource
Error Handling Template
Automations fail silently by default. This is how to make them fail loudly instead.
The four failure modes
| Mode | What happens | How you find out |
|---|---|---|
| Hard error | A step throws; the run stops | Usually a platform notification |
| Silent skip | A filter excludes everything; nothing runs | Never, without monitoring |
| Bad data through | A step returns junk; later steps use it | When a human notices the output |
| Runaway | A loop or trigger fires far more than expected | The invoice |
The platform tells you about the first one. The other three are yours to catch, and they are the expensive ones.
The minimum every automation needs
- An error route that notifies a human with enough detail to debug
- A run log capturing the AI step's input and output
- A heartbeat — an alert if it has NOT run when it should have
- A volume alert — if runs exceed expected by 3x
- A documented kill switch
The error notification
Make it actionable. "Workflow failed" tells you nothing at 9pm.
AUTOMATION FAILED
Workflow: [NAME]
Step: [WHICH STEP]
Time: [TIMESTAMP]
Run ID: [ID]
Input that caused it:
[THE DATA, TRUNCATED]
Error:
[THE MESSAGE]
What this means: [YOUR PLAIN-ENGLISH NOTE]
What to do: [RETRY / IGNORE / FIX / ESCALATE]
Write the last two lines when you build it, not when it breaks. You understand the workflow now; you will not in four months.
Validating AI output before using it
The most common source of bad-data-through. An AI step returns something malformed and the next step happily uses it.
After every AI step, before the next step:
1. Is the output non-empty?
2. If JSON was requested, does it parse?
3. Are the required fields present?
4. Is the classification one of the allowed values?
5. Is the length within expected bounds?
If any check fails -> error route. Do not continue with bad data.
Normalise before comparing: trim, lowercase, strip trailing punctuation. "Billing." is not "billing", and that one character sends everything to your fallback.
Retries
Retry transient failures. Do not retry logical ones.
| Failure | Retry? |
|---|---|
| Network timeout | Yes — 3 attempts, increasing delay |
| Rate limit | Yes — wait, then retry |
| Provider 5xx | Yes |
| Malformed AI output | Once, then error route |
| Invalid input data | No — error route |
| Authentication failure | No — error route and alert loudly |
Cap retries and add delay between them. Immediate retries against a rate limit make the rate limit worse.
The heartbeat
The failure nobody catches: the automation stops running entirely. No errors, because nothing ran.
Separate scheduled check, once a day:
If [WORKFLOW] has not run in the last [EXPECTED INTERVAL]
-> alert me
Twenty minutes to build. It is the only thing that catches a disabled trigger, an expired credential, or a platform change.
Cost runaway protection
- Maximum runs per hour, enforced at the trigger
- Maximum iterations on every loop
- Alert when daily runs exceed [N]
- Alert when daily spend exceeds [AMOUNT]
- A hard spending cap at the provider, where available
Calculate before launch: worst-case cost per run × maximum possible runs per day × 30. If that number is uncomfortable, lower the caps.
The incident log
| Date | Workflow | What failed | Cause | Fix | Prevented recurrence? |
|---|---|---|---|---|---|
The last column is the point. Most incidents are a previous incident that was patched rather than fixed.
The runbook
One per automation. Someone else should be able to use it.
AUTOMATION: [NAME]
Owner: [WHO] Backup owner: [WHO]
What it does:
Runs: [SCHEDULE OR TRIGGER]
Expected volume: [N] per [PERIOD]
Cost: [PER RUN] / [PER MONTH]
TO STOP IT: [EXACT STEPS]
Common failures:
| Symptom | Cause | Fix |
|---|---|---|
| | | |
If you cannot fix it: [WHO TO CALL]
Manual fallback while it is down: [WHAT TO DO INSTEAD]
The manual fallback line matters. If an automation being down stops work entirely, that is a dependency you should know about before it happens.
Review
Monthly, per automation:
- Did it run as expected every day?
- Any errors, and were they handled?
- Cost in line with the estimate?
- Does the runbook still describe what it actually does?
- Is it still worth running?
That last question retires things. Automations accumulate, keep running, keep costing money, and stop being useful long before anyone turns them off.