Course resource

Debugging Checklist

When your AI workflow does not do what you expected. Work down in order — the cheap checks first.

First, isolate

Which step is wrong? Run the workflow and look at the input and output of every step individually. Do not guess from the final output.

Most people rewrite the prompt when the actual problem is that the step before it passed in an empty field.

In the order they actually occur:

1. The input is not what you think. An empty field, a different column name, whitespace, a date as text. Print the input and look at it.

2. The output is not normalised. "Billing." does not match "billing". Trim, lowercase, strip punctuation before comparing.

3. Temperature too high on a judgement step. Same input, different answer on different runs. Set classification and extraction to 0–0.2.

4. The prompt is under-specified. You know what you meant; the model does not. Add the constraint you assumed was obvious.

5. The filter is wrong. Either letting everything through, or nothing. Test it in isolation with a known-good and a known-bad input.

Symptom	Check first
Nothing runs	Trigger enabled? Filter too tight? Credentials expired?
Runs on everything	No filter, or filter condition inverted
Different result each run	Temperature; add a seed if testing
Everything hits the fallback route	Output normalisation before comparison
Empty output from AI step	Input was empty; check the step before
AI invents details	No "do not infer" instruction; add the escape hatch
Works on one record, fails on another	Edge case in the data — find what differs
Worked last week, not now	Model version changed, credential expired, or upstream format changed
Costs far more than expected	No filter; or a loop without a cap; or the expensive model on a cheap step
Truncated mid-sentence	max_tokens too low

If the AI step is genuinely the problem:

Run the exact prompt manually in the chat interface with the exact input. Does it fail there too? If not, the input is different from what you think.
Ask it what it understood. Add "before answering, restate what you think I am asking" temporarily.
Simplify. Strip the prompt to the minimum that should work, confirm it does, then add back one instruction at a time.
Check for conflicting instructions. Long prompts often contain two rules that cannot both hold.
Check instruction position. Instructions at the end of a long prompt get dropped first. Move the critical ones up.
Check the input length. A very long input can push your instructions out of effective attention.

If it is inventing facts:

The quote-back instruction catches more than anything else, because paraphrasing is where drift happens.

Check the model version before rewriting the prompt. It is the most common cause of "nothing changed and it broke".

Change one thing at a time. Change three and you learn nothing about which mattered.

Write down what you changed and what happened. Half an hour in, you will not remember what you have already tried.

Keep a working version. Before editing anything live, copy it. Roll back if you go backwards.

If you have spent more than an hour:

Rebuild the step from scratch rather than fixing it. Often faster.
Simplify the requirement. Does the workflow need to handle this case at all, or can a human take the exception?
Ask whether the process is the problem. Some things are hard to automate because they are badly defined, not because the tool is weak.

That last one is usually worth ten minutes. Most people have the same mistake in three places.