Building and Debugging It

You have a design. Now build it in the order that makes failures findable.

The temptation is to build all six steps and then run it. Do that and when something is wrong you are debugging six things at once, with no idea which one broke.

---

1. Build one step at a time

Step 1 — Trigger only. Nothing else. Run it. Confirm it fires, and look at the data it produces. Not "it probably works" — look at it.

Step 2 — Add the filter. Test twice: once with something that should pass, once with something that should not. Do not proceed until both behave.

Step 3 — Add the first AI step. Run it on ten real inputs. Check two things: is the output correct, and is it the same on a repeat run of the same input?

Step 4 — Add validation. Does the output parse? Is it one of the allowed values? If not, route to the error path rather than continuing with bad data.

Step 5 — Add the remaining steps. One at a time. Test after each.

Step 6 — Add the output. Drafts to a drafts folder. Never straight to send.

Step 7 — Add the error path. Every failure tells a human which step, which input, and what the error was.

---

2. Test small, then scale

Do not run your new system on 1,000 items.

1 item    -> read every intermediate output by hand
3 items   -> check all three end to end
10 items  -> check for consistency between runs
50 items  -> watch the cost
Then full volume, watching for a day

The one-item stage is where you catch the problems that would otherwise be a thousand wrong records.

---

3. Debugging: isolate before you fix

Which step is wrong? Look at the input and output of every step individually. Do not infer it from the final output.

Most people rewrite the prompt when the real problem is that the previous step passed in an empty field.

• Is the trigger firing?
• Is the filter letting the right things through?
• Does each step receive what you think it receives?
• Does the AI step's raw output look right, before anything parses it?
• Does the output land where you think?

---

4. The five causes, in the order they actually occur

1. The input is not what you think. An empty field, a renamed column, whitespace, a date stored as text. Print it and look.

2. The output is not normalised. "Billing." does not match billing. Trim, lowercase, strip punctuation before comparing. This is the single most common "my router is broken".

3. Temperature too high on a judgement step. Same input, different answer between runs. Set classification and extraction to 0–0.2.

4. The prompt is under-specified. You know what you meant. The model does not. Add the constraint you assumed was obvious.

5. The filter is wrong. Letting everything through, or nothing. Test it in isolation.

---

5. Symptom table

Symptom	Check first
Nothing runs	Trigger enabled? Filter too tight? Credentials expired?
Runs on everything	No filter, or the condition is inverted
Different result each run	Temperature
Everything hits the fallback	Output normalisation
Empty AI output	The input was empty — check the previous step
It invents details	Missing "do not infer" instruction
Works on one record, fails on another	An edge case in the data. Find what differs.
Worked last week, not now	Model version changed, credential expired, upstream format changed
Costs far more than expected	No filter; or a loop without a cap
Truncated mid-sentence	Output token limit too low

---

6. Debugging the AI step specifically

If the AI step really is the problem:

Run the exact prompt manually in a chat window, with the exact input. If it works there, your input is not what you think it is.

Ask it what it understood. Temporarily add: "Before answering, restate what you think I am asking."

Simplify to the minimum that works, confirm it does, then add instructions back one at a time.

Check for conflicting instructions. Long prompts often contain two rules that cannot both hold.

Check instruction position. Instructions at the end of a long prompt get dropped first. Move the critical ones to the top.

If it is inventing facts

Use only the information above. If it is not there, write UNKNOWN.

and

For each claim, quote the exact sentence you relied on.

The quote-back instruction catches more than anything else, because paraphrasing is where drift happens.

---

7. The self-correction step

Adding a review pass to the end of a workflow is genuinely useful — with one caveat.

Review the output above against these criteria:
- Does it answer the actual question?
- Is every fact present in the source material?
- Is it under [N] words?

If it fails any, rewrite it. If it passes, return it unchanged.

The caveat: this costs an extra call every run, and it is not a substitute for a human on anything that reaches a customer. It catches sloppiness; it does not catch confident wrongness, because the same model that produced the error is now checking for it.

---

8. The discipline

Change one thing at a time. Change three and you learn nothing about which mattered.

Write down what you tried. Half an hour in, you will not remember.

Attempt	Changed	Result
1
2
3

Keep a working copy. Before editing anything that works, duplicate it. Roll back if you go backwards.

---

9. When to stop debugging

If you have spent more than an hour on one problem:

• Rebuild the step from scratch. Often faster than finding the fault.
• Simplify the requirement. Does the workflow need to handle this case, or can a human take the exception?
• Ask whether the process is the problem. Some things are hard to automate because they are badly defined, not because the tool is weak.

---

⚠️ Common Mistakes

• Building everything then testing. Debugging six things at once.
• Testing on real volume first. One item, then three, then ten.
• Changing three things at once. You learn nothing.
• Rewriting the prompt when the input was empty. Isolate first.
• No working copy. Nowhere to roll back to.
• Testing in production. Running a new system on real customers before a single dry run.
• Assuming the model is the problem. It usually is not. It is usually the input or the normalisation.

---

What's Next: It works. Now make it survivable — documented well enough that it still runs when you are not there.