When Not to Hand the Analysis to an AI
This is not an argument against the tools. It is a list of the seven places where handing the work over reliably costs more than doing it, drawn from the shape of the errors rather than from a position.
What you do: check the request against the seven cases before delegating any of it. Most requests fail none of them and are fine to accelerate.
The short version. Generation is cheap, verification is not, and verification is the part you cannot skip.
1. The measure is not defined yet
If nobody has agreed what counts as an active customer, no amount of generated SQL helps. You will get a fluent answer to one of the possible definitions, chosen invisibly, and the disagreement surfaces later with a number attached to it. Settle the definition first, using the intake questions, then delegate the mechanics.
2. The data is small enough to count
Forty rows in a spreadsheet. By the time you have described the columns, the shape and the edge cases, you could have sorted the column and read the answer. The overhead of specifying a task is roughly constant, so it dominates on small ones.
3. The number is regulated or externally reported
Anything that goes into statutory accounts, a regulatory return, a clinical report or a contractual calculation. Not because the output is worse, but because you must be able to show the derivation and defend it under scrutiny. Generated code that you have read, understood and tested is fine. Generated code that you have run is not the same thing, and the difference only becomes visible during an audit.
4. You cannot verify the output
This is the general case behind several of the others. If there is no independent number to reconcile against, no smaller version you can compute by hand, and no domain expert who can say whether the result is plausible, then you have no way to detect a wrong answer, and a fluent wrong answer is worse than no answer at all.
The test: before asking, write down how you would know if the result were wrong. If you cannot answer, that is the finding.
5. The exploration is the point
Exploratory analysis works by building a feel for the data: noticing that one region has too many round numbers, that a column of dates has a suspicious cluster on the first of the month, that a category appeared in March and never before. That noticing happens while you look, and it does not transfer through a summary. Hand over the summary and you get the summary; the anomalies stay behind. See exploratory data analysis.
6. The data is sensitive
Personal data, health records, payroll, unreleased financials, anything under a client confidentiality clause. The question is not whether the tool is good, it is whether sending the data is permitted, and that is answered by your organisation policy and the contract covering the tool, not by convenience. Sample it, synthesise it, or use an approved route. Pasting is not reversible.
7. The request is really a decision
"Should we close the Tuesday slot?" is not a data question with a data answer. The data narrows it: volume, margin, the customers affected. The decision involves things not in any table, such as a contract, a relationship, or what the team can staff. Presenting a generated recommendation as an analytical result hides the judgement inside it, which is precisely the part that needed a human name attached.
Where it genuinely helps
| Good use | Why it works |
|---|---|
| Boilerplate code and format conversions | Verification is instant: it runs or it does not |
| Explaining an unfamiliar function or error | You check the explanation against the documentation |
| A first draft of a long query you will read line by line | You retain the reading, which was the expensive part |
| Naming things and writing the caveats section | Low risk, and you can judge the output directly |
| Generating test data | Nothing real depends on it |
| Rubber-ducking a definition problem | The output is a prompt for your thinking, not the answer |
The pattern in that column is that verification is cheap and the failure is loud. That is the condition worth looking for, and it holds far more often for code than for conclusions.
How to apply this to your own work
- Before delegating any task, write one sentence on how you would detect a wrong answer. Keep the sentence.
- Check your organisation policy once, properly, and know what you may paste. Do it before the day you are in a hurry.
- Split every request into definition, mechanics and interpretation. The middle third is the part worth accelerating.
- For anything reported externally, keep the derivation in a file a colleague could follow without you.
- Time yourself once on a small task done both ways. The result is usually not what either camp expects.
The one habit to keep
Own the number. Whatever produced it, your name is on it, and the only question that matters in the room is whether you can explain how it was derived and what would make it wrong.
On the last number you published, could you show the derivation without opening a chat history?
Thinking Like an Analyst is about the part no tool does for you: framing the question, choosing the measure, saying what the data cannot support, and defending a number in a room.
Thinking Like an Analyst, $19 →Verifying AI agent work and reviewing AI-generated SQL cover the checks. The request intake covers the part that has to happen before either.
Read Verify AI Agent Work →