Cover image for Chain-of-Thought Prompting Explained

Chain-of-Thought Prompting Explained

By AppliedAI

Most AI outputs fail not because the model lacks knowledge, but because it skips the work of actually thinking through the problem.

Ask an LLM a complex question with a single sentence and it will pattern-match to the most statistically probable answer. On simple questions, that’s fast and fine. On multi-step problems — math, logic, planning, causal analysis — that shortcut produces confident-sounding nonsense. Chain-of-thought (CoT) prompting is the technique that forces the model to stop skipping and actually show its reasoning.

What Is Chain-of-Thought Prompting

Chain-of-thought prompting is a prompting strategy where you instruct the model to generate the intermediate reasoning steps before producing a final answer — in the same way a human might write out a calculation rather than trying to do it entirely in their head.

It was formally introduced and named in a 2022 Google Brain paper by Wei et al., but the underlying mechanic is intuitive: if you ask someone to explain their thinking out loud, they tend to catch their own errors. The same dynamic applies to language models.

The critical insight from that research is that CoT dramatically improves model performance on tasks that require multi-step reasoning — tasks where each step depends on the previous one being correct.

Why Skipping Reasoning Steps Breaks LLM Output

To understand why CoT works, you need a basic model of how LLMs generate text.

An LLM doesn’t “think” and then “write.” It generates one token (roughly one word) at a time, with each token influenced by everything before it. If the model jumps straight to a conclusion without generating intermediate reasoning tokens, those reasoning steps never actually happen — they’re just absent from the computation.

Generating the reasoning in the output isn’t just showing your work for the reader’s benefit. Writing the steps is how the model does the steps. This is the counterintuitive core of why CoT works mechanically.

When a model produces “Let me break this down step by step…” and then executes those steps in its output, it is using those generated tokens as working memory. Remove those tokens and you remove the cognitive scaffolding the model relies on to get the answer right.

The Two Main Forms of Chain-of-Thought Prompting

Few-Shot CoT (Demonstrated Reasoning)

The original form of CoT is few-shot: you provide the model with one or more examples of a solved problem that include the reasoning trace, not just the final answer. The model learns the expected output format from those examples and replicates the pattern.

Example format:

Q: A store buys apples for $0.40 each and sells them for $0.65 each. If they sell 300 apples, what is the total profit?

A: First, I calculate the profit per apple: $0.65 - $0.40 = $0.25.
Next, I multiply by the number sold: $0.25 × 300 = $75.00.
The total profit is $75.00.

Q: [Your actual question here]

A: [Model generates step-by-step reasoning]

The example teaches the model two things simultaneously: the pattern of working through a problem and the depth of reasoning you expect.

Zero-Shot CoT (Instruction-Triggered Reasoning)

In 2022, researchers at Google and elsewhere discovered something almost absurdly simple: appending the phrase “Let’s think step by step” to a prompt — with no example at all — significantly improved model accuracy on reasoning tasks.

This is zero-shot CoT. It works because that phrase shifts the model’s output distribution toward explanatory, sequential content rather than direct concluding statements. The model has been trained on enormous amounts of text where “let’s think step by step” is followed by careful reasoning, and it reproduces that pattern.

Common zero-shot CoT triggers that reliably work:

  • “Think through this step by step before answering.”
  • “Break this down into logical steps and reason through each one.”
  • “Before giving your final answer, explain your reasoning.”
  • “Work through this problem carefully. Show every step.”

The precise phrasing matters less than the core instruction: generate reasoning before conclusions.

When Chain-of-Thought Prompting Is Worth Using

CoT is a deliberate overhead — it produces longer outputs, takes more time, and on advanced models costs more tokens. You don’t need it for every task.

Use chain-of-thought when:

  • The task involves multiple dependent steps (math, logic puzzles, code debugging, multi-condition decisions)
  • Accuracy matters more than speed, and a wrong answer has real consequences
  • You need the model’s reasoning to be auditable — you need to verify how it got to the answer, not just what the answer is
  • The model is consistently producing wrong answers on a complex task and you need to diagnose where the reasoning breaks down

Skip it when:

  • The task is single-step (summarize this, translate this, classify this)
  • You’re generating content where reasoning traces are noise (marketing copy, simple Q&A)
  • Speed and token efficiency matter and the task is well within the model’s zero-shot competence

CoT prompts are meaningfully longer than standard prompts, and at scale that token overhead compounds. If you’re running reasoning-heavy CoT prompts across automated workflows, it’s worth modeling the cost before you commit. The LLM Cost Calculator shows you exactly how output length affects API cost across GPT-4o, Claude 3.5, and Gemini — useful before you scale a CoT pipeline that runs hundreds of times per day.

Structuring a High-Quality Chain-of-Thought Prompt

The trigger phrase alone is often enough to activate reasoning in powerful models. But a well-structured CoT prompt does more.

A complete CoT prompt combines the elements of a structurally sound prompt — role, context, task, constraints — with an explicit reasoning instruction. If you haven’t already read through The Anatomy of a Perfect Prompt, the structural framework there applies directly here: CoT is a constraint you layer on top of an already well-formed prompt, not a substitute for the other components.

Here’s what a complete CoT prompt looks like in practice:

You are a financial auditor reviewing an expense report for policy compliance.

Policy rules:
- Meals may not exceed $75 per person per day
- International travel requires VP approval if total cost exceeds $5,000
- Equipment purchases over $2,500 must have three vendor quotes attached

Here is the expense report:
[INSERT REPORT]

Review each line item against the policy rules above.
For each item, state: (1) which rule applies, (2) whether it is compliant or not, 
and (3) what action is required if it is non-compliant.
Think through each line item carefully before flagging any violations.

Notice what this prompt does: it defines the model’s role, provides the exact data and rules, specifies the output format at the item level, and then instructs careful step-by-step reasoning as the final directive. That ordering matters. The reasoning instruction at the end of a prompt acts as the final contextual weight before generation begins.

The Difference Between Chain-of-Thought and Self-Consistency

CoT tells the model to reason. Self-consistency takes it further: you run the same CoT prompt multiple times, collect several independent reasoning chains, and take the most common final answer as the output.

This works because individual reasoning chains can still go wrong — they’re probabilistic. Sampling multiple chains and taking the majority answer reduces variance significantly on tasks where correctness is binary (math problems, factual questions with definitive answers).

Self-consistency is expensive (you’re multiplying your token cost by however many samples you take) and impractical in real-time applications. But for high-stakes, batch-processing contexts where accuracy is worth the cost, it’s a legitimate upgrade to standard CoT.

Common Mistakes That Negate Chain-of-Thought

Applying CoT to the wrong task type. Asking a model to “think step by step” before writing a product description adds noise without benefit. The model now generates filler reasoning about marketing principles before producing essentially the same copy. Reserve CoT for genuinely multi-step reasoning tasks.

Treating the reasoning as ground truth. CoT improves accuracy — it doesn’t guarantee it. The model can reason coherently through a chain of steps and still reach a wrong conclusion if one premise is wrong or hallucinated. Always verify numerical answers and factual claims independently.

Using weak trigger phrases. “Please explain your answer” is not the same as “think through this step by step before answering.” The former invites a post-hoc explanation of a conclusion already reached. The latter demands that the reasoning precede the conclusion. The distinction is mechanically significant.

Embedding CoT in an otherwise poor prompt. Adding “think step by step” to a vague, contextless prompt produces vague, contextless reasoning. CoT amplifies the quality of whatever prompt you’ve written — a well-structured prompt with CoT gives you auditable, accurate reasoning; a weak prompt with CoT gives you a longer, more elaborate way of being wrong.

This connects directly to what separates users who get reliable results from AI from those who don’t. It’s rarely the technique itself that’s missing — it’s the baseline prompt structure underneath it. As I covered in Stop Treating AI Like Google, the model needs precise constraints to operate within before any advanced technique produces consistently useful output.

A Note on Model Capability Thresholds

Chain-of-thought prompting does not dramatically improve weaker or smaller models. The capability needs to be present for CoT to surface it.

The research is consistent on this point: CoT shows significant benefits on models above a certain scale threshold. Below that threshold, asking the model to reason step by step may actually produce confident-looking but incorrect intermediate steps, leading to a wrong final answer with the appearance of rigor. On smaller, faster, cheaper models used for high-volume low-complexity tasks, CoT is often wasted or counterproductive.

For most applications running on flagship models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro and above), CoT is a reliable and significant performance booster on complex tasks. On smaller distilled models optimized for speed and cost, test before assuming it helps.

If you take one practical change from this: add a reasoning instruction to your next complex prompt. Not “explain your answer” after the fact — but “think through this step by step” before the model produces the conclusion. Run it with and without the instruction on the same problem. The accuracy difference on anything involving more than one logical step is typically substantial and immediately visible.

Related reading: