Graph of thoughts (GoT) is a reasoning framework for large language models that represents intermediate ideas as vertices in a directed graph and the operations between them as edges, so that partial solutions can be generated, combined, refined and scored before a final answer is produced. In prompting, it extends linear and tree-shaped reasoning by allowing several lines of thought to merge into one improved result.
The approach treats the model as one component inside a controller program. The controller decides which thoughts to create, which to aggregate and which to discard, while the model performs the individual reasoning steps that each prompt requests.
How graph of thoughts structures reasoning
Graph of thoughts works by running a predefined graph of operations over a set of thoughts, where each thought is a piece of text such as a partial plan, a sorted sublist or a draft paragraph. The controller sends a prompt for each operation, stores the response as a new vertex and records which earlier vertices it depends on.
The structure is usually described in two layers. The graph of operations is the static plan written by the developer, and the graph reasoning state is the dynamic record of thoughts and scores produced during one run. Separating them makes a pipeline reusable across inputs while keeping every run inspectable.
The table below lists the core operations that a graph of thoughts pipeline combines.
| Operation | What it does | Typical prompt |
|---|---|---|
| Generate | Creates one or more new thoughts from a parent | Propose three ways to split this task |
| Aggregate | Merges several thoughts into one | Combine these partial lists into one result |
| Refine | Improves a thought through a self-loop | Correct the errors in this draft |
| Score | Assigns a quality value to a thought | Rate this solution from 1 to 10 |
| Keep best | Selects the highest-scoring thoughts | Handled in code, not by the model |
Why aggregation is the defining feature
The capability that separates graph of thoughts from earlier methods is aggregation of independent thoughts. A tree can only branch outward, so useful fragments found on different branches stay isolated, whereas a graph lets a vertex have several parents, which mirrors how human problem solving often merges partial insights.
This makes GoT well suited to decomposable problems, in which an input can be split, processed in parts and then reassembled. Sorting long lists, intersecting sets, merging documents and counting keywords across many passages are the canonical demonstrations, because each subproblem is easier and the merge step is well defined.
- Split the input into chunks that fit comfortably in one prompt.
- Solve each chunk independently, often several times.
- Score and keep the best candidate for each chunk.
- Aggregate chunk results, then refine the merged output.
Graph of thoughts compared with related prompting methods
Graph of thoughts belongs to a family of structured reasoning techniques that differ mainly in the shape of the reasoning path. Chain-of-thought follows one sequence, self-consistency samples several independent chains and votes, and tree of thoughts branches and backtracks with search.
The comparison below shows how the topology of each method affects what it can do and what it costs.
| Method | Topology | Merges thoughts | Relative cost |
|---|---|---|---|
| Chain-of-thought | Single path | No | Lowest |
| Self-consistency | Parallel independent paths | Only by voting | Moderate |
| Tree of thoughts | Tree with search | No | High |
| Graph of thoughts | Arbitrary directed graph | Yes, by aggregation | Variable, often high |
A useful metric for comparing these methods is the volume of a thought, meaning the number of earlier thoughts that can influence it. In a chain the volume grows linearly with depth, while in a graph with aggregation a final answer can draw on every preceding thought at once.
Designing a graph of thoughts pipeline
Building a GoT pipeline is primarily a software design task, not a single prompt. The developer writes prompt templates for each operation, a parser that turns model output into structured thoughts, a scoring function that may be a model call or deterministic code, and the controller that schedules everything.
The following decision table helps choose sensible design options for common situations.
| Situation | Recommended choice | Reason |
|---|---|---|
| Correctness can be checked in code | Deterministic scoring | Cheaper and more reliable than model ratings |
| Subtasks are independent | Run generations in parallel | Reduces wall-clock latency |
| Merged output often has errors | Add a refine loop | Aggregation can introduce inconsistencies |
| Budget is tight | Limit branching factor | Cost grows with every extra thought |
| Input fits in one prompt | Use simpler prompting | Graph overhead brings little benefit |
Limitations and common mistakes
The main trade-off of graph of thoughts is higher cost and engineering effort in exchange for better quality on suitable problems. A single task can require dozens of model calls, and the gains reported in research are strongest on synthetic, clearly decomposable benchmarks rather than open-ended writing or conversation.
Several mistakes recur in practice. Teams apply GoT to problems that do not decompose cleanly, rely on the model to score its own output without calibration, or design aggregation prompts that silently drop or duplicate content. Modern reasoning models that plan internally also narrow the advantage, so a baseline with plain chain-of-thought should always be measured first.
Error propagation is another risk. Because later vertices depend on earlier ones, a flawed thought that survives scoring contaminates every aggregation built on it, so pipelines benefit from verification at each merge point rather than only at the end.
A non-technical view
Imagine a team writing a long report. Several people draft different sections. Someone checks each draft. Then an editor combines the best parts into one document and polishes it.
Graph of thoughts asks an AI model to work the same way. It does not answer in one go. It produces several partial ideas, rates them, joins the good ones and fixes the result. The key idea is merging: good pieces from different attempts end up in one better answer.
A concrete example
A research analyst must merge 40 short customer interview summaries into one list of product complaints without duplicates. A single prompt with all 40 summaries produces a list that misses issues and repeats others, because the model loses track across the long input.
The analyst builds a small graph of thoughts pipeline. It splits the summaries into eight groups of five, asks the model to extract complaints from each group three times, keeps the version with the most items that a simple script confirms appear in the source text, then aggregates the eight lists pairwise in three rounds and runs a final refine step to remove near-duplicates.
The process takes about 50 model calls instead of one, but the final list covers every complaint verified by manual review, and each item can be traced back through the graph to its source group.