Skeleton-of-thought

Also known as: SoT

On this page 7

Skeleton-of-thought (SoT) is a prompting technique that makes a language model first produce a short outline of the answer, called the skeleton, and then expand every outline point in parallel, so the full response arrives faster than with ordinary token-by-token generation. Prompting methods usually aim at accuracy, but SoT is unusual because its primary goal is lower latency.

The method was introduced in 2023 research on efficient inference and mirrors how people often write: decide the main points, then develop each one. It belongs to the family of structured reasoning prompts alongside chain-of-thought, tree of thoughts and plan-and-solve prompting, but it changes how decoding is scheduled rather than how the model reasons step by step.

How skeleton-of-thought works in two stages

SoT splits answer generation into a skeleton stage and a point-expanding stage. Both stages use prompt templates, and no change to the model weights is required.

  1. Skeleton stage: the model receives the question and an instruction to return only a numbered list of brief points, typically 3 to 10 points of a few words each.
  2. Parsing: application code extracts each numbered point from the skeleton with a simple pattern match.
  3. Point-expanding stage: for every point, the model receives the question, the full skeleton and the instruction to expand only that point in one to two sentences.
  4. Assembly: the expansions are concatenated in skeleton order to form the final answer.

The speed gain comes from the third step. With a hosted API, the expansions are sent as parallel requests; with a locally served open model, they are processed as one batch, which uses GPU capacity that a single sequential decode leaves idle.

StageInputOutputRuns in parallel
SkeletonQuestion and outline instructionNumbered list of brief pointsNo
ExpansionQuestion, skeleton and one pointOne to two sentences per pointYes
AssemblyAll expanded pointsFinal ordered answerNot applicable
Stages of skeleton-of-thought

Why parallel expansion reduces latency

Language models generate text one token at a time, and each token depends on the previous ones, so a long answer takes time roughly proportional to its length. SoT replaces one long sequential decode with one short decode plus several shorter decodes running together.

In the original experiments across about a dozen models, SoT reported end-to-end speed-ups of up to roughly 2.4 times, with typical gains between about 1.1 and 2 times depending on the model, the question type and the serving setup. These figures are not guaranteed in production, because rate limits, batching policies and network overhead can erase much of the benefit.

The trade-off is that SoT usually consumes more total tokens, since the question and skeleton are repeated in every expansion request. It therefore saves wall-clock time, not compute or cost.

Where skeleton-of-thought helps and where it hurts

SoT works best when an answer naturally consists of independent points that can be written without knowing the details of the others. It performs poorly when each step depends on the result of the previous one.

Question typeSoT suitabilityReason
Knowledge and explanationGoodPoints such as causes or features are separable
Advice and recommendationsGoodTips can be expanded independently
Comparisons and pros and consGoodEach criterion forms its own point
Math and logic problemsPoorLater steps depend on earlier results
Code generationPoorFunctions and variables must stay consistent
Creative and narrative writingPoorCoherent flow requires sequential context
Question types and SoT suitability

Evaluations of answer quality also showed a mixed picture. SoT often made answers more diverse and relevant on list-like questions, but it could reduce coherence and immersion because each point is written without seeing the others.

SoT is often confused with other structured prompting methods. The key distinction is that SoT structures output for speed, whereas most neighbors structure reasoning for accuracy.

TechniqueMain goalStructureEffect on latency
Skeleton-of-thoughtFaster responsesOutline, then parallel expansionLower
Chain-of-thoughtBetter reasoning accuracySequential intermediate stepsHigher
Tree of thoughtsSearch over reasoning pathsBranching and evaluationMuch higher
Plan-and-solve promptingFewer missing stepsPlan, then sequential executionHigher
Least-to-most promptingSolve hard problemsOrdered subproblemsHigher
SoT and neighboring techniques

Plan-and-solve prompting resembles SoT because both start with a plan, but plan-and-solve executes the plan sequentially in one generation, so it keeps dependencies intact and gives up the speed advantage.

Implementing SoT with a router

Because SoT fails on some question types, practical systems add a router that decides per request whether to use SoT or normal decoding. The research variant called SoT with router (SoT-R) used either a prompt asking a model to classify the question or a small fine-tuned classifier.

  • Route math, code and long creative tasks to normal decoding by default.
  • Cap the number of skeleton points, because very long outlines raise token usage without improving speed.
  • Include the whole skeleton in every expansion prompt, so each point avoids repeating its siblings.
  • Measure wall-clock latency and quality together, since a faster but disjointed answer may be worse for users.

A frequent mistake is assuming that any hosted API will run expansions truly in parallel. Concurrency limits and per-request overhead can make SoT slower than one direct call, so the benefit must be confirmed with real traffic.

In simple terms

Skeleton-of-thought is like writing the bullet points first and then asking several helpers to fill them in at once. A single writer would finish each paragraph before starting the next. With helpers working side by side, the whole text is ready sooner.

This works well when the points do not depend on each other, such as a list of travel tips. It works badly for a math problem, where each step needs the answer from the step before. The main benefit is speed, not a smarter answer.

A practical example

An online electronics retailer runs a support assistant on a locally hosted open model. Questions such as how to extend laptop battery life produce answers of about 400 tokens, and customers wait around 12 seconds for each one, which leads many to abandon the chat.

The engineering team adds SoT behind a router. The model first returns a skeleton such as “1. Lower screen brightness 2. Limit background apps 3. Use power saving mode 4. Avoid heat 5. Keep charge moderate”, and the five points are then expanded as one batch on the GPU. Troubleshooting questions that require ordered diagnostic steps bypass SoT and use normal decoding.

After rollout, the median response time for tip-style questions drops to about 6 seconds, while total token usage rises by roughly a third. Reviewers find answer quality comparable, so the team keeps SoT for advice questions only.