A February 2025 video outline of how ChatGPT-style models are built still matches the primary papers: web-scale pretraining, curated conversations, then reinforcement learning. What the papers add is a boundary. Reward-based reasoning gains are best documented where an answer can be checked, and OpenAI says hallucination persists partly because scoring rewards guessing. Buyers should plan around both limits.
Table of Contents
The standard recipe for large language models has three stages, and the public record now puts numbers on each one. Web text comes first: Hugging Face’s open FineWeb dataset holds 15 trillion tokens drawn from 96 Common Crawl snapshots. Curated conversations come second, and OpenAI’s 2022 InstructGPT work found that a 1.3-billion-parameter model fine-tuned this way was preferred in human evaluations to the 175-billion-parameter GPT-3. Reinforcement learning comes third. That last stage is proven where answers can be checked, such as maths and code, and the open questions now sit at its edge.
A video of more than three hours by Andrej Karpathy, uploaded on 5 February 2025, walks through the same stack, from pretraining data to DeepSeek-R1 and AlphaGo. Its description presents him as a founding member of OpenAI and now a founder at Eureka Labs, and it links a Discord channel for that company; it carries no sponsor note. The video is free for educational use. This article checks the stack against papers and reporting available in November 2026.
Pretraining data shrinks from 36 trillion tokens to 15 trillion
The video’s chapter list opens with internet pretraining data and links FineWeb, and the FineWeb paper documents what that step involves. The authors start from 96 Common Crawl snapshots. After basic filtering they had roughly 36 trillion tokens, deduplicating each snapshot separately cut that to 20 trillion, and further filters brought the release to 15 trillion. They say the datasets behind Llama 3 and Mixtral are not public and that curation recipes are often treated as trade secrets, which is the gap FineWeb was built to fill.
The most useful finding is a negative one. Deduplicating across all snapshots at once did not help: it left 4 trillion tokens with only a modest gain, and in the oldest crawl the 10 percent that survived was of worse quality than the 90 percent removed. A second result is that FineWeb-Edu, a 1.3-trillion-token subset picked by a classifier, lifted MMLU from 33 to 37 percent and ARC from 46 to 57 percent in test models of 1.71 billion parameters. The authors add that most of their experiments were small and that they judged models only on academic benchmarks.
Costs run from $20 at GPT-2 scale to 30.84 million GPU hours
The video’s GPT-2 chapter points to llm.c, code for training the model in C and CUDA, and its author posted the numbers himself. In May 2024 he reported reproducing the smallest GPT-2, at 124 million parameters, in about 90 minutes on one rented node of eight A100 GPUs at roughly $14 an hour, about $20 in total. The run used 10 billion FineWeb tokens. It reached 29.9 on the HellaSwag benchmark against 29.4 for OpenAI’s released checkpoint, although he notes that the benchmark has no maths or code, which slightly favours a model trained on web crawls, and that GPT-2’s original WebText data was never released. His 350-million-parameter run took 14 hours and about $200. One commenter who repeated the small run on three other setups reported between $27 and $163.
Scale changes the bill. Meta’s paper describes a dense Transformer with 405 billion parameters and a 128,000-token context window, and claims quality comparable to GPT-4, which is Meta’s own assessment. The Register reports that Meta trained it on more than 15 trillion tokens with 16,000 Nvidia H100 GPUs, and that Meta puts the effort at the equivalent of 30.84 million GPU hours.
A fine-tuned 1.3B model won human preference tests against GPT-3 at 175B
The video’s post-training chapters and its link to the InstructGPT paper match the method OpenAI published in 2022. Labelers wrote demonstrations of good answers, GPT-3 was fine-tuned on them, and a second step collected rankings of model outputs for reinforcement learning from human feedback. In human evaluations on OpenAI’s prompt distribution, outputs from the 1.3-billion-parameter InstructGPT were preferred to those of GPT-3, a model with 100 times more parameters. The paper also reports gains in truthfulness, less toxic output and minimal losses on public benchmarks, and it concedes that the model still makes simple mistakes.
Two cautions follow. The preference data come from labelers and from prompts submitted through an API, so the result measures taste on that distribution, not truth. And human feedback is already a reward signal for reinforcement learning, only one that costs people’s time to produce, which is the cost the next stage tries to avoid.
Reinforcement learning needs an answer key
The video’s late chapters cover reinforcement learning, DeepSeek-R1 and AlphaGo. DeepSeek’s paper claims that reasoning can be developed through pure reinforcement learning, without human-labelled reasoning examples, and that the model then beats supervised counterparts on verifiable tasks such as mathematics, coding contests and STEM questions. It appeared in Nature in September 2025, after the video, and the arXiv version was revised in January 2026. The Register’s report on the Nature paper lists limits: the model was trained only on clear-cut right or wrong answers, it has yet to show skill at subjective or long-form responses, it sometimes switches between English and Chinese, and its reasoning can run past 10,000 words.
AlphaGo follows the same pattern. Its 2016 Nature paper describes training that began with human expert games and moved on to self-play reinforcement learning, and it reports a 5 to 0 win over the European champion. In Go and in maths, a rulebook or an answer key settles who is right. That is an inference of this article, not a claim either paper makes, and it is a plausible reason these domains came first.
Rubric rewards are an early push past checkable answers
Researchers are trying to move reinforcement learning outside the checkable zone. A paper at ICLR 2026, Rubrics as Rewards, states the problem plainly: training on verifiable rewards has proven effective where correctness is clear, as in maths and code, while real-world tasks depend on nuanced judgments rather than a binary right or wrong. Its answer is to score outputs against written checklists, called rubrics. The best variant improved by up to 31 percent on HealthBench and 7 percent on GPQA-Diamond, relative to popular judge-based baselines that use direct rating-scale rewards.
Read that comparison carefully. Both figures are relative gains over other methods that also rely on a model acting as judge, not over verified ground truth, and the authors say the use of rubrics as training rewards “remains underexplored”. The idea is promising, but the evidence is young.
Scoreboards that reward guessing keep hallucination alive
The video covers hallucinations in a chapter that also takes in tool use and working memory. OpenAI’s research from September 2025, which came later, offers an account of the problem. OpenAI says language models hallucinate partly because standard evaluations grade accuracy alone, so a lucky guess beats an honest admission of ignorance. Its example is a birthday: a guess has a 1-in-365 chance of scoring, and abstaining scores zero. In OpenAI’s SimpleQA table, gpt-5-thinking-mini abstained on 52 percent of questions and erred on 26 percent, while the older o4-mini abstained on 1 percent and erred on 75 percent, even though o4-mini scored slightly higher on raw accuracy, 24 percent against 22.
On where the errors start, the page points to arbitrary low-frequency facts, such as a person’s birthday, which cannot be predicted from patterns in text. Pretraining therefore leaves errors there, and later stages were meant to remove them and only partly do. OpenAI proposes penalising confident errors more heavily than uncertainty. These figures come from OpenAI’s own GPT-5 system card, and the page offers no independent replication.
Limits that should shape how teams buy and test models
Apple researchers reported in 2025, as InfoQ summarises, that reasoning models tested on controllable puzzles such as Tower of Hanoi went through three regimes. Reasoning and standard models did about equally well on simple problems, reasoning models led at medium complexity, and both collapsed to zero at high complexity. Apple acknowledges that it relied on black-box API access and that puzzles may not generalise to other reasoning domains. Critics split: Gary Marcus read the paper as proof that these models do not replace well-specified conventional algorithms, while Simon Willison argued that useful applications matter more than the road-to-AGI question.
For a team buying or building on these models, the sources point to a practical rule. Give reasoning models work whose output can be checked by code, tests or a reference answer, and keep a person or a conventional program in the loop elsewhere. Ask vendors for abstention and error rates, not accuracy alone. The video’s chapters on tokenisation, spelling and “jagged intelligence” fall outside what these sources test. Three developments would change this reading: rubric rewards beating verified baselines, major benchmarks that penalise confident errors, and an independent check of OpenAI’s SimpleQA figures.
Author:
Jan Bielik
CEO & Founder of Webiano Digital & Marketing Agency
This article is an original analysis supported by the sources cited below
Deep Dive into LLMs like ChatGPT
The video itself: its description, chapter list and linked resources set out the training stack, and its licence terms, upload date and Eureka Labs links supplied the relationship details.
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Documented the 96-snapshot, 15-trillion-token pretraining dataset, the 36 and 20 trillion intermediate sizes, the failed global deduplication, the FineWeb-Edu gains and the authors’ stated limits.
The Llama 3 Herd of Models
Meta’s paper establishing the 405-billion-parameter dense Transformer, the 128,000-token context window and Meta’s own claim of GPT-4-comparable quality.
Reproducing GPT-2 (124M) in llm.c in 90 minutes for $20
Primary source for the roughly $20 and 90-minute GPT-2 reproduction, the HellaSwag comparison and its caveats, the $200 350M run, and one commenter’s costs on other hardware.
Training language models to follow instructions with human feedback
The InstructGPT abstract: labeler demonstrations, rankings and reinforcement learning from human feedback, with the 1.3B versus 175B preference result and the admitted simple mistakes.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Established the claim that reasoning can be developed through pure reinforcement learning, its focus on verifiable tasks, the Nature reference and the January 2026 revision.
Mastering the game of Go with deep neural networks and tree search
The AlphaGo abstract: supervised learning from human expert games followed by self-play reinforcement learning, and the 5 to 0 win over the European champion.
Why language models hallucinate
OpenAI’s account of hallucination as a product of guessing-friendly scoring and unpredictable rare facts, including the SimpleQA abstention and error figures for two models.
Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
The ICLR 2026 abstract stating where verifiable-reward training works, how rubric rewards try to extend it, and the 31 and 7 percent relative gains over judge-based baselines.
Meta claims ‘world’s largest’ open AI model with Llama 3.1 405B debut
Independent report giving the 15-trillion-token, 16,000-GPU scale and Meta’s stated 30.84 million GPU hours for training the 405B model.
China’s DeepSeek applying trial-and-error learning to its AI ‘reasoning’
Independent report on the Nature paper, including its limits: right-or-wrong training answers only, no proven skill on subjective tasks, language switching and very long reasoning.
Apple’s Illusion of Thinking Paper Explores Limits of Large Reasoning Models
Independent summary of Apple’s puzzle study: three complexity regimes, collapse to zero at high complexity, Apple’s stated limits, and the Marcus and Willison reactions.
| Citing this article? Brief excerpts are welcome. Please credit Webiano.digital, name the author where stated, and include a link to https://webiano.digital and to this original article. Full or substantial republication requires prior written permission. Read our Copyright and Content Use Policy. |
This article was prepared with the assistance of artificial intelligence tools. The content underwent expert human review, and Webiano Digital & Marketing Agency assumes editorial responsibility for its final version and publication.
