A popular two-hour walkthrough of ChatGPT, Claude and Gemini from February 2025 has aged unevenly. Claude gained web search, deep research reached free accounts and GPT-5 reopened an argument about picking models. The advice to check chatbot answers against primary sources lasted, and OpenAI’s own research now explains why. The coding segment is the part most overtaken by later evidence.
Table of Contents
Much of what people were told about AI chatbots in early 2025 has a shelf life of months: which app can search the web, who gets deep research, whether you should pick the model yourself. The advice to check answers against primary sources has not expired, and the best evidence for it now comes from OpenAI’s own researchers.
The clearest record of that early advice is “How I use LLMs”, a video of 2 hours 11 minutes that AI researcher Andrej Karpathy published on 27 February 2025. He made it himself, shows his own subscriptions and workflow, and links the products it features in the description without disclosing a sponsor. Reading it against what happened next separates the habits worth keeping from the settings worth rechecking.
A tour built on the context window
The video runs through 22 chapters, from how a chat becomes a stream of tokens to custom GPTs, with Claude, Gemini, Grok, Perplexity, Cursor and NotebookLM beside ChatGPT. Its frame is simple. A chatbot is, in his telling, a lossy compressed copy of the internet with a personality added by human labelers, and the context window is its working memory. Two habits follow: start a new chat when the topic changes, because old tokens distract the model, and give it current facts through search or uploaded documents instead of trusting its recall.
The wording here comes from an unofficial transcript archive of the video’s captions. It is auto-generated text that ends during the memory segment, so the closing minutes are not covered.
Feature advice that aged within months
The advice most tied to a date is the buying and tool advice. When Karpathy recorded the video, Claude had no web search, which he shows by asking about a TV season’s release date, and he guessed Anthropic was working on it. Engadget reported on 20 March 2025 that paid US users could now search the web in Claude, three weeks after the video went up.
Deep research followed the same path. The video presents it as a $200-a-month Pro feature that he had used on 10 to 20 queries. By 25 April 2025, according to Engadget, Plus, Team, Edu and Enterprise subscribers had it too, and free accounts were getting five runs a month of a lightweight version that OpenAI calls nearly as capable. Plus accounts got 10 full and 15 lightweight queries a month.
Model choice reversed more messily. Karpathy tells viewers to check which model they are talking to, since free and paid plans differ. OpenAI launched GPT-5 on 7 August 2025 with a router meant to remove that decision. The next day Sam Altman said the autoswitcher had been out of commission for part of launch day and GPT-5 “seemed way dumber”, per TechCrunch. Five days after launch the same outlet reported on 12 August that Auto, Fast and Thinking settings had been added and older models returned for paid users.
The checking habit and why it lasted
The most durable instruction in the video is about verification, and Karpathy demonstrates it with the tool’s own mistakes. In the data-analysis demo, ChatGPT silently put 0.1 into its code for an unknown 2015 valuation of OpenAI, then reported a 2030 extrapolation of 1.7 trillion when the printed variable pointed to about 20 trillion. He catches both by reading the code and printing the variable, and says he would hesitate to recommend such tools to people who cannot. He also checked a medicine box against the model’s ingredient list, and he describes deep research output as a reading list: “Treat this as your first draft.”
OpenAI’s September 2025 paper explains why that advice has not aged. Models hallucinate, the authors argue, because standard scoring rewards guessing over admitting uncertainty, and accuracy will never reach 100% because some questions cannot be answered. OpenAI says GPT-5 hallucinates less, especially when reasoning, but still does. In its own example on a factual-question test, a newer reasoning model declined to answer 52% of questions where an older one declined 1%, and its error rate was 26% against 75%, while accuracy was 22% against 24%. That is a company describing its own models.
Usage data that matches the first hour
The video spends its first hour on questions, search, deep research and reading documents together with a model, and about eight minutes on Cursor. A working paper from NBER, based on a sample of consumer ChatGPT conversations, finds the same weighting in what people do. Non-work messages grew from 53% to more than 70% of usage, and practical guidance, information seeking and writing make up nearly 80% of conversations, while programming holds a relatively small share.
Two cautions apply. At least two of the authors say they work for OpenAI, and the data covers one product, so it describes ChatGPT users and not chatbot users in general. It still supports a narrow point: the advice most people need is about asking and checking, and the coding workflow in the video is the specialist branch.
Coding claims and the 19% slowdown
The coding segment shows Cursor’s Composer agent, running Claude 3.7 Sonnet, building a tic-tac-toe app and adding confetti and a victory sound from a few sentences. Karpathy calls the approach vibe coding, admits he is not tracking every change, and keeps the option of falling back to hand-written code. In METR’s randomized trial, published on 10 July 2025, developers worked mostly with Cursor Pro and Claude 3.5 or 3.7 Sonnet, a comparable stack. Across 246 real issues from 16 experienced developers’ own repositories, tasks took 19% longer with AI allowed. The developers had forecast a 24% speedup and afterwards still believed they had gained 20%.
METR now labels that result out of date. Its February 2026 follow-up, with 57 developers and more than 800 tasks, estimates that the 10 returning developers needed 18% less time with AI and the 47 new ones 4% less, but both intervals include zero. METR calls the new data unreliable: more developers refused to work without AI, and 30% to 50% said they held back tasks they did not want to do unaided. METR still thinks speedups are now likely, and says its evidence for their size is very weak.
A reversal from the video’s own author
In October 2025 Karpathy said that Claude Code and Codex “didn’t work well enough at all and net unhelpful” for his nanochat project. Asked about it on Dwarkesh Patel’s podcast, he explained, as quoted by Simon Willison, that agents do well on boilerplate and on code that appears often online, and badly on a precisely arranged, unusual repository, where the models kept misreading his code because they expected the common patterns. He also argued that agents with employee-like abilities need about a decade of work on gaps such as continual learning.
That is a scoped claim. The video’s workflow suits ordinary code, and its author did not use it for the code he cared most about. Willison adds his own view that coding agents which run tools in a loop are already good, so the two accounts differ in emphasis more than in fact.
A team lead’s keep-and-recheck list
For someone writing guidance for a team, or deciding what to pay for, the evidence sorts the video into two piles. Keep the habits that rest on how the models work: a fresh chat for each topic, a request for search when facts are recent, reading the code or the source behind any number that will be used, and treating long research reports as a reading list. Recheck every quarter which plan includes which tool, whether the model is chosen for you, and how deep research is limited. The video’s prices of $20 and $200 are the least reliable part, so use the live pricing page.
On coding, the 2025 slowdown figure should not be quoted as current, and neither should a speedup. The developers who believed they were 20% faster had been 19% slower, so feelings are a poor measure; a team can time its own tasks with and without AI. This reading rests on an unofficial transcript, a 16-developer trial and one company’s usage data, and none of them compares today’s products head to head. What would change this reading is the redesigned controlled trial METR says it is working on, provided developers accept going without AI on some tasks.
Author:
Jan Bielik
CEO & Founder of Webiano Digital & Marketing Agency
This article is an original analysis supported by the sources cited below
How I use LLMs
The video itself: published on 27 February 2025 by Andrej Karpathy, with a 22-chapter list, links to the tools shown, an educational-use licence and no sponsor named in the description.
How I use LLMs — Andrej Karpathy — Rosetta
An unofficial transcript archive of the video’s captions, used for what is said on screen: pricing tiers, the search tests, the data-analysis errors and the Cursor demo. It ends during the memory segment.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
METR’s July 2025 randomized trial: 16 developers, 246 issues, tasks 19% slower with AI allowed, mainly with Cursor Pro and Claude 3.5 or 3.7 Sonnet, plus the gap between forecasts and results.
We are Changing our Developer Productivity Experiment Design
METR’s February 2026 update: a larger follow-up showing apparent speedups with intervals that include zero, and METR’s explanation of why selection effects make the new data unreliable.
Why language models hallucinate
OpenAI’s September 2025 research note on why guessing is rewarded and why accuracy cannot reach 100%, with an abstention and error comparison of two models. A company’s account of its own models.
How People Use ChatGPT
NBER working paper 34255 (September 2025): non-work use above 70%, three topics making up nearly 80% of conversations, and disclosures that some authors are employed by OpenAI.
Andrej Karpathy — AGI is still a decade away
Simon Willison’s post quoting Karpathy on why coding agents did not help his nanochat project, where they do help, and why he expects agents to need a decade.
Anthropic’s Claude chatbot can now search the web too
Engadget’s report of 20 March 2025 that Claude web search began for paid US users, with the model’s earlier October 2024 knowledge cutoff.
OpenAI’s Deep Research tool is coming to free accounts
Engadget’s report of 25 April 2025 on free-tier deep research, the lightweight version and the monthly limits for Free, Plus and Pro accounts.
Sam Altman addresses ‘bumpy’ GPT-5 rollout, bringing 4o back, and the ‘chart crime’
TechCrunch on the GPT-5 launch: the router failing on day one, Altman’s explanation, and his promise to look at keeping GPT-4o for Plus subscribers.
| Citing this article? Brief excerpts are welcome. Please credit Webiano.digital, name the author where stated, and include a link to https://webiano.digital and to this original article. Full or substantial republication requires prior written permission. Read our Copyright and Content Use Policy. |
This article was prepared with the assistance of artificial intelligence tools. The content underwent expert human review, and Webiano Digital & Marketing Agency assumes editorial responsibility for its final version and publication.
