Broken link checker

On this page 7

A broken link checker is a testing tool that crawls a website, requests every hyperlink and resource reference, and reports the ones that return errors, time out or lead somewhere unintended. In web testing, it is the standard way to detect link rot, the gradual decay of internal and external links that accumulates as pages move, domains expire and content is deleted.

The tool matters for users, search engines and accessibility alike. A dead link interrupts a task, wastes crawl budget, weakens internal linking signals and erodes trust in the site. Running a checker regularly, ideally as part of a continuous integration pipeline, turns a slow, invisible quality problem into a measurable list of fixes.

A broken link checker starts from one or more seed URLs, downloads each page, parses its HTML and extracts every reference it finds: anchors, images, scripts, stylesheets, iframes and sometimes srcset or CSS url() values. Internal links are queued for further crawling, while external links are usually validated without being crawled further.

Validation relies on the HTTP status code of each response. Many checkers send a lightweight HEAD request first and fall back to GET when a server rejects HEAD, because some servers answer HEAD incorrectly. The table below shows how typical responses are classified.

ResponseMeaningTypical classification
200 OKResource delivered successfullyValid link
301 or 308 redirectPermanent move to a new URLWarning; update the link to the final target
302 or 307 redirectTemporary moveUsually accepted, flagged when chained
404 Not FoundResource does not existBroken link
410 GoneResource removed intentionallyBroken link; remove or replace the reference
5xx server errorServer failed to respond correctlyBroken or retry later
Timeout or DNS failureNo usable response at allBroken, often an expired domain
How status codes are usually interpreted

Good checkers also detect problems that a status code hides. A soft 404 returns 200 OK while displaying an error page, and an anchor fragment such as #pricing may point to an ID that no longer exists on the target page. Detecting these requires content heuristics or parsing the target document.

Broken link checkers differ mainly in where they run and how deeply they render pages. A static crawler is fast but cannot see links that JavaScript inserts after load, while a checker built on a headless browser sees the rendered DOM at a higher cost in time and memory.

TypeHow it worksBest suited for
Command-line crawlerFetches and parses raw HTML from a terminalStatic sites and build pipelines
Desktop SEO crawlerGraphical crawl with reports and filtersPeriodic audits of large sites
Hosted monitoring serviceScheduled cloud crawls with alertsOngoing monitoring without local setup
CMS pluginScans stored content inside the CMSEditorial sites with frequent updates
Headless browser scriptRenders pages, then collects links from the DOMSingle-page applications and dynamic content
Markdown or docs linterChecks links in source files before publishingDocumentation repositories
Common categories and their best use

Teams often combine two layers: a pre-deployment check on source files or a staging build, and a scheduled production crawl that catches external links breaking long after the content was published.

A broken link checker answers one narrow question: does every reference resolve correctly? Other web testing tools address performance, appearance or behavior, and some include link checks only as a side feature.

Tool or methodPrimary focusLink coverage
Broken link checkerReference integrity across the whole siteComplete and systematic
LighthousePerformance, accessibility and best practices per pageLimited to signals on the audited page
Playwright or Cypress testsScripted user flows and assertionsOnly links exercised by the test scenarios
Visual regression testingPixel differences between buildsNone; a dead link can look identical
Accessibility testingBarriers for people with disabilitiesFlags empty or unclear link text, not dead targets
Scope of link checking versus neighboring tools

Because end-to-end tests cover only chosen paths, they rarely notice a dead link in a footer or an old blog post. The checker complements them by sweeping every page rather than simulating particular journeys.

Useful results depend more on configuration than on the tool chosen. An unconfigured crawl can overload a server, miss authenticated areas or drown real problems in false positives from sites that block automated clients.

SettingTypical valueWhy it matters
Concurrency2 to 10 parallel requestsBalances speed against server load and rate limits
Request timeout10 to 30 secondsSeparates slow servers from dead ones
Retry policy1 to 3 retries with delayReduces false alarms from transient errors
Crawl depth or scopeSame host, optional subpathsPrevents crawling the entire internet
Exclusion patternsLogout URLs, calendars, query loopsAvoids side effects and infinite URL spaces
User agentDescriptive, generic client nameSome servers reject unknown or empty agents
Key settings and their effect

In a CI pipeline, the check usually runs against a staging build and fails the job only on internal broken links, while external failures are reported as warnings rather than blockers. This prevents a third-party outage from stopping an unrelated deployment.

  • Respect robots.txt and crawl-delay directives on sites you do not own.
  • Store results between runs to see which links broke recently.
  • Report the source page and anchor text for every failure so editors can fix it quickly.

No broken link checker is perfectly accurate. False positives arise when servers block bots, demand cookies or return errors to HEAD requests, while false negatives come from soft 404 pages, parked domains that answer 200 OK and links hidden behind logins or scripts.

Some major platforms routinely return 403 or 429 responses to automated clients even when a page is perfectly available to people. A practical rule is to verify suspicious external failures manually before editing content, and to maintain an allowlist of hosts known to reject crawlers.

Fixing the report is also a judgment call. An internal 404 should normally be resolved with a 301 redirect or a corrected link, while a dead external source may need a replacement, an archived copy or removal of the reference. Mass-redirecting every missing URL to the homepage is a common mistake that search engines often treat as a soft 404.

Understanding it simply

Websites are full of links, and over time some of them stop working because a page was moved, renamed or deleted. A broken link checker visits every page for you and clicks every link automatically, then gives you a list of the ones that lead nowhere.

It works like a building inspector testing every door in a large office. Most doors open, some lead to a renovated corridor, and a few open onto a brick wall. The inspector writes down each problem and where it is, so the repair team knows exactly which door to fix.

Applied in practice

A university library maintains a research guide site with about 4,000 pages and more than 60,000 outbound links to journals, databases and government reports. Students complain that many recommended sources lead to error pages, so the web team configures a command-line broken link checker to crawl the site every Sunday night with five parallel requests, a 20-second timeout and two retries.

The first full report lists roughly 3,100 failing links, but manual sampling shows that about a fifth are false positives from publishers blocking automated clients. After adding those hosts to an allowlist and treating permanent redirects as update tasks, the team assigns the remaining fixes to subject librarians by page. Within two months, confirmed broken links fall to a few dozen per weekly run, and the check is added to the build pipeline so that new guides cannot be published with internal dead links.