A broken link checker is a testing tool that crawls a website, requests every hyperlink and resource reference, and reports the ones that return errors, time out or lead somewhere unintended. In web testing, it is the standard way to detect link rot, the gradual decay of internal and external links that accumulates as pages move, domains expire and content is deleted.
The tool matters for users, search engines and accessibility alike. A dead link interrupts a task, wastes crawl budget, weakens internal linking signals and erodes trust in the site. Running a checker regularly, ideally as part of a continuous integration pipeline, turns a slow, invisible quality problem into a measurable list of fixes.
How a broken link checker crawls and validates links
A broken link checker starts from one or more seed URLs, downloads each page, parses its HTML and extracts every reference it finds: anchors, images, scripts, stylesheets, iframes and sometimes srcset or CSS url() values. Internal links are queued for further crawling, while external links are usually validated without being crawled further.
Validation relies on the HTTP status code of each response. Many checkers send a lightweight HEAD request first and fall back to GET when a server rejects HEAD, because some servers answer HEAD incorrectly. The table below shows how typical responses are classified.
| Response | Meaning | Typical classification |
|---|---|---|
| 200 OK | Resource delivered successfully | Valid link |
| 301 or 308 redirect | Permanent move to a new URL | Warning; update the link to the final target |
| 302 or 307 redirect | Temporary move | Usually accepted, flagged when chained |
| 404 Not Found | Resource does not exist | Broken link |
| 410 Gone | Resource removed intentionally | Broken link; remove or replace the reference |
| 5xx server error | Server failed to respond correctly | Broken or retry later |
| Timeout or DNS failure | No usable response at all | Broken, often an expired domain |
Good checkers also detect problems that a status code hides. A soft 404 returns 200 OK while displaying an error page, and an anchor fragment such as #pricing may point to an ID that no longer exists on the target page. Detecting these requires content heuristics or parsing the target document.
Types of broken link checkers
Broken link checkers differ mainly in where they run and how deeply they render pages. A static crawler is fast but cannot see links that JavaScript inserts after load, while a checker built on a headless browser sees the rendered DOM at a higher cost in time and memory.
| Type | How it works | Best suited for |
|---|---|---|
| Command-line crawler | Fetches and parses raw HTML from a terminal | Static sites and build pipelines |
| Desktop SEO crawler | Graphical crawl with reports and filters | Periodic audits of large sites |
| Hosted monitoring service | Scheduled cloud crawls with alerts | Ongoing monitoring without local setup |
| CMS plugin | Scans stored content inside the CMS | Editorial sites with frequent updates |
| Headless browser script | Renders pages, then collects links from the DOM | Single-page applications and dynamic content |
| Markdown or docs linter | Checks links in source files before publishing | Documentation repositories |
Teams often combine two layers: a pre-deployment check on source files or a staging build, and a scheduled production crawl that catches external links breaking long after the content was published.
Broken link checking compared with related testing tools
A broken link checker answers one narrow question: does every reference resolve correctly? Other web testing tools address performance, appearance or behavior, and some include link checks only as a side feature.
| Tool or method | Primary focus | Link coverage |
|---|---|---|
| Broken link checker | Reference integrity across the whole site | Complete and systematic |
| Lighthouse | Performance, accessibility and best practices per page | Limited to signals on the audited page |
| Playwright or Cypress tests | Scripted user flows and assertions | Only links exercised by the test scenarios |
| Visual regression testing | Pixel differences between builds | None; a dead link can look identical |
| Accessibility testing | Barriers for people with disabilities | Flags empty or unclear link text, not dead targets |
Because end-to-end tests cover only chosen paths, they rarely notice a dead link in a footer or an old blog post. The checker complements them by sweeping every page rather than simulating particular journeys.
Configuring a broken link checker for reliable results
Useful results depend more on configuration than on the tool chosen. An unconfigured crawl can overload a server, miss authenticated areas or drown real problems in false positives from sites that block automated clients.
| Setting | Typical value | Why it matters |
|---|---|---|
| Concurrency | 2 to 10 parallel requests | Balances speed against server load and rate limits |
| Request timeout | 10 to 30 seconds | Separates slow servers from dead ones |
| Retry policy | 1 to 3 retries with delay | Reduces false alarms from transient errors |
| Crawl depth or scope | Same host, optional subpaths | Prevents crawling the entire internet |
| Exclusion patterns | Logout URLs, calendars, query loops | Avoids side effects and infinite URL spaces |
| User agent | Descriptive, generic client name | Some servers reject unknown or empty agents |
In a CI pipeline, the check usually runs against a staging build and fails the job only on internal broken links, while external failures are reported as warnings rather than blockers. This prevents a third-party outage from stopping an unrelated deployment.
- Respect robots.txt and crawl-delay directives on sites you do not own.
- Store results between runs to see which links broke recently.
- Report the source page and anchor text for every failure so editors can fix it quickly.
Limitations and false positives in link checking
No broken link checker is perfectly accurate. False positives arise when servers block bots, demand cookies or return errors to HEAD requests, while false negatives come from soft 404 pages, parked domains that answer 200 OK and links hidden behind logins or scripts.
Some major platforms routinely return 403 or 429 responses to automated clients even when a page is perfectly available to people. A practical rule is to verify suspicious external failures manually before editing content, and to maintain an allowlist of hosts known to reject crawlers.
Fixing the report is also a judgment call. An internal 404 should normally be resolved with a 301 redirect or a corrected link, while a dead external source may need a replacement, an archived copy or removal of the reference. Mass-redirecting every missing URL to the homepage is a common mistake that search engines often treat as a soft 404.
Understanding it simply
Websites are full of links, and over time some of them stop working because a page was moved, renamed or deleted. A broken link checker visits every page for you and clicks every link automatically, then gives you a list of the ones that lead nowhere.
It works like a building inspector testing every door in a large office. Most doors open, some lead to a renovated corridor, and a few open onto a brick wall. The inspector writes down each problem and where it is, so the repair team knows exactly which door to fix.
Applied in practice
A university library maintains a research guide site with about 4,000 pages and more than 60,000 outbound links to journals, databases and government reports. Students complain that many recommended sources lead to error pages, so the web team configures a command-line broken link checker to crawl the site every Sunday night with five parallel requests, a 20-second timeout and two retries.
The first full report lists roughly 3,100 failing links, but manual sampling shows that about a fifth are false positives from publishers blocking automated clients. After adding those hosts to an allowlist and treating permanent redirects as update tasks, the team assigns the remaining fixes to subject librarians by page. Within two months, confirmed broken links fall to a few dozen per weekly run, and the check is added to the build pipeline so that new guides cannot be published with internal dead links.