Van Born Labs / Guides / Marketing & Ads

Website SEO Audit Crawler: Meta, Sitemap and Broken Links

A weekly technical crawl you will actually run: titles, canonicals, robots, sitemaps, broken links, TTFB and a per-page score.

By Arkadiusz Van Born. Updated 2026-11-18 (America/Phoenix).

A technical SEO audit, at the boring and useful end, is a crawl: titles and meta descriptions, canonicals, hreflang, Open Graph tags, H1 and H2 structure, status codes and redirects, broken links, image alt text, noindex, robots.txt, sitemap coverage, time to first byte, and page size. Every page gets a list of issues. Optionally, every page gets a score so a weekly regression is visible in a spreadsheet.

You do not need a full desktop suite for that. You need a lightweight HTTP crawler you can run on a schedule, cheaply enough that you actually run it.

What to check, in order

  1. Fetch and status. Redirect chains, 4xx and 5xx, unexpected 200s on deleted URLs.
  2. Indexability. noindex, robots.txt disallows, and sitemap membership should agree with each other. A page that is in the sitemap and also noindex is a bug.
  3. Titles and meta descriptions. Length, uniqueness, and whether they exist at all. Duplicate titles across a site are one of the highest-value cheap finds.
  4. Canonical and hreflang. Present, absolute, and consistent with the URL you crawled.
  5. Headings and text. A missing H1, or a page with almost no words, is worth a look. Do not treat heading order as a moral failing.
  6. Broken outgoing links and missing alt text. Mechanical, easy to leave unfixed, embarrassing in a client report.
  7. Performance hints. TTFB and page size will not replace a lab test, but a sudden jump is a useful alert.

A crawler that does this over HTTP, without rendering every page in a browser, will miss issues that only appear after JavaScript runs. Say that out loud if you work on a heavy client-rendered site, and pair the crawl with a smaller rendered sample when it matters.

Van Born Labs

Try it live

Paste a public website URL for a short SEO audit: title, issues and a site score. Three free checks a day.

3 free checks per day, per visitor

How to run a weekly pass

  • Start from the sitemap when you have a good one; otherwise start from a short list of important URLs and a conservative follow limit.
  • Cap the crawl (for example a few hundred pages on a large site) so the job is cheap and the report is readable.
  • Store the results. The value is often the diff: new broken links, a title that disappeared, a noindex that landed on a money page.
  • Fix the issues in the CMS. A crawl that nobody applies is a PDF.

The Website SEO Audit actor is this kind of crawler: $0.80 per 1,000 pages, no proxy, a per-page issue list and a score. A RapidAPI listing is prepared if you want the same check inside an application.

For ecommerce feed issues (GTINs, prices, Shopping disapprovals), that is a different tool — see How to fix Google Merchant Center feed errors.

Independent software by Van Born Labs, not a replacement for a full site crawl in a browser lab, and not affiliated with Google Search Console.

Questions

What should a technical SEO audit crawler check?

Status and redirects, noindex and robots.txt, sitemap coverage, title and meta uniqueness, canonical and hreflang, headings, broken links, image alt text, TTFB and page size.

Will an HTTP crawler catch JavaScript-only issues?

No. A lightweight HTTP crawl can miss problems that appear only after rendering. Pair it with a smaller rendered sample on heavy client-rendered sites.

Tools mentioned

Related guides

Canonical URL: https://arkadiuszvanborn.com/labs/guides/website-seo-audit-crawler. Support: arkadiuszvanborn@gmail.com.