SiteOne Crawler

Open-source website crawler that produces audit reports, offline mirrors and Markdown exports of a site.

FreeOpen sourceWindowsmacOSLinuxCommand line
Official websitecrawler.siteone.ioDownload pagegithub.comSource codegithub.comDocumentationcrawler.siteone.io

These buttons open the developer's own site, repository or store listing in a new tab. wares.gg does not host downloads.

About SiteOne Crawler

SiteOne Crawler is a single dependency-free binary that deep-crawls a website and turns it into an interactive HTML audit report, a full offline copy, or Markdown and llms.txt output. It runs as a desktop GUI, a command-line tool with around 200 options, an interactive wizard or a CI/CD quality gate.

The audit gives a 0 to 10 quality score and covers SEO and OpenGraph, security headers and cookies, SSL/TLS certificates, WCAG accessibility, slow pages, broken links, redirects and caching headers. It can also crawl single-page apps with a headless browser, generate sitemaps and run load tests.

Strengths

  • Broad audit covering SEO, security, accessibility and performance
  • Both a desktop GUI and a scriptable CLI
  • Offline mirror and Markdown export of a whole site
  • Packages for Windows, macOS and several Linux distributions

Limitations

  • The large number of CLI options takes time to learn
  • Crawling large sites can put load on the target server

Details

Pricing
FreeFree and open source.
License
Open source, license not stated
Developer
Jan Reges
Platforms
Windows, macOS, Linux, Command line
How it runs
Downloadable app
Account
Not required
Best suited for
Web developers and SEO staff auditing a whole site from their own machine
Last verified
Added
Provenance
Facts checked against the developer's own pages and store listings, 1 sources on file.

Alternatives to SiteOne Crawler

Compare all

Software that can replace SiteOne Crawler for an important use case, and what changes if you switch.

  • Screaming Frog SEO Spider

    A desktop website crawler for Windows, macOS and Linux that audits sites for technical SEO issues.

    FreemiumProprietaryWindowsmacOSLinux

    Screaming Frog SEO Spider is a closed-source desktop crawler for the same three systems with over 300 SEO checks, but the free version stops at 500 URLs.

  • Sitebulb

    A website crawler for technical SEO audits, available as desktop software and a cloud service.

    PaidProprietaryWeb

    Sitebulb is a paid, closed-source crawler with desktop and cloud options, JavaScript crawling and custom PDF reports for clients.

  • Netpeak Spider

    A desktop SEO crawler that audits websites for technical issues.

    PaidProprietaryWindowsmacOS

    Netpeak Spider is a paid, closed-source desktop crawler for Windows and macOS focused on technical SEO, without a scriptable CLI.

  • Siteliner

    An online scanner that finds duplicate content, broken links and internal linking issues across a website.

    FreeProprietaryWeb

    Siteliner is a free web service that finds duplicate content and broken links across public sites, with no install but no offline mirror or CLI.

  • Checkbot

    Chrome extension that crawls a website and tests its pages for SEO, speed and security problems.

    FreemiumProprietaryBrowser extension

    Checkbot runs as a Chrome extension crawling sites for SEO, speed and security issues, with some features paid and no Markdown export.

  • Semrush Site Audit

    A website audit tool in the Semrush platform that crawls a site for SEO and AI search issues.

    FreemiumProprietaryWeb

    Semrush Site Audit moves crawling to a hosted platform with severity-ranked issues, but the free crawl is limited and full use needs a subscription.

SiteOne Crawler as an alternative

Listings that name SiteOne Crawler as an alternative.

  • Asqatasun

    Open-source, self-hosted engine that audits web pages and whole sites for accessibility and SEO.

    FreeOpen sourceSelf-hosted

    SiteOne Crawler is an open-source desktop and CLI crawler auditing SEO, security, accessibility and performance from your own machine instead of a server.

  • webhint

    A customizable linting tool that checks websites for accessibility, speed, compatibility and security problems.

    FreeOpen sourceBrowser extensionCommand line

    SiteOne Crawler audits whole sites for SEO, security, accessibility and performance with a desktop GUI and CLI, but has no editor integration.

Similar software

Related functionality, not necessarily a direct replacement.

Report a wrong fact or a dead link on this listing