Screaming Frog SEO Spider homepage with the green crawler banner for Windows, macOS and Linux, download and pricing tabs, and the offer of 500 free URLs or a $279 yearly licence — Screaming Frog SEO Spider

How much does Screaming Frog cost?

A Screaming Frog SEO Spider licence costs $279 a year per user, dropping to $265 from five licences, $249 from ten and $235 from twenty. The same tiers are £199, £189, £179 and £169 in pounds, or €245, €235, €219 and €209 in euros. Each key is for one named person, and licences do not auto-renew. When a licence lapses, the software falls back to the free version and saved crawls can no longer be opened.

What the free version of Screaming Frog can and cannot do

The free version crawls up to 500 URLs at a time, with no sign-up and no time limit. It finds broken links and redirects, audits titles and meta descriptions, reads robots directives and builds an XML sitemap. It cannot save crawls or change any setting, so no custom user-agent, no cookies and no JavaScript rendering. A tube with a page per video blows through 500 URLs fast.

Crawling an adult site behind an age gate

Google asks adult sites to let Googlebot crawl past the age gate, warning that hidden images and videos may rank poorly and that safe content behind the gate can get a whole site classified as explicit and filtered out by SafeSearch. With a licence, Screaming Frog can check what a crawler actually sees:

switch the user-agent to Googlebot, Bingbot, an AI crawler or your own string,
send custom HTTP headers, including the cookie your age gate sets after the "I am 18" click,
use forms-based authentication to log in to a members' area in the built-in browser before crawling.

If your server verifies Googlebot by IP address, as Google recommends, spoofing the Googlebot user-agent from your own connection will still hit the gate. Use the age-gate cookie instead.

The hardware Screaming Frog needs for large sites

Screaming Frog needs a 64-bit OS: Windows, macOS on Intel or Apple Silicon, Ubuntu or Fedora. For millions of URLs, the developer recommends 16 GB of RAM, a 1 TB SSD and database storage mode. Version 24.3 shipped in June 2026, and recent releases added an MCP server for driving crawls from AI assistants.

Strengths and limitations of Screaming Frog SEO Spider

Strengths

Cookie, header and user-agent control to get past age gates.
Over 300 checks, exportable page by page.
One yearly licence with every feature.
A free tier with no sign-up.

Limitations

No configuration in the free version.
One user per licence.
Saved crawls locked once the licence expires.
No keyword or backlink data.

Screaming Frog vs Semrush Site Audit

Semrush runs site audits from its own servers as part of plans starting at $139.95 a month, alongside keyword and backlink data. Its SiteAuditBot cannot carry an age-gate cookie, so the gate has to let it through server-side. A year of Screaming Frog costs about two months of Semrush and handles the age gate on its own.