sitelift
Sign in

Sitelift’s crawler

Last updated 13 September 2026

Sitelift turns websites built with Webflow, Framer, Wix, Squarespace and Bubble into static files for the people who own them. To do that, it visits the site it is asked to export. This page explains how to recognize us, when we visit, and how to allow or block us.

How to recognize us

Every request carries this token in its user agent:

Sitelift/1.0 (+https://sitelift.download/bot)

Plain HTTP requests (Webflow, Squarespace) send this user agent:

Mozilla/5.0 (compatible; Sitelift/1.0 (+https://sitelift.download/bot))

Sites that only render in a browser (Framer, Wix, Bubble) are opened in a real Chromium, which adds the same token to its user agent. Requests come from Cloudflare’s network (published IP ranges), never from anyone’s home connection.

When we visit

  • When someone checks a URL on our site, we fetch its start page and its /sitemap.xml once, to tell them the builder and the page count.
  • When a signed-in user starts an export, having declared that they own the site or are authorized to export it, we crawl the site’s pages and the files they use.

We don’t crawl on our own initiative, index sites, or revisit them on a schedule.

How we crawl

  • At most 4 requests at a time per host.
  • A 429 Too Many Requests makes us wait before trying again: as long as its Retry-After asks, up to 30 seconds. After three attempts we skip that page or file.
  • By default we follow the robots.txt rules for User-agent: * and User-agent: Sitelift. Someone exporting a site they own can switch that off.
  • We never try to get around bot protection. When a challenge answers instead of the site, the export stops.

How to block us

  • In robots.txt: User-agent: Sitelift followed by Disallow: /.
  • In your firewall or WAF: block requests whose user agent contains Sitelift/1.0.
  • Or ask us to block your site for good: see takedown requests. We then run no export of it.

How to let us through

If your own site’s bot protection stops your export, allow requests whose user agent contains Sitelift/1.0 while the export runs. A user agent can be copied, so remove the rule afterwards. Or export the builder’s own address (for example your-site.webflow.io), which often has no protection.

Contact

Questions about our crawler: support@sitelift.download.