Before learning how to use each tool, it’s worth knowing where we get our data from.

At Ahrefs, we crawl the web much like search engines do: using crawlers.

Our crawler is called AhrefsBot. It’s the #1 most active SEO crawler (and 8th most active crawler after Google’s and Bing's), visiting over 8 billion web pages every 24 hours and updating its index every 15–30 minutes.

AhrefsBot

Here’s how it works:

  1. Our crawler starts from a big list of known URLs.

  2. These URLs are then sent to the Scheduler, which decides when a URL should be crawled.

  3. When a URL is ready to be analyzed, AhrefsBot will send the URL to the Crawler and download its contents; all while strictly respecting the allow/disallow rules set in the robots.txt file.

  4. The crawler will deliver the raw data to the Parser, which extracts links on that page, the title, and other relevant metadata.

  5. The extracted data is then sent to the Indexer.

  6. It’s then added to the Link Index and made available in various reports in Ahrefs.

How Ahrefs crawls data

This is a linear and simplified explanation of how our crawler works. In order to continuously discover new pages and increase the size of our backlink index, we feed our list of URLs by surfacing new links from:

  • Parsed data

  • Redirects found via the Crawler

Currently, we show live link data and links that were deleted in the last 60 days.

About this course

How to use Ahrefs

64 lessons
3h 24m
Learn practical ways to use Ahrefs' SEO tools and reports to improve SEO.

What you’ll learn

  • How to analyze yours and your competitors’ websites with Site Explorer

  • How to master keyword research with Keywords Explorer

  • How to improve your on-page and technical SEO with Site Audit

  • How to track and improve your Google rankings with Rank Tracker

  • How to discover untapped keyword and link building opportunities with Content Explorer

  • How to get keyword and link building opportunities on autopilot with Alerts

Course by

Andrei ȚițAndrei Țiț is a product marketer at Ahrefs. He develops content to help you make the most out of Ahrefs.
Sam OhSam Oh is VP of Marketing at Ahrefs. He incorporates his commitment to education and love for entrepreneurship into actionable and easy-to-digest tutorials.