AhrefsBot

Learn about the web crawler that powers Ahrefs.

Version
7.0
Bot type
Good (Identifies itself, has an official moniker)
Category
Marketing
Obeys robots.txt
Yes
Obeys crawl delay
Yes
User-agent string
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Reverse DNS suffix
ahrefs.com
IP address range

Website status

Check the status of your website and whether it can be crawled

What is AhrefsBot?

AhrefsBot is a web crawler that powers the database for both Ahrefs, an online data toolset, and Yep, a revenue-sharing web search engine. It’s the third most active crawler after Google’s and Bing's, visiting over 8 billion web pages every 24 hours and updating its index every 15–30 minutes.

Our bot indexes fresh, accurate information about websites and their content, and how they link to each other. This data is incredibly useful and can be harnessed in many ways:

  • Search with Yep to find top-quality content and all the information you need, all while supporting an alternative search engine that aims to put money in the pockets of your favorite content creators.

  • If you’re a website owner, use Ahrefs to analyze your website's structure, content, and backlinks to provide insights into its performance – then optimize it to improve user experience, rank higher in search engines, and get more traffic.

  • When it comes to web-scale research, Ahrefs lets you dig into how the popularity of a website or specific topic has changed over time, which websites are linking to what types of content, how many times a brand has been mentioned recently, and so on.

/

Rank

Bot

Owner

Category

1

GoogleBot

Google

Search Engine Crawler

2

BingBot

Microsoft

Search Engine Crawler

3

DataDog Synthetic Bot

DataDog

Monitoring & Analytics

4

AhrefsBot

Ahrefs

Search Engine Optimization

5

Qualys

Qualys

Search Engine Crawler

6

YandexBot

Yandex

Search Engine Crawler

7

PetalBot

Huawei

Search Engine Crawler

8

Google AdsBot

Google

SAdvertising & Marketing

9

Google Image Proxy

Google

Page Preview

10

Google Images

Google

Search Engine Crawler

Bots ranked by number of requests. Data supplied as of April, 2023

Questions? We have answers

  • Does AhrefsBot respect robots.txt files?

    Yes. Absolutely.We strictly respect robots.txt on both disallow and allow rules.

  • How do I control AhrefsBot on my website?

    Since AhrefsBot strictly follows the robots.txt file on your website, you're in full control of what it does.To change the frequency at which AhrefsBot visits your site, just specify the minimum acceptable delay between two consecutive requests in your robots.txt file:
    User-agent: AhrefsBotCrawl-Delay: [value](Where Crawl-Delay value is time in seconds.)
    If you want to stop AhrefsBot from visiting your site, add these two lines to the robots.txt file on your server:
    User-agent: AhrefsBotDisallow: /
    Please note that AhrefsBot may need some time to pick up the changes in your robots.txt file. This is done before the next scheduled crawl.Also, if your robots.txt contains errors, AhrefsBot won’t be able to recognize your commands and it'll continue crawling your website the way it did before.Read more about robots.txt at www.robotstxt.org.If you think that AhrefsBot is misbehaving on your website or if you have any questions, please reach out to [email protected] and we'll do our best to help.

Have other questions?

If you have more questions,
contact us so we can help.