AI crawlers outpace search engines in Rubik Digital website study

AI crawlers generated more requests than traditional search engines combined across 17 business websites hosted and managed by UK agency Rubik Digital. The research analysed 1,440,971 crawler requests recorded over 14 days from 28 September to 11 October 2026. AI crawlers accounted for 40.4% of requests, compared with 28.5% for traditional search engines.

Activity was heavily concentrated among providers: one operator accounted for 57% of AI crawler requests and another for 38%, leaving 5% shared among the remaining providers. AI crawlers made 28.7 times as many requests as one major search engine’s crawler, although that comparison fell to 8.7 times after excluding the three busiest sites. AI assistants also fetched pages live for users about 940 times a day across the sample. Server errors occurred in 1.6% of AI crawler requests.

The analysis counted only page requests from crawlers identifying themselves by user-agent name, used no visitor data and published only combined findings across the managed sites. Rubik Digital CTO Giedrius Kudzinskas warned that content appearing only after JavaScript runs may remain inaccessible to many AI tools, creating visibility problems for businesses.