Understand AI traffic and crawler activity
Distinguish human AI referrals, realtime assistant fetches, indexing crawlers, and training crawlers.
AI-related activity includes several different kinds of requests. Neopress separates them so that a crawler reading your site is not confused with a person visiting it. Use the reports to understand recorded access, then check actual search results or AI answers separately when that is your question.
Know which activity you are reading
- AI referral: a human visit whose available source information points to an AI service. It depends on the referrer or tracking parameters being passed to the site.
- AI bots: realtime AI assistant fetches, such as an assistant retrieving a page while responding to a user.
- Indexing: background search and answer-index crawler activity, including recognized search crawlers.
- Training: requests identified as model-training crawler activity.
These classifications describe recognized requests. A training-crawler visit does not prove the content entered a training dataset. An assistant fetch does not prove the page was cited, recommended, or even displayed in the final answer.
Review activity over a period
- Open Insights → Analytics and select the period you want to review.
- Look at the AI bots metric and chart alongside human visitors and pageviews.
- Review the source panel to distinguish AI referrals from bot requests. Where available, open the bot details to see the recognized operator or crawler.
- If you apply the AI-bot filter, remember that human-only measures such as visitor journeys and campaign parameters are not equivalent bot metrics.
Detailed bot identities and page crawl histories require Growth. Summary counts may be visible while details remain locked; a lock is an access boundary, not evidence of zero activity.
Check which pages were crawled
Open Indexing or Training in the search/crawler panel. Expand a page row to review its recorded crawler activity and last edit. The page crawl-history view is not limited by the dashboard's selected date range.
- No crawl since edit: the recorded edit is newer than the latest recorded crawl. Check that the intended change is actually published.
- Crawled: there is a recorded crawl. This is not Google's index approval.
- Not crawled yet: no matching crawl is recorded in this view. It does not prove that no service has ever seen any version of the page.
Choose a useful next action
Open important public pages and check their availability, links, metadata, and sitemap entries. Review customized robots.txt rules if an intended crawler appears blocked. Use Missing URLs to investigate requested addresses returning 404 and decide whether a moved page needs a redirect.
Then connect the evidence to your objective. For search inclusion, inspect the URL in Google Search Console. For inquiries, review human traffic and Leads. For AI citations, inspect actual answers. Neopress's crawl counts are not a citation count or a share-of-answer ranking.