geo-crawlers
This skill provides a framework for auditing website accessibility to AI crawlers. It identifies specific AI bots, evaluates their impact on search visibility, and provides a systematic procedure for checking robots.txt, meta tags, HTTP headers, and AI-specific files to ensure optimal AI search presence.
Is geo-crawlers safe to install?
Review the source first: our audit of geo-crawlers's source files found 0 shell commands, 1 external URL, file reads and writes (high risk). Every command and URL listed appears verbatim in the skill's source. The skill performs network requests to fetch robots.txt and website content. It reads local files to perform analysis.
How we audit skills: our security review methodology.
Who is this skill for?
SEO professionals, web developers, and site owners who want to manage their visibility in AI-generated search results and AI training datasets.
What can you do with it?
- Audit robots.txt files for AI crawler blocking rules.
- Identify page-level AI access restrictions via meta tags and HTTP headers.
- Verify the presence of AI-specific configuration files like llms.txt and ai.txt.
- Assess the impact of JavaScript rendering on AI crawler accessibility.
- Generate recommendations for robots.txt configurations to maximize AI search visibility.
How good is this skill?
Quality score: 5/10. The skill provides a clear, well-structured methodology for auditing AI crawler access. It includes a comprehensive reference list of major AI bots and actionable configuration examples.
What does the skill file contain?
# AI Crawler Access Analysis Skill ## Purpose This skill analyzes a website's accessibility to AI crawlers -- the bots that AI companies use to discover, index, and train on web content. If AI crawlers are blocked, the site's content cannot appear in AI-generated responses regardless of its quality. Crawler access is the foundational technical requirement for GEO. ## Key Insight As of early 2026, many websites inadvertently block AI crawlers through overly aggressive robots.txt rules, inherited from legacy SEO configurations. An Originality.ai 2025 study found that over 35% of the top 1,00...
Frequently asked questions
Does blocking Google-Extended affect my Google Search rankings?
No. Blocking Google-Extended only prevents Google from using your content for Gemini model training and AI Overviews. Standard Google Search rankings are controlled by Googlebot.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot fetches content for ChatGPT's browsing and model training. OAI-SearchBot is used exclusively for ChatGPT's search feature and does not use content for model training.
Why should I block Bytespider?
The skill recommends blocking Bytespider for most Western businesses because it exhibits aggressive crawling behavior and provides minimal benefit for search visibility in Western markets.
Related skills
seo-audit
133.3KUsers seeking to diagnose SEO issues, improve organic search performance, or conduct a health check on their website
The seo-audit skill provides a structured framework for diagnosing and recommending improvements for website search engine optimization. It guides the agent through technical, on-page, and content quality assessments, including specific checks for international SEO and site-type-specific issues.
content-strategy
86.4KContent marketers, founders, and marketing teams planning content roadmaps and editorial calendars
The content-strategy skill assists users in planning content strategies, identifying topics, and building editorial calendars. It guides users through gathering business context, researching customers, analyzing competitors, and mapping content to the buyer's journey. It provides frameworks for searchable and shareable content, pillar identification, and topic clustering.
programmatic-seo
83.5KContent strategists, SEO specialists, and marketers building large-scale, data-driven website content
The programmatic-seo skill provides a framework for generating SEO-optimized pages at scale using templates and data. It guides users through keyword research, data sourcing, template design, internal linking, and indexation strategies while emphasizing content uniqueness to avoid search engine penalties.
ai-seo
70.6KContent creators, SEO professionals, and digital marketers aiming to improve their brand's presence in AI-generated search results
The ai-seo skill provides a framework for optimizing web content to increase visibility and citation rates in AI-powered search engines and LLM-based assistants like ChatGPT, Perplexity, and Google AI Overviews.