TametheBots
TametheBots
Pricing
- Contact Vendor
Description
ROBOTS.TXT TESTING TOOL
Checks a list of URLs against a robots.txt file to see if they are allowed, or blocked and if so, by what rule.
Uses the Google Robots.txt Parser and Matcher Library, which matches the one used in production at Google.
Parsing and matching is one part of the picture, sometimes a search engine or other service might choose to fall back or ignore certain rules, like Googlebot-Image falling back to Googlebot rules if no specific User-agent rule is found. This tool attempts to mimic the behaviour here for Google and Applebot.
Similar Tools
101 Data Solution
DATA SOLUTIONS THAT PROTECT & ENHANCE YOUR BUSINESS Welcome to 101 Data ...
1010data
Smarter Decisions. Better Outcomes. 1010data’s Insights Platform activates e...
10Duke
Software Licensing and Entitlements Made Easy Connect to your online custom...
11antsanalytics
Turn Your ‘Loyalty’ Program Into A Loyalty Program 11Ants Retail Insights Cl...
123FormBuilder
One form builder. Infinite data collection use cases. Focus on the work t...
140 Proof
The most precise way to advertise in mobile apps. Our patented technology ...
1CRM
1pointinteractive
Email Marketing Obsessed With Your Success 1 Point Interactive creates an em...
1sales
Welcome to your new cockpit! Simple & clear In your 1sales cockpit, you...
