User-agent: * Disallow: /ss/ Disallow: /log/auth/ Disallow: /b/ Disallow: /f/ Disallow: /wp-login.php Crawl-delay: 10 #https://wbsrch.com/faq/ -- 'We support robots.txt and obey rules for the Google user agent. We don't look for own user agent because a statistically significant number of websites have not added rules specifically for WbSrch.' # Let's increment the number, then. You are not the Googlebot. User-agent: WbSrch User-agent: YandexBot Disallow: / User-agent: YandexImages Disallow: / User-agent: Baiduspider Disallow: / User-agent: AhrefsBot Disallow: / User-agent: AMZNKAssocBot Disallow: / User-agent: MJ12Bot Disallow: / User-agent: sistrix Disallow: / User-agent: siclab Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: ScoutJet Disallow: / User-agent: Docoloc Crawler Disallow: / User-agent: Math. Comp. PA Crawler Disallow: / User-agent: meanpathbot Disallow: / User-agent: Abonti Disallow: / User-agent: aiHitBot Disallow: / User-agent: GarlikCrawler Disallow: / User-agent: Aboundex Disallow: / User-agent: Aboundexbot Disallow: / User-agent: rogerbot Disallow: / User-agent: SWEbot Disallow: / User-agent: EC2LinkFinder Disallow: / User-agent: EventGuruBot Disallow: / User-agent: panscient.com Disallow: / User-agent: Lipperhey Link Explorer Disallow: / User-Agent: lipperhey Disallow: / User-Agent: Lipperhey Disallow: / User-agent: Lipperhey-Kaus-Australis Disallow: / User-agent: Lipperhey Spider Disallow: / User-agent: Comodo-Certificates-Spider Disallow: / User-agent: COMODO SSL Checker Disallow: / User-agent: Linguee Disallow: / User-agent: linkdex.com Disallow: / User-agent: CompSpyBot Disallow: / User-agent: oBot Disallow: / User-agent: R6_FeedFetcher Disallow: / User-agent: R6_FeedFetcher_ Disallow: / User-agent: R6_CommentReader Disallow: / User-Agent: integromedb Disallow: / User-agent: BacklinkCrawler Disallow: / User-agent: 4SeoHuntBot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: UnisterBot Disallow: / User-agent: 008 Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: awcheck Disallow: / User-agent: SiteExplorer Disallow: / User-agent: Sosospider Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: CatchBot Disallow: / User-agent: ltbot Disallow: / User-agent: EasouSpider Disallow: / User-agent: ProCogSEOBot Disallow: / User-agent: NCBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: Nutch Disallow: / User-agent: fr-crawler Disallow: / User-agent: XoviBot Disallow: / User-agent: dotbot Disallow: / User-agent: NetShelter ContentScan Disallow: / User-agent: Exploratodo Disallow: / User-agent: BLEXBot Disallow: / User-agent: MegaIndex.ru Disallow: / User-agent: finbot Disallow: / User-agent: MixrankBot Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: gimmeusa Disallow: / User-agent: ltx71 Disallow: / User-agent: TeeRaidBot Disallow: / User-agent; Dataprovider Disallow: / User-agent: CrazyWebCrawler-Spider Disallow: / User-agent: TestCrawler Disallow: / User-agent: DomainAppender Disallow: / User-agent: PrivateSearch Disallow: / User-agent: DomainSigmaCrawler Disallow: / User-agent: Feosey Mohk Crawler Disallow: / User-agent: HELLO Crawler Disallow: / User-agent: CanBot Disallow: / User-agent: CanBot-BETA Disallow: / User-agent: CheckMarkNetwork Disallow: / User-agent: CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html) Disallow: / User-agent: LinkosmosBot Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sogou Disallow: / User-agent: CheckMarkNetwork Disallow: / User-agent: SafeDNSBot Disallow: / Uwser-agent: cis455crawler Disallow: / User-agent: DomainCrawler Disallow: / User-agent: WBSearchBot Disallow: / User-agent: LinqiaCrawlerBot Disallow: / User-agent: CheckMarkNetwork Disallow: / User-agent: panscient.com Disallow: / User-agent: Seoterritory.com bot Disallow: / User-agent: seoscanners.net Disallow: / User-agent: rogerbot Disallow: / User-agent: BUbiNG Disallow: / User-agent: netEstate Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: Barkrowler Disallow: / User-agent: ExtLinksBot Disallow: / User-agent: AlphaSeoBot Disallow: / User-agent: AlphaSeoBot-SA Disallow: / User-agent: 007AC9 Crawler Disallow: / User-agent: BDCbot Disallow: / User-agent: G-i-g-a-b-o-t Disallow: / User-agent: Riddler Disallow: / User-Agent: The Knowledge AI Disallow: /