Read "how to report an issue with Bingbot"

How do I Report an Issue With Bingbot?

Bingbot is the name of the crawler used by Bing to crawl or “spider” the web. It is Bingbot's job to find new and updated pages on websites across the Internet, so that they can be processed for indexation. When crawling a website, Bingbot looks at robots.txt for special instructions from the website owner. Bingbot honors robots.txt directives, including the crawl-delay: setting, and, in the absence of a crawl-delay, respects the input from Webmasters in the Crawl Control Feature.

Generally, Bingbot does a good job at determining how frequent it should visit pages on your site, taking robots.txt and Crawl Control rules and hints into consideration. We call this “Crawl Politeness”. However, there still may be cases were you feel Bingbot is not polite enough and is visiting your pages more than works for you (overcrawling).

Images Credits - http://madmonsterstudio.com/bingbot.html

REPORTING OVER-CRAWLING

Here’s the steps you can follow if you think Bingbot is over-crawling your site or not observing robots.txt rules:

Verify that the bot traffic you are seeing is in fact from a valid Bingbot server. You can do this by not only looking at the User-Agent string (which can be easily spoofed by anyone) but also at the IP address and using the Verify Bingbot tool to get a verdict

Once you have verified that this involves genuine Bingbot traffic, you can reduce crawler traffic as follows (if you haven’t done so already):

Lower the speed of crawl during busy hours using the Crawl Control feature

If that’s not sufficient, add a crawl-delay: directive to your robots.txt: Bing supports whole number values ranging from 1 – 20. Each number maps to the length in seconds of time slices in which we divide a 24-hour crawl cycle. In this context, the value 1 means you allow us a maximum of one request for each 1-second time slice – which is slow, but still adequate for smaller sites. 20 is extremely slow and means we are allowed only one request per every 20 second time slice for a 24-hour crawl cycle.

If you have followed step 1 and 2 and the issue is still present, you can contact Bing Webmaster Support. Fill out the required fields and in the “What type of problem do you have?” dropdown, select “Under-Crawling or Over-Crawling inquiry” and describe the problem you are seeing. You can expect a reply within 24-48 hours. When you report over-crawling issues, the support team will ask you to provide server log samples that show the Bingbot activity over a certain period of time in a next step, so make sure to have those ready.

Source - https://www.bing.com/webmaster/help/how-to-report-an-issue-with-bingbot-25c19802

About SLU Injection Blog

This is a short description in the author block about the author. You edit it by entering text in the "Biographical Info" field in the user admin panel.