Started by Theo Gottwald, November 07, 2009, 09:53:30 PM
0 Members and 1 Guest are viewing this topic.
QuoteSubject: Reporting a ProblemHi there. I manage a little web site at www.thinbasic.com and community.thinbasic.comProblem is that your spiders are visiting my site so much frequently that a lot of my limited bandwidth is consumed by your spiders. Since about two weeks I can see even more that 1 hundred spiders present almost at the same time scanning all posts in our forum.I have a limited band with my provider and my users have to support somedelays when using our sites.I can understand your needs but please understand mine.Hope someone will read this post and reply to me soon.ThanksEros
QuoteHello Eros, Thank you for writing to Yahoo! Search.We apologize for the excess traffic to your site. In a continuing effortto improve our index freshness and comprehensiveness we are bringing online additional crawler machines to prepare them for production. To limit the amount of traffic your site receives, you can place a crawl-delay in your robots.txt, specifying the number of seconds each crawler will wait between each request. Please check out the following pages for detailed information about this: http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html http://www.ysearchblog.com/archives/000078.htmlAgain, my apologies for the inconvenience and we appreciate your patience as we're improving Yahoo! Search.Thank you again for contacting Yahoo! Search.Regards,YasmenYahoo! Search Customer Care
Page created in 0.103 seconds with 22 queries.