Jump to content

Spider/search Bots stopped visiting

Guest rct2·com

Recommended Posts

I took a couple of domains down for around 24 hours while I moved them to a new server. So as a result, the IP address has changed. I had set up DNS earlier in the week so that the new IP should propagate in 30 minutes.

Less than 24 hours after I brought the sites back up, one board has had a record of almost 3000 different visitors in a 60 minute interval. Almost 2800 of them were Guests.

Another 24 hours later and the numbers are still running consistently above 1500 in 60 minutes.

But NONE of them are spiders/bots. Not a single search bot has visited the domain since 21st March just before I shut it down.

Does it NORMALLY take the search bots this long to revisit a domain after it's been taken out for 24 hours?

Is it possible that the new IP on the server, which presumably was previously allocated to somebody else has been 'blacklisted' by the search engines? [I have 2 boards on the server, both on different IPs. Neither board has been 'crawled' since 21st March, so I doubt it could be a 'black list' issue.]

Link to comment
Share on other sites

Sounds like the one board is being hit by bots, just not the ones you want.

Is the server handling the increased load?

Your sure your bot recognition in IPB is turned on?

Have you looked at the actual Apache access logs to see if any search engines are actually hitting your sites?

Link to comment
Share on other sites

Server is handling the increased load.

Bot recognition is turned on in IPB. [I turned it off, then on again, in case it forced some kind of recache]

I KNOW the search engines are hitting the site because I've looked at the latest cached Google page and it is from today. But here's an access log entry anyway. - - [24/Mar/2009:04:03:58 +0000] "GET /index.php?act=Post&CODE=02&f=2&t=97844&qpid=415605 HTTP/1.1" 200 4434 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Create New...