Invision Community 4: SEO, prepare for v5 and dormant account notifications Matt November 11, 2024Nov 11
Posted May 6, 20168 yr Okay, so I have been trying to get google to stop crawling and returning errors on everything from images to uploads to javascript_global and javascript_forums. I'm still working on lessening the errors. I'm trying to figure out why my robots.txt looks like this with the red X's. Can someone tell me what is wrong? This is only the first half of the robots.txt in the Google Webmaster tools/robots tester.
May 6, 20168 yr What is the red dot before User-agent? Check the robots.txt live on your site - does that show there?
May 6, 20168 yr 47 minutes ago, Windwalker222 said: Okay, so I have been trying to get google to stop crawling and returning errors on everything from images to uploads to javascript_global and javascript_forums. I'm still working on lessening the errors. I'm trying to figure out why my robots.txt looks like this with the red X's. Can someone tell me what is wrong? This is only the first half of the robots.txt in the Google Webmaster tools/robots tester. Something like this may help -> https://productforums.google.com/forum/#!msg/webmasters/4ussUp3ZS3M/zJpkjolWe2MJ Where it might be a white space issue. So use notepad or the Apple equivalent and try it again to see if that helps remove the error signs.
May 6, 20168 yr Author Thanks for the input. I went to the online editor in the webmaster tools robots area. Once I removed the red error dot google had inserted all the errors cleared. then when I tried to submit it would say try again later. I added the space in front of User-Agent (which I didn't think should be there to start with) and it submitted fine and shows no errors. Puzzled by that. Here is the live robots.txt The added portions (last few lines after dname) are my attempts to stop some of the huge number of errors in my error log from those areas. I seem to have all the "upload" errors stopped but these still occur. [Fri May 06 14:35:42 2016] [error] [client 66.249.64.13] File does not exist: /home/countrydepotforum.com/uploads/javascript_global/root_map.js.6f7380432846e004a22764ab56bd8f6f.js [Fri May 06 14:39:48 2016] [error] [client 66.249.64.13] File does not exist: /home/countrydepotforum.com/uploads/javascript_core/plugins_plugins.js.d33f44408b37ee1066fbb06fdb71ccd5.js [Fri May 06 14:40:04 2016] [error] [client 66.249.64.13] File does not exist: /home/countrydepotforum.com/uploads/javascript_global/root_front.js.80fb46fd530720c2ed68d66e67a4c81a.js [Fri May 06 14:43:12 2016] [error] [client 66.249.64.13] File does not exist: /home/countrydepotforum.com/uploads/javascript_calendar/front_front_browse.js.4af46542188362ba525882b5c1b1e852.js Current robots text that is not producing the red errors in google. User-agent: * Disallow: /forum/admin/ Disallow: /forum/cache/ Disallow: /forum/converge_local/ Disallow: /forum/hooks/ Disallow: /forum/ips_kernel/ Disallow: /forum/retail/ Disallow: /forum/public/style_captcha/ Disallow: /forum/index.php?app=core&module=task Disallow: /forum/index.php?app=forums&module=moderate§ion=moderate Disallow: /forum/index.php?app=forums&module=extras§ion=forward Disallow: /forum/index.php?app=members&module=chat Disallow: /forum/index.php?&unlockUserAgent=1 Disallow: /forum/*&module=search Disallow: /forum/*app=core&module=global§ion=reputation Disallow: /forum/*app=core&module=usercp Disallow: /forum/*app=members&module=messaging Disallow: /forum/*&p= Disallow: /forum/*&pid= Disallow: /forum/*&hl= Disallow: /forum/*&start= Disallow: /forum/*view__getnewpost$ Disallow: /forum/*view__getlastpost$ Disallow: /forum/*view__old$ Disallow: /forum/*view__new$ Disallow: /forum/*view__getfirst$ Disallow: /forum/*view__getprevious$ Disallow: /forum/*view__getnext$ Disallow: /forum/*view__getlast$ Disallow: /forum/*&view=getnewpost$ Disallow: /forum/*&view=getlastpost$ Disallow: /forum/*&view=old$ Disallow: /forum/*&view=new$ Disallow: /forum/*&view=getfirst$ Disallow: /forum/*&view=getprevious$ Disallow: /forum/*&view=getnext$ Disallow: /forum/*&view=getlast$ Disallow: /forum/*?s= Disallow: /forum/*&s= Disallow: /forum/index.php?app=core&module=global§ion=login&do=deleteCookies Disallow: /forum/index.php?app=forums&module=extras§ion=rating Disallow: /forum/index.php?app=forums&module=forums§ion=markasread Disallow: /forum/*&do=who Disallow: /forum/*§ion=dname Disallow: /forum/javascript_core/ Disallow: /forum/arcade/* Disallow: /forum/arcade/images/ Disallow: /forum/uploads/ Disallow: /forum/uploads/javascript_global/ Disallow: /forum/uploads/javascript_forums/
May 6, 20168 yr Remove /forum/ from all of the things in your robots.txt - That should fix it. Although I noticed you have IPS4 - and I don't know if any of that is actually relevant anymore. As that looks like a robots.txt from IP.Board 3.x
May 6, 20168 yr Author Thanks, I will try that next. I appreciate all the help today from all of you. I have weeded out most of the errors in my server error log caused by google and have no errors showing in the google robots tester now.
Archived
This topic is now archived and is closed to further replies.