Windwalker222 Posted May 6, 2016 Share Posted May 6, 2016 Okay, so I have been trying to get google to stop crawling and returning errors on everything from images to uploads to javascript_global and javascript_forums. I'm still working on lessening the errors. I'm trying to figure out why my robots.txt looks like this with the red X's. Can someone tell me what is wrong? This is only the first half of the robots.txt in the Google Webmaster tools/robots tester. Link to comment Share on other sites More sharing options...
tnn Posted May 6, 2016 Share Posted May 6, 2016 So your forum is www.thisdomain.com/forum/? Link to comment Share on other sites More sharing options...
Nathan Explosion Posted May 6, 2016 Share Posted May 6, 2016 What is the red dot before User-agent? Check the robots.txt live on your site - does that show there? Link to comment Share on other sites More sharing options...
tnn Posted May 6, 2016 Share Posted May 6, 2016 47 minutes ago, Windwalker222 said: Okay, so I have been trying to get google to stop crawling and returning errors on everything from images to uploads to javascript_global and javascript_forums. I'm still working on lessening the errors. I'm trying to figure out why my robots.txt looks like this with the red X's. Can someone tell me what is wrong? This is only the first half of the robots.txt in the Google Webmaster tools/robots tester. Something like this may help -> https://productforums.google.com/forum/#!msg/webmasters/4ussUp3ZS3M/zJpkjolWe2MJ Where it might be a white space issue. So use notepad or the Apple equivalent and try it again to see if that helps remove the error signs. Link to comment Share on other sites More sharing options...
Windwalker222 Posted May 6, 2016 Author Share Posted May 6, 2016 Thanks for the input. I went to the online editor in the webmaster tools robots area. Once I removed the red error dot google had inserted all the errors cleared. then when I tried to submit it would say try again later. I added the space in front of User-Agent (which I didn't think should be there to start with) and it submitted fine and shows no errors. Puzzled by that. Here is the live robots.txt The added portions (last few lines after dname) are my attempts to stop some of the huge number of errors in my error log from those areas. I seem to have all the "upload" errors stopped but these still occur. [Fri May 06 14:35:42 2016] [error] [client 66.249.64.13] File does not exist: /home/countrydepotforum.com/uploads/javascript_global/root_map.js.6f7380432846e004a22764ab56bd8f6f.js [Fri May 06 14:39:48 2016] [error] [client 66.249.64.13] File does not exist: /home/countrydepotforum.com/uploads/javascript_core/plugins_plugins.js.d33f44408b37ee1066fbb06fdb71ccd5.js [Fri May 06 14:40:04 2016] [error] [client 66.249.64.13] File does not exist: /home/countrydepotforum.com/uploads/javascript_global/root_front.js.80fb46fd530720c2ed68d66e67a4c81a.js [Fri May 06 14:43:12 2016] [error] [client 66.249.64.13] File does not exist: /home/countrydepotforum.com/uploads/javascript_calendar/front_front_browse.js.4af46542188362ba525882b5c1b1e852.js Current robots text that is not producing the red errors in google. User-agent: * Disallow: /forum/admin/ Disallow: /forum/cache/ Disallow: /forum/converge_local/ Disallow: /forum/hooks/ Disallow: /forum/ips_kernel/ Disallow: /forum/retail/ Disallow: /forum/public/style_captcha/ Disallow: /forum/index.php?app=core&module=task Disallow: /forum/index.php?app=forums&module=moderate§ion=moderate Disallow: /forum/index.php?app=forums&module=extras§ion=forward Disallow: /forum/index.php?app=members&module=chat Disallow: /forum/index.php?&unlockUserAgent=1 Disallow: /forum/*&module=search Disallow: /forum/*app=core&module=global§ion=reputation Disallow: /forum/*app=core&module=usercp Disallow: /forum/*app=members&module=messaging Disallow: /forum/*&p= Disallow: /forum/*&pid= Disallow: /forum/*&hl= Disallow: /forum/*&start= Disallow: /forum/*view__getnewpost$ Disallow: /forum/*view__getlastpost$ Disallow: /forum/*view__old$ Disallow: /forum/*view__new$ Disallow: /forum/*view__getfirst$ Disallow: /forum/*view__getprevious$ Disallow: /forum/*view__getnext$ Disallow: /forum/*view__getlast$ Disallow: /forum/*&view=getnewpost$ Disallow: /forum/*&view=getlastpost$ Disallow: /forum/*&view=old$ Disallow: /forum/*&view=new$ Disallow: /forum/*&view=getfirst$ Disallow: /forum/*&view=getprevious$ Disallow: /forum/*&view=getnext$ Disallow: /forum/*&view=getlast$ Disallow: /forum/*?s= Disallow: /forum/*&s= Disallow: /forum/index.php?app=core&module=global§ion=login&do=deleteCookies Disallow: /forum/index.php?app=forums&module=extras§ion=rating Disallow: /forum/index.php?app=forums&module=forums§ion=markasread Disallow: /forum/*&do=who Disallow: /forum/*§ion=dname Disallow: /forum/javascript_core/ Disallow: /forum/arcade/* Disallow: /forum/arcade/images/ Disallow: /forum/uploads/ Disallow: /forum/uploads/javascript_global/ Disallow: /forum/uploads/javascript_forums/ Link to comment Share on other sites More sharing options...
Tripp★ Posted May 6, 2016 Share Posted May 6, 2016 Remove /forum/ from all of the things in your robots.txt - That should fix it. Although I noticed you have IPS4 - and I don't know if any of that is actually relevant anymore. As that looks like a robots.txt from IP.Board 3.x Link to comment Share on other sites More sharing options...
Windwalker222 Posted May 6, 2016 Author Share Posted May 6, 2016 Thanks, I will try that next. I appreciate all the help today from all of you. I have weeded out most of the errors in my server error log caused by google and have no errors showing in the google robots tester now. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.