0
Welcome Guest! Login
0 items Join Now

SOLVED robots.txt blocks rokboosters cache files and messes up googlebot rendering

  • SOLVED robots.txt blocks rokboosters cache files and messes up googlebot rendering

    Posted 8 years 1 month ago
    • Hello there,

      I am working on my website and try to increase my google bot ranking (almost just for fun...). Now, obviously I am aware that performance is part of the whole ranking process and I want to excel at least that one. Joomla seems (or at leasted seemed) pretty bulky in terms of unneccessary code but I will say that rokbooster does do a nice job on increasing performance at ranking on google page insights. Well done so far!

      Now I do run into this issue here however, that I think affects my ranking. In webmaster tools, when I fetch and render as google I stumble over this:
      This image is hidden for guests.
      Please log in or register to see it.

      The google bot isn't able to render my site.

      So I check if rokbooster is working at all:
      This image is hidden for guests.
      Please log in or register to see it.

      Seems like it's firing...

      So the source of all evil is the robots.txt:
      This image is hidden for guests.
      Please log in or register to see it.

      The robots.txt blocks the neccessary cache files from rokbooster to render the page.

      I found something here .
      User-agent: googlebot
      Allow: /cache/rokbooster/

      And using his adjustment, google bot is now able to render it correctly:
      This image is hidden for guests.
      Please log in or register to see it.


      I wonder about a few things:
      Is this practice safe?
      Should I allow the other cache files also? -- this is obviously not recommended in the default joomla robots.txt
      Should I enable the cache/rokbooster for all bots out there?
      Does this affect my ranking at all?

      I didn't find anything about this in the rokbooster documentation. Did I miss it?

      Thanks for your input! Best,
      Max
    • MrT's Avatar
    • MrT
    • Preeminent Rocketeer
    • Posts: 101084
    • Thanks: 13481
    • Web Designer/Developer

    Re: SOLVED robots.txt blocks rokboosters cache files and messes up googlebot rendering

    Posted 8 years 1 month ago
    • You should post in the extension forum or template forum as appropriate. Most of these forums are for members who have a current PAID membership. Our priority is always to assist paying members.

      Thank you for your understanding.

      Your subscription appears to have expired. Support does require a current subscription.



      Google is not "seeing" your site as end user would - if you look at the image of the site they show you you will see it doesn't look right - this is because googlebot has been prevented from accessing certain directories because of the content in your robots.txt file in your joomla root directory. You will need to comment out the lines that currently specify "disallow" on cache,components,libraries,modules and plugins (add a # at the start of the line). You could also make this change specifically for the googlebots too.

      This is what I use:
      User-agent: *
      Disallow: /administrator/
      Disallow: /bin/
      #Disallow: /cache/
      Disallow: /cli/
      #Disallow: /components/
      Disallow: /includes/
      Disallow: /installation/
      Disallow: /language/
      Disallow: /layouts/
      #Disallow: /libraries/
      Disallow: /logs/
      #Disallow: /modules/
      #Disallow: /plugins/
      #Disallow: /templates/
      Disallow: /tmp/

      Regards, Mark.
    • Last Edit: 8 years 1 month ago by MrT.
    • Please search forums before posting. Please make sure your post includes the version of the CMS you are using and a link to the problem. Annotations on screenshots can also be helpful to explain problems/goals. Please use the "secure" tab for confidential information.
  • Re: SOLVED robots.txt blocks rokboosters cache files and messes up googlebot rendering

    Posted 8 years 1 month ago
    • Hello Mark,

      well I understand the issue with the Googlebot having restricted access to the webserver. My questions are different ones:

      If googlebot can't render the website correctly, does it affect my pagerank?
      Is it an unsafe practice to allow the google bot to crawl these directories, like cache?
      Why is the default joomla robots.txt suggesting to disallow all the above directories that you commented out?
      And last but not least, shoud I allow this only for the google bot, or all bots really?
    • MrT's Avatar
    • MrT
    • Preeminent Rocketeer
    • Posts: 101084
    • Thanks: 13481
    • Web Designer/Developer

    Re: SOLVED robots.txt blocks rokboosters cache files and messes up googlebot rendering

    Posted 8 years 1 month ago
    • 1. yes - google needs to see the page as a user would
      2. no - it's safe
      3. Really you need to ask Joomla but this is my take on it.... That robots.txt file has been in Joomla for years. Google is constantly changing how they index sites (and it's a secret too - so you'll never know what they actually do). About a year ago Google made a significant change and that's when the robots.txt issue suddenly became important (Google mobile check). AS robots.txt is part of Joomla, RT does not make any changes to Joomla installation.
      4. I would allow for all bots - I don't see any harm. The worst thing that would happne is some things might be indexed that you may not want to.

      Regards, Mark.
    • Please search forums before posting. Please make sure your post includes the version of the CMS you are using and a link to the problem. Annotations on screenshots can also be helpful to explain problems/goals. Please use the "secure" tab for confidential information.
  • Re: SOLVED robots.txt blocks rokboosters cache files and messes up googlebot rendering

    Posted 8 years 1 month ago
    • thanks for your answer! that was very helpful

Time to create page: 0.051 seconds