Edit your robots.txt file.

Status
Not open for further replies.

Megumi

FS Member
Code:
User-agent: *
Disallow: /admincp/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /members/
Disallow: /images/
Disallow: /members/
Disallow: /modcp/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

That doesn't seem to stop Google! I hear you need to add a specific line for google spiders now, they don't just look for a blank line and accept * like they used to.


 

Attachments

  • capture_01.jpg
    capture_01.jpg
    47.6 KB · Views: 29
Then why did google go on register.php when it is disallowed?

we can all tell that you were looking at the FS XXX project
xD, If I was the one on the XXX project, why would I censor other fields also. BTW, once you go to whosonline.php it updates your status to "viewing who's online".
 
Then why did google go on register.php when it is disallowed?


xD, If I was the one on the XXX project, why would I censor other fields also. BTW, once you go to whosonline.php it updates your status to "viewing who's online".

Doesn't really matter w/ the spider. Also, the whos online does not auto-update, it gives the last location you were at, typically in the index.
 
it was you because it fits the order. You looked at the porn thread, and then 1 minute later were in the Sig Request thread where you have made posts and then you quickly moved to the who's online page.
 
No adsense is used to crawl for relevant adverts, more access = more relevancy, and the ability to place ads anywhere.

It's not included in the Google index so it doesn't matter anyway.
 
The robots.txt will not be changed and echo has it set-up and working properly as he sees to be the best way. Discussion no longer needs to continue with bickering back and forth.
 
Status
Not open for further replies.
Back
Top