Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 15 of 15
  1. #1
    New Coder
    Join Date
    Oct 2005
    Location
    Near some trees
    Posts
    66
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Hiding pages from SearchBots

    Does anybody know how I can exclude certain pages being submitted to search engines whilst still allowing the majority of my pages to be submitted.

    In a nutshell, how do tell bots to only record particular pages and ignore others?

    Thanks in advance.

  • #2
    Senior Coder gsnedders's Avatar
    Join Date
    Jan 2004
    Posts
    2,340
    Thanks
    1
    Thanked 7 Times in 7 Posts
    Google for robots.txt (sorry, I don't have the time to give any specific info).

  • #3
    New Coder
    Join Date
    Oct 2005
    Location
    Near some trees
    Posts
    66
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Ta for that

  • #4
    $object->toCD-R(LP); vinyl-junkie's Avatar
    Join Date
    Jun 2003
    Posts
    3,100
    Thanks
    2
    Thanked 23 Times in 23 Posts
    What you're talking about is called robots exclusion. Here is one of the best sites I've found to explain how that works. Hope it helps.
    Music Around The World - Collecting tips, trade
    and want lists, album reviews, & more
    SNAP to it!

  • #5
    Registered User
    Join Date
    Oct 2004
    Posts
    592
    Thanks
    0
    Thanked 1 Time in 1 Post

  • #6
    New Coder
    Join Date
    Oct 2005
    Location
    Near some trees
    Posts
    66
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I have added a Robots.txt file to the root directory. I guess only time will tell if it works or not?

  • #7
    Senior Coder gsnedders's Avatar
    Join Date
    Jan 2004
    Posts
    2,340
    Thanks
    1
    Thanked 7 Times in 7 Posts
    Quote Originally Posted by Rocket Boy
    I have added a Robots.txt file to the root directory. I guess only time will tell if it works or not?
    Yeah, I don't think there are any other ways of testing it.

  • #8
    New Coder
    Join Date
    Oct 2005
    Location
    Near some trees
    Posts
    66
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I can't make up my mind wether or not their was a hint of sarcasm in their so I'd better ask, is there a test?

  • #9
    Senior Coder gsnedders's Avatar
    Join Date
    Jan 2004
    Posts
    2,340
    Thanks
    1
    Thanked 7 Times in 7 Posts
    Quote Originally Posted by Rocket Boy
    I can't make up my mind wether or not their was a hint of sarcasm in their so I'd better ask, is there a test?
    Probably. Don't know where though.

  • #10
    Regular Coder
    Join Date
    Oct 2005
    Location
    NYgamers.net
    Posts
    108
    Thanks
    0
    Thanked 0 Times in 0 Posts
    The robots will stop visiting once they see the disallow command in the robots.txt file. However, the search engine might still keep a copy of those pages on file. It would be best to also use the meta tags for the search engines, robots, googlebot, etc., to tell them that the content has expired and not to re-index that page/site.

    http://www.searchenginewatch.com is a pretty usefull site.

    * Security Note * Never put ANY file/directory references to secured information in your robots.txt file. That's one of the first places a hacker will hit up when looking for exploitable files.

  • #11
    Registered User
    Join Date
    Oct 2004
    Posts
    592
    Thanks
    0
    Thanked 1 Time in 1 Post
    Both robots.txt and meta tags only suggest bots for stopping.
    They don't force to prevent them.

  • #12
    New Coder
    Join Date
    Oct 2005
    Location
    Near some trees
    Posts
    66
    Thanks
    0
    Thanked 0 Times in 0 Posts
    In light of opinion here and advice from Google etc, robots.txt is out and meta tags are in

  • #13
    $object->toCD-R(LP); vinyl-junkie's Avatar
    Join Date
    Jun 2003
    Posts
    3,100
    Thanks
    2
    Thanked 23 Times in 23 Posts
    Quote Originally Posted by Rocket Boy
    In light of opinion here and advice from Google etc, robots.txt is out and meta tags are in
    Bots can (and some do) ignore both.
    Music Around The World - Collecting tips, trade
    and want lists, album reviews, & more
    SNAP to it!

  • #14
    Senior Coder gsnedders's Avatar
    Join Date
    Jan 2004
    Posts
    2,340
    Thanks
    1
    Thanked 7 Times in 7 Posts
    and meta tags are more often ignored than robots.txt

  • #15
    Regular Coder
    Join Date
    Oct 2005
    Location
    NYgamers.net
    Posts
    108
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Bottom Line is turn off directory browsing rename the files and don't like to those files from other web pages. If they are listed they should disappear in a few weeks.


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •