Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 4 of 4
Thread: robots txt file
05-14-2011, 02:32 PM #1
- Join Date
- Jul 2009
- Thanked 0 Times in 0 Posts
robots txt file
I placed the contents of the old version of my site into a folder under my site. I have no link whatsoever to that folder but still search bots might enter there i guess...
How can I prevent search bots to enter to that folder? a robot.txt file or something? what do i write in that file?
the reason is i dont want to be penalized by search engines for duplicate content
05-14-2011, 04:08 PM #2
You need to have something like this:
The "User-agent: *" means this section applies to all robots. The "Disallow: /yourfoldername" tells the robot that it should not visit any pages on the site within that folder.Code:User-agent: * Disallow: /yourfoldername
05-14-2011, 04:19 PM #3
- Join Date
- Mar 2007
- Thanked 2,224 Times in 2,211 Posts
It'd better to protect that folder by an HTTP authentication, so that you can prevent users also from any "unauthorised" access. (You host's control panel might have an easy interface to add this)
The Dream is not what you see in sleep; Dream is the thing which doesn't let you sleep. --(Dr. APJ. Abdul Kalam)
05-14-2011, 04:28 PM #4