web crawlers and bots

**Steven Ragatz** · Feb-10-2003, 04:04 AM

From the link you posted:

Q: How can I completely exclude TurnitinBot from my site?
To exclude TurnitinBot from all or portions of your site all you have to to do is create a file called robots.txt and put it in the top most directory of your web site.
Below is an example of a robots.txt file which exludes ONLY our robot from a portion or all of your site.

#This is an example robots.txt file
User-agent: TurnitinBot
Disallow: /hide/ #Will disallow any url starting with /hide/

#This is an example robots.txt file
User-agent: TurnitinBot
Disallow: / #Will disallow all urls on your site

Another alternative is to contact us with your domain name and all its aliases (for instance www.somewhere.com, www2.somewhere.com, somewhere.com, etc.) and we'll add them to our blacklist of sites to avoid.

Steve

**le pire** · Feb-10-2003, 11:22 AM

Steve,

Yeah I saw that, I was wondering if there was some way to keep other bots out-- bots that may not leave a calling card like "turnitin.com" and "nameprotect.com".

I was wondering also if anyone knew of other kinds of bots and if they pose any kind of threat...

étienne

**Jim** · Feb-10-2003, 04:59 PM

You can put a line of code in your <head> meta tags... check out this page:

404 Not Found

http://www.robotstxt.org/wc/meta-user.html

This will keep all robots from indexing your site, but that includes search engine robot, too. It's up to you.

Jim

web crawlers and bots

web crawlers and bots

Comment

Comment

Comment