Page 1 of 1

Robots.txt

Posted: Sun Mar 01, 2015 11:19 pm
by dzcadii
Simple robots.txt example
* means all browsers
Disallow tells the robots where they can not go (should not index it)
All tells the robots where they can go (will index it)


To allow complete access to your site/server

Code: Select all

User-agent: *
Disallow:
To disallow all robots from the server

Code: Select all

User-agent: *
Disallow: /
To allow all but one directory
*Note: If you want to disallow more than one directory you must place it on a separate line

Code: Select all

User-Agent: *
Disallow: /images/
Allow: /
To allow all but two directories
*Note: If you want to disallow more than one directory you must place it on a separate line

Code: Select all

User-Agent: *
Disallow: /images/
Disallow: /cgi-bin/
Allow: /
To stop only one robot

Code: Select all

User-agent: Google
Disallow: /
To allow only one robot

Code: Select all

User-agent: Google
Disallow:
User-agent: *
Disallow: /
robots.txt for phpbb
provided by: http://www.askapache.com/seo/seo-with-r ... -for-phpbb

Code: Select all

User-agent: *
Disallow: /cgi-bin/
Disallow: /phpbb/admin/
Disallow: /phpbb/cache/
Disallow: /phpbb/db/
Disallow: /phpbb/images/
Disallow: /phpbb/includes/
Disallow: /phpbb/language/
Disallow: /phpbb/templates/
Disallow: /phpbb/faq.php
Disallow: /phpbb/groupcp.php
Disallow: /phpbb/login.php
Disallow: /phpbb/memberlist.php
Disallow: /phpbb/modcp.php
Disallow: /phpbb/posting.php
Disallow: /phpbb/privmsg.php
Disallow: /phpbb/profile.php
Disallow: /phpbb/search.php
Disallow: /phpbb/viewonline.php
 
User-agent: Googlebot
# disallow files ending with these extensions
Disallow: /*.inc$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$

# disallow all files with? in url
Disallow: *mark=*
Disallow: *view=*

# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow:
Allow: /*
 
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

upload the robots.txt to the root directory on your server

Below is a file with some common robots in it. Please give credit to http://www.robotstxt.org/
http://www.robotstxt.org/robotstxt.html and http://www.askapache.com/seo/seo-with-robotstxt.html
robot_DB_fromRobotOrg.zip
Below is a file with some common robots in it. Please give credit to http://www.robotstxt.org/
(61.79 KiB) Downloaded 91 times