|
Author: Sandra Waggett Article source: http://searchenginewatch.com/. Used with author's permission.
Search engines index millions of web sites to generate the search results they return for key words. They do this using "spiders".
Most search engines have their own spider that crawls around the web looking for web pages. Spiders are also known as "robots" because they are simply tiny little programs that run automatically, looking for web pages and recursively traveling through the embedded text links to index them.
Most robots look for a robots.txt file in the top-level directory of your web site, also known as the "root" where your home page is located on the web server.
The robots.txt file is a simple text file created in a basic text editor, like Notepad. It allows you to control what the spider is allowed to access and what it is not allowed to access or index.
The format of the basic robots.txt file is pretty simple:
User-Agent: [Spider Name]
Disallow: [File Name]
For example, to allow ALL robots complete access to your web site, your robots.txt file will look like this:
User-agent: *
Disallow:
The asterisk is a "wild card" character that represents ALL robots. Leaving the Disallow line blank indicates to the robots, that nothing on the site is disallowed.
The next example bars all robots from the cgi-bin (where your scripts are typically located), images directories, and the portfolio directories:
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /portfolio/
Note: You should use a separate Disallow line for each directory or individual file.
In this example, you may wonder why you would want to disallow a robot from indexing your portfolio directory.
If you are a photographer and you have thumbnail images on a portfolio page that link to enlargement pages launched in a pop-up window, you may not want those pop-up pages indexed. These are called "dead-end" or "orphaned" pages because only the enlarged image appears on the page with no contact info or menu links back to the main site. If the visitor entered your site on one of these pages, they would have nowhere to go and no way to contact you.
For a live example, check out www.AnJPhotography.com and look at her wedding portfolio. When you click on an image, it opens in a new window. The page in the new window is a "dead-end" page. A robots.txt file can keep search engines from indexing these "dead" pages so you don't leave site visitors stranded.
This example keeps googlebot (the Google spider) from getting at the
private.htm file:
User-agent: googlebot
Disallow: private.htm
When you create your robots.txt file it is extremely important that you use a basic text editor (like Notepad) and NOT a word processing application like Microsoft Word. Applications like Microsoft Word can insert hidden characters that may make your robots.txt file unreadable. After you post your robots.txt file to the web server, you can validate it to make sure it is properly formatted. There are several free validators on the web. Here is one:
http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
There are several advantages and some disadvantages of having the robots.txt file in your root directory. Protocol requires that all search engine robots start indexing your web site with the robots.txt file. This is the default entry point for robots if the file is present. Major search engines will never violate the Standard for Robots Exclusion. This is the primary reason it should be there. Beyond that, it can help with your search engine rankings when used correctly, and it can keep dead pages on your web site from being indexed. The primary disadvantage is that the robots.txt file may be viewed by nefarious individuals on the web, so you never want to use the robots.txt file to try to hide sensitive pages or directories on your web site (like passwords or private information).
For more information about the robots.txt file and complete list of robots, visit the following web site: http://www.robotstxt.org/wc/robots.html Sandra Waggett is the founder and principal designer of MSW Interactive Designs LLC (MSW-ID) major products and websites. MSW-ID provides custom website design, hosting, ecommerce and online marketing solutions to nearly 400 small business clients nationwide. MSW-ID helps small business professionals achieve an effective Internet presence.
Prior to founding MSW Interactive Designs LLC, she spent nearly 5 years working as a Senior Engineer for BAE Systems on the Lockheed Martin Mission Systems Team in Colorado Springs, CO.
While with BAE, she was the training lead for the proposal phase of the Integrated Space Command and Control (ISC2) program. In this role, she authored the 10 year training plan for the proposal and developed web-based training prototypes for presentation to to the Government decision makers.
Sandy earned her Master of Arts of degree from the University of CO, Colorado Springs, in Curriculum and Instruction, Corporate Track. Her specialties include web design, interface design, instructional design, and computer-based training development.
When You Attract Negative People, Be Thankful
Master Practitioner in the Laws of Attraction, Catherine
Franz, shares her thoughts and story on why it is important
to embrace the negative people you attract...
Whats a Professional Sales Manager?
I was in the depths of a major depression. As a third year salesperson with a good company, I was doing well, and was on my way to becoming the top salesperson ...
Getting To The Core Of Your Golf Swing
At some point, almost all of us struggle with efforts to improve our golf game. Many will try videos, books, hours with a teaching pro, the newest club, and sw...
Prepaid Cell Phone Plans – Are They For You?
Most cell phone users are on plans that offer a fixed number of minutes per month. An alternative to this -- that's becoming increasingly popular -- is known as...
6 Ways To Prevent Identity Theft
These tips offer you protection against identity theft so you won't become a new victim of this terrible crime.
Health Savings Accounts (HSA): Do They Vary From State to State?
The Health Savings Account (HSA) is becoming a popular tool to reduce insurance payments and even to help us be healthier. But are they available in all states?...
Home Buying Checklist – Windows
If you are in the market to purchase a home, it can be easy to fall in love at first sight. Do so at your own risk. In this article, I provide a checklist of wi...
What is Prenatal Intelligence?
Can expectant parents increase their unborn child's intelligence? Prenatal intelligence is a hot topic, and one you should be interested in.
Facial Skin Care Tips For Your Type Of Skin
Everywhere you see companies are selling facial skin care products promising to reduce aging lines and give you a younger appearance. All the information can be...
How to Waste Money and Annoy Potential Customers
Why do some companies bid on keywords for products they don't even sell? Or fail to provide the basic information people need to make the decision to buy? Here'...
Hows Your Wheel Of Life?
Do you experience stress in your life? Are you burned out? Is your life out of control?
Is Conflict Healthy in Marriage?
How can you disagree with your spouse and maintain a happy balance in your marriage? Find ways to keep your relationship strong with this insightful article.
Product Not Selling? This Could Be Your Lucky Day!
Are you having trouble selling your product online? No problem! Sit back as I share with you three simple and ingenious ways to pile on the profits and turn lem...
How To Buy a Used RV - Part 1
If you are on a tight budget and can't afford a new model RV you might consider buying a used RV. Here are a few suggestions for finding a used, affordable RV
Dog Flea Control Management: How To Prevent, Treat, And Kill Dog Fleas
Dog flea control and management requires an integrated approach. For effective treatment both the host animal and the environment must be treated at the same ti...
I Spy...Something Terribly Wrong (In Your Computer)
This really chapped my lips...
Relaxation Techniques to Revitalize Body and Mind
We all get to a stage where we've just had it. We need a break. Well if a holiday is out of the question, try these relaxation techniques to revitalize your bod...
Wedding Gifts - Something Old, Nothing New!
Forget the conventional, new and useful wedding gift. Thnk old but valuable!
Beluga Whale Hits the River
An inspirational 350-word essay about what a Beluga Whale, who recently got off-course and headed up the Delaware River in New Jersey, can teach us about our ow...
Escrow Accounts – Are You Paying Too Much?
Many people are paying more than they need to into their escrow accounts. A simple check of your own account can save you thousands in interest over the course ...
The Transformation Of Political Science And The Rise In Crime Rates
The current field of political sciences is dominated by a multitude of ideas that have never in its history featured so prominently in this discipline. The gene...
How To Pay Credit Card Debt Off
Here are the quickest and best ways for total credit card debt elimination.
Reselling Products with Resale Rights: is this Trend Here to Stay?
Is the resale rights phenominum here to stay, or is it a passing fad?
Do Cellular Phones Pose Health Risks?
You hear it in the media on a slow news day. Do cellular phones pose health risks? The available scientific reports ....
|
Print this page