|
Author: Michael Kralj Article source: http://www.macosx.com/. Used with author's permission.
Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!
The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.
Many search engines cannot view dynamically generated content properly, mainly created by programming languages, such as PHP or ASP. If you have an online store programmed in your hosting account, and it is in a seperate directory, you would be wise to block out the spiders from this directory so it only finds relevant information.
The robots.txt file should be placed in the directory where your main files for your hosting are located. So you would be advised to create a blank text file, and save it as robots.txt, and then upload it to your web hosting to the same directory your index.htm file is located.
Here is examples of the use of the robots.txt file:
To block out a directory in a robots.txt file, such as a subdirectory for your online store called /store/ you would do the following:
Disallow: /store/
Another example to block out your stats directory:
Disallow: /stats/
You may also want to disallow individual files that you do not want searched by the search engines. For example you dont want search.php to be parsed by the Search Engines. To do this you type in the following on its own line:
Disallow: /search.php
Following the rules outlined and creating the robots.txt file, you will keep search engine spiders out of unwanted files and directories, and letting them go through the important files to see what your web site is all about! Michael Kralj is owner of Emenki Web Solutions and Domains at Retail. Emenki Web Solutions are web site designers and programmers based in Hamilton, Ontario, providing businesses with an informative and strategic approach to establishing an online presence on the web.
Please visit Emenki Web Solutions & Web Design in Hamilton, Ontario on the web http://www.emenki.com
Please visit Domains at Retail - Cheap Domain Name Registration on the web: http://www.domainsatretail.com
Fast Weight Loss Using Herbs And Spices
I wonder how much it is generally known that the benefits of
using herbs and spices can speed up your weight loss efforts,
by quite a considerable amount actu...
Retire Dollar Smart
Jim Miller is a registered investment advisor. This means that he is not beholden to a particular brokerage or financial institution. As such, he does not cha...
Fast Track Out of Debt
You go to the mail box and scan - a couple fliers (nah), your magazine subscription (yes!) and bills (groan). Every month the bills show up and as you sigh and ...
Solutions to Rosacea & Acne Rosacea - Treatments That Eradicate the Redness
This article examines the causes of acne rosacea and rosacea, and gives lifestyle advice and alternative treatment information for those who would like to treat...
Time For A Quickie!
Technique! We seem to have become obsessed by it! There are probably thousands of books offering detailed explanations of ways to make love to your partner. But...
What is Debt Management?
Debt management is a topic most people will have to deal with at some point. Debt is acquired by not living within your means. Living within your means is sim...
Tiffany Lamps - The Artistry of Dale Tiffany
Dale Tiffany is the world's leading designer and manufacturer of quality reproductions and Tiffany style lamps. Based on the original colored art glass designs ...
A Simple White Hat Technique To Get Indexed By Google
Using Google Sitemaps to get your site indexed better...
Small Business Secrets: Self-Confidence Can Be Arranged
You can arrange to have the self-confidence you need by arranging support structures and encouragement and designing situations that feed you the confidence you...
Real Estate Investors Kick Away $1,000
Real estate investors who buy property "subject to" existing financing must properly plan on collect impound funds when they sell or they will be giving away mo...
How to Listen on the Web
Listening to customers is always important. But how do you listen to them on the web? Read through this article to disover techniques that will allow you to be ...
Aerospace Design Thoughts, The Bionic Man, and The Future of Design
Where do you want to go today? Really, and what do you want to fly in?
Product Marketing for Internet Based Home Businesses
Product marketing and how to increase traffic to their web pages in order to increase sales.
Choosing A Network Marketing Opportunity - 5 Red Flags
Although network marketing offers the potential for long term residual profits on your original efforts, it is important to take the time initially to choose th...
Do You Really Value What You Wish For?
I am sitting in a shopping mall, waiting for my wife to finish her Christmas shopping. Next to me is a rather pathetic-looking fountain, which contains hundreds...
Eugenics and the Future of the Human Species
"It is clear that modern medicine has created a serious dilemma ... In the past, there were many children who never survived - they succumbed to various disease...
Do You Know These Surefire Ways to Attain Moneymaking Success?
Aditudes and beliefs the assist or limit your success.
Best Vitamin Supplements - 3 Criteria a Best Vitamin Supplement Should Meet
The best vitamin supplements must meet high standards in three critical areas. Quality in all three areas is vital not only for health benefits but for safety ...
Canadian Physicians on Slippery Slope of Burnout
In a recent CMA survey of 2251 physicians, over 45% were found to be in an advanced state of burnout. If this reflects the seriousness of the problem in the gen...
Autoresponders An Important Part Of Internet Marketing?
Setting up and using email autoresponders on your site is an easy way to make repeated contact with your visitors. Studies have shown that more than 60% of a si...
How To Choose Great Bathroom Light Fixtures
Everyone has different tastes; that's why there are so many different types of bathroom décor. But one thing is constant despite your décor: the need for clear,...
Can Money Buy You Happiness?
Many people will answer that question with a negative response.
But isn't that just a "knee jerk" reaction?
Chemical Peels
Three levels of chemical peels exist to correct different skin conditions. Some have dramatic effects eliminating wrinkles and removing precancerous skin lesion...
Network Marketing, Multi Million Dollar Business or Just a Spam?
Network marketing or sometimes known as Multi Level Marketing (MLM) is actually a good business. The market for this business is so big and the prospect is good...
|
Print this page