|
Author: Michael Hart Article source: http://www.selfseo.com/. Used with author's permission.
Making effective use of the Internet is increasingly about creating better and more intelligent applications and search engines. Here is a brief introduction into how search engines work:
01) Define the corpus, search space/data;
02) Separate the corpus into documents;
03) Generate features for each document;
04) Generate a representation of each document;
05) Study the feature/vector space;
06) Cluster documents;
07) Reduce dimensionality;
08) Accept input Queries;
09) Find the cosine angles against the query vector;
10) Find the sought vector column;
11) Output results to user in some way;
Each document in a corpus (database) is described by a set of keywords called index terms. We assign weights to index terms according to their relevance (frequency of occurrence for instance), this is how we go about creating the index, that we can then search.
Corpus preparation:
Web pages of interest are analysed and cleaned by removing hypertext tags or any other hyper language; Pages are then broken down into documents where each document is scanned through searching for words/terms of interest: those which make a document unique, not standard words.
Extract terms of interest:
Bear in mind that terms of interest must be invariant, that is be characteristic of a document, not generic and easy to find in any corpus/document. The idea is to find a signature per document.
Build term-by-document matrix:
The search space is defined by N dimensions where the chosen terms/features of a document is a point in the N term space, this allows conceptual/semantic searches.
Each document becomes a column vector, each row represents a term. Each row identifies the frequency of a term across the analysed corpus, at first we simply build the matrix by counting the terms for each document.
Compress the matrix:
There are two basic techniques/methods, Compress Row Storage
(Scans matrix row by row) and Compress Column Storage (Scans matrix column by column) Both use three arrays.
Normalis the matrix:
Normalisation implies transforming column vectors to unit vectors: i.e. vectors of unit length
Unit document vectors contain frequency of terms; the normalisation is applied because the semantic content of a document is generally determined the relative frequency of terms.
Singular Value Decomposition:
This simplifies a symmetric matrix into three matrices
Two are identical and represent the eigenvectors: the new dimensions. The third is diagonal and represents the eigenvalues, that is the spread of the corpus along these new dimensions.
A geometric interpretation:
The corpus is first formated, stemmed and is then stored in a compact term-by-document matrix. Each column of such matrix is then normalised to produce the likelihood of a term across the corpus, or, equivalently, the frequency of terms in a document.
The term-by-document matrix is then decomposed to calculate eigen values and vectors. Eigen vectors represent a new Cartesian coordinate frame spanning the same search space, BUT, they indicate the most important dimenions/axis along which documents mainly lie. Eigen value do quantify the spread of documents along these new axes/eigen vectors.
Queries:
Queries must be based on defined features/terms within the term-by-document matrix, matching in a vector space such as this is implemented by multiplying the query vector against the terms by document matrix,ie matching a query vector q against the documents of the matrix. © I am the website administrator of the Wandle industrial museum (http://www.wandle.org). Established in 1983 by local people determined to ensure that the history of the valley was no longer neglected but enhanced awareness its heritage for the use and benefits of the community.
Start a New Career
After coaching hundreds of people on how to break free from the corporate world, I have created 7 fundamental practices to help you successfully start a new car...
Bedroom Furniture
Some ideas on choosing and buying bedroom furniture...
42 Deadly Ad Copy Sins That Ive Made
Writing good ad copy for your online business, some of the do's and don'ts that I have discovered in 5 years of being online. I hope they help you.
Marketing-Minded Financial Planners, Join Your Professional Organization to Get Free Publicity
If you want to take advantage of a great way to get free publicity, you marketing-minding financial professionals will join an association.
Self Improvement and Self Growth
The subconscious mind is one of the major keys to self-improvement and self-growth. By changing the contents of the subconscious mind you change your habits, be...
Are You A Leader in the Chrysalis?
Are you a leader in the chrysalis? What of and when and how? Do you have the skills and most importantly, what's your measure?
Greasing the Path to Success: Finding the Confidence to Step Up to Key Moments
Whether it's making a prospecting call to a promising business contact, giving an informal "elevator speech" to a networking group, or attempting to close a tr...
Conquer Stress and Anxiety Naturally With This Safe and Effective Method
Rhodiola Rosea is the latest natural remedy to join the arsenal
of natural anxiety and stress reducers.
Payday Loans And Pawnshops: Comparing Two Popular Ways Of Getting Cash Quickly
Are payday loans really a ripoff? Comparing payday loans with the age old practice of pawnbroking
Stop Your Dog From Pulling You Down The Street
Does your dog pull you down the street? Find out how to stop that behavior fast.
Find Your Russian Wife By Yourself: Briefly About Ways
Let's say from the beginning that under the title "Russian Woman" we will keep in mind all the ladies, natives from CIS countries, former territory of the Sovie...
Wedding Favors: A Brief History
For many centuries wedding favors have been a part of traditional wedding ceremonies, across a wide range of cultures. Initially wedding favors were an extravag...
Good Credit -- Essential for the Prospective Home Buyer
Attaining good credit is a necessary step to getting a quality home mortgage loan. Here are some tips that will keep you out of trouble, and boost your credit ...
Turn Your Email Sign-Off Into Results: Seven-Step Checklist for Success
Is your email signature talking to you - or to your customers? In this article, learn how making a change or two to your sign-off can make a big change in your...
Orlando Vacation Rentals - Close To the Action
By renting a privately owned Orlando vacation home you are on the first step to a fantastic vacation. Not only will you have plenty of space and privacy to enjo...
Super-Affiliates: What Are They, How Do I Find Them, and Why Theyre Your Ticket to Online Success!
This article explains what a Super-Affiliate IS, their Role, How to find them and WHY they're your ticket to online Success!
How To Avoid Post Partum Depression
Here are some tips that can help prevent or ease the symptoms of postpartum depression.
Winning Websites
More and more potential customers will look at your website
to find out more about your product or service or even to
place an order. Make sure it grabs attenti...
The Three 8 Mile Battles
The three 8 Mile battles close the 8 Mile movie. To gain some respect and some regognition in the local Detroit rap scene, Jimmy Smith aka Bunny Rabbit has to b...
Holistic Medicine
Holistic Medicine involves natural treatment options like holistic nutrition, herbology, environmental medicine, energy work, mind-body medicine, anatomy, and m...
Network Marketing? What Is That?
Nowadays, network marketing is very popular in our society. Almost every body knows about that. But, part of them are skeptical about network working. They thou...
The Birth of Incident Response: The Story of the First Internet Worm
When was the first Internet worm launched? What did it do? What happened to the author? (Hint: he became a multi-millionaire) The answers will surprise you!
A Personal Loan And Your Rights
A Personal Loan And Your Rights - You can use a personal loan for many different things including, but not limited to, paying off bills, taking a vacation, buyi...
Top Five Tips for Marketing that Gets Results
Five crucial tips. Three key elements. Four critical marketing questions. Here is what is needed to get reults in today's world...
|
Print this page