|
|
|
Five FAQ about Google PageRank
1. What is PageRank and why should I care about it? PageRank is a formula that assigns a value to every page in the Google index. Google displays search results based on an algorithm which includes the value of PageRank. So the higher your site's...
How to Sell Your Product on the Forums?
1. Your signature file:
Most of the forums allow you to carry a 4 - 6 lines of signature files with your name. So these files appears below your name when you answer a forum thread.
Most people advertise about their product. Don't do that....
In Mourning for the Google Cash System
Do you remember when you could pick an affiliate program, hustle up a few hundred keywords toss it all into your Adwords account & wait for the commissions. No me neither! I was busy doing something else during those glory days. If it ever was that...
Search Engine Robots - How They Work, What They Do (Part II)
Search Engine Robots - How They Work, What They Do (Part II) Why Isn't My Website In The Search Engine? If your site isn't found in the search engines, it is probably because the robots couldn't deal with it. It could be something as simple as...
Shopping Carts For The Faint Of Heart
The chief criteria for judging an ecommerce shopping cart are the number of credit card processors and shipping services it supports, and the number of people that support *it*. Why? Because credit card processors and shipping services mutate all...
|
|
| |
|
|
|
|
|
|
Search Engine Robots - How They Work, What They Do (Part I)
Automated search engine robots, sometimes called "spiders" or "crawlers", are
the seekers of web pages. How do they work? What is it they really do? Why are
they important?
You'd think with all the fuss about indexing web pages to add to search engine
databases, that robots would be great and powerful beings. Wrong. Search engine
robots have only basic functionality like that of early browsers in terms of
what they can understand in a web page. Like early browsers, robots just can't
do certain things. Robots don't understand frames, Flash movies, images or
JavaScript. They can't enter password protected areas and they can't click all
those buttons you have on your website. They can be stopped cold while indexing
a dynamically generated URL and slowed to a stop with JavaScript navigation.
How Do Search Engine Robots Work?
Think of search engine robots as automated data retrieval programs, traveling
the web to find information and links.
When you submit a web page to a search engine at the "Submit a URL" page, the
new URL is added to the robot's queue of websites to visit on its next foray out
onto the web. Even if you don't directly submit a page, many robots will find
your site because of links from other sites that point back to yours. This is
one of the reasons why it is important to build your link popularity and to get
links from other topical sites back to yours.
When arriving at your website, the automated robots first check to see if you
have a robots.txt file. This file is used to tell robots which areas of your
site are off-limits to them. Typically these may be directories containing only
binaries or other files the robot doesn't need to concern itself with.
Robots collect links from each page they visit, and later follow those links
through to other pages. In this way, they essentially follow the links from one
page to another. The entire World Wide Web is made up of links, the original
idea being that you could follow links from one place to another. This is how
robots get around.
The "smarts" about indexing pages online comes from the search engine engineers,
who devise the methods used to evaluate the information the search engine robots
retrieve. When introduced into the search engine database, the information is
available for searchers querying the search engine. When a search engine user
enters their query into the search engine, there are a number of quick
calculations done to make sure that the search engine presents just the right
set of results to give their visitor the most relevant response to their query.
You can see which pages on your site the search engine robots have visited by
looking at your server logs or the results from your log statistics program.
Identifying the robots will show you when they visited your website, which pages
they visited and how often they visit. Some robots are readily identifiable by
their user agent names, like Google's "Googlebot"; others are bit more obscure,
like Inktomi's "Slurp". Still other robots may be listed in your logs that you
cannot readily identify; some of them may even appear to be human-powered
browsers.
Along with identifying individual robots and counting the number of their
visits, the statistics can also show you aggressive bandwidth-grabbing robots or
robots you may not want visiting your website. In the resources section of the
end of this article, you will find sites that list names and IP addresses of
search engine robots to help you identify them.
How Do They Read The Pages On Your Website?
When the search engine robot visits your page, it looks at the visible text on
the page, the content of the various tags in your page's source code (title tag,
meta tags, etc.), and the hyperlinks on your page. From the words and the links
that the robot finds, the search engine decides what your page is about. There
are many factors used to figure out what "matters" and each search engine has
its own algorithm in order to evaluate and process the information. Depending on
how the robot is set up through the search engine, the information is indexed
and then delivered to the search engine's database.
The information delivered to the databases then becomes part of the search
engine and directory ranking process. When the search engine visitor submits
their query, the search engine digs through its database to give the final
listing that is displayed on the results page.
The search engine databases update at varying times. Once you are in the search
engine databases, the robots keep visiting you periodically, to pick up any
changes to your pages, and to make sure they have the latest info. The number of
times you are visited depends on how the search engine sets up its visits, which
can vary per search engine.
Sometimes visiting robots are unable to access the website they are visiting. If
your site is down, or you are experiencing huge amounts of traffic, the robot
may not be able to access your site. When this happens, the website may not be
re-indexed, depending on the frequency of the robot visits to your website. In
most cases, robots that cannot access your pages will try again later, hoping
that your site will be accessible then.
Resources
SpiderSpotting - Search Engine Watch
http://searchenginewatch.com/webmasters/spiders.html
Robotstxt.org
List of robots and protocols for setting up a robots.txt file.
http://www.robotstxt.org/
Spider-Food
Tutorials, forums and articles about Search Engine spiders and Search Engine
Marketing.
http://spider-food.net/
Spiderhunter.com
Articles and resources about tracking Search Engine spiders.
http://www.spiderhunter.com/
Sim Spider Search Engine Robot Simulator
Search Engine World has a spider that simulates what the Search Engine robots
read from your website.
http://www.searchengineworld.com/cgi-bin/sim_spider.cgi
About the Author
Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing (www.searchinnovation.com), a Search Engine Promotion company serving small businesses. She has specialized in search engine optimization since 1998, including three years as the Search Engine Specialist for O'Reilly & Associates, a technical book publishing company.
|
|
|
|
|
| Gentoo Forums :: Index |
| Discussion forums for topics around the distribution, Linux applications, installation, administration, portage, frequently asked questions. |
| forums.gentoo.org |
|
| Readers' Opinions - New York Times |
| Books Reading Group. Readers are discussing “All Aunt Hagar’s Children,” Edward P. Jones’s new short story collection. More Books Forums ... |
| www.nytimes.com |
|
| Ubuntu Forums |
| This is a discussion help and support forum for Ubuntu Linux. |
| ubuntuforums.org |
|
| Joomla! Community Forum - Index |
| Joomla! Forum - The Joomla Community. Support, discussion and interaction with the Core Team. |
| forum.joomla.org |
|
| Internet forum - Wikipedia, the free encyclopedia |
| A typical Internet forum discussion, with common elements such as emoticons, ... Like other forms of online communication, Internet forums are home to many ... |
| en.wikipedia.org |
|
| Java Technology Forums |
| Online discussion forums offering help for users in solving issues. Includes FAQs, and links to other Sun technical databases. |
| supportforum.sun.com |
|
| Adobe - Forums |
| Share questions, suggestions, and information about Adobe graphics products. |
| www.adobe.com |
|
| WoW Forums -> Forum Index |
| Read our Introduction to the Forums. This includes information about getting ... blizzard provides the World of Warcraft community forums for its players to ... |
| forums.worldofwarcraft.com |
|
| MySQL AB :: MySQL Forums |
| Forum to discuss the sample database included with MySQL ... The main discussion forum for MySQL. Please use specialized forums for specific topics. ... |
| forums.mysql.com |
|
| SitePoint Forums: Resources, Design, HTML, CSS, PHP, ASP, MySQL ... |
| Webmaster Forums for Design, CSS, XHTML, PHP, Ruby, JavaScript, Affiliate Marketing, ASP.NET, Promotion. |
| www.sitepoint.com |
|
| Delphi Forums |
| Offers free and paid forum hosting. The paid offer is ad-free. |
| www.delphiforums.com |
|
| Home - MozillaZine Forums |
| Search the forums for others who may be having the same issues you are. You can also browse the sticky'd posts at the top of the forum for the product ... |
| forums.mozillazine.org |
|
| Web Hosting, Free Web Hosting and Web Tools from Bravenet.com. |
| Maintain a direct mailing list. Message Forum Host your own online discussions. Vote Caster Perform a multi-question poll. Web Poll Perform a quick poll. ... |
| www.bravenet.com |
|
| Forums | NewsTalk |
| The Detroit News Online does not tolerate offensive language in its forums. The News will kill offensive posts. Participants can and will be denied access ... |
| info.detnews.com |
|
| osCommerce Community Support Forums |
| Official community support forums for the osCommerce project. |
| forums.oscommerce.com |
|
| The GardenWeb Forums - GardenWeb |
| The GardenWeb Forums comprise the largest community of gardeners on the Internet. ... If this is your first visit to the forums, you might want to visit our ... |
| forums.gardenweb.com |
|
| AnandTech |
| Is server configuration a pain? DSL or Cable Modem giving you fits? Get your questions answered and your topics discussed in the AnandTech Networking Forum. ... |
| forums.anandtech.com |
|
| Chronicle Forums - Index |
| Chronicle Forums - Index. ... Talk online about graduate-school life in our new forum. ... Forum Stats, 220919 Posts in 17109 Topics by 3306 Members. ... |
| chronicle.com |
|
| Forums | drupal.org |
| Try searching the forums first or a specific project's bug reports. Pre installation questions. Is Drupal a viable solution for my website? ... |
| drupal.org |
|
| Dev Shed Forums - Open Source web development |
| Dev Shed Forums - over 195246 members on our Dev Shed forums. |
| forums.devshed.com |
|
|