Subscribe to SEM Basics using an RSS Reader or by Email
Click here to Sign Up to SEM Basics FREE Link Building Course!
Archive for August, 2006
How to design a search engine friendly website.
The content on this page is based on SearchEngineNews.com’s e-book Winning The Search Engine Wars.
To learn more about Search Engine News, click here.
How to design a search engine friendly website
The first step to properly optimizing your site for the search engines is learning how to design a search engine friendly website. Simply put, a search engine friendly web site means that you make it as easy as possible for the search engines to download and process your web pages. After all, your optimization efforts mean little if the search engines can’t access and/or process your website. But creating a search engine friendly site is more than a preventative measure, it also can help improve your rankings in the search engine, as will become clear in this tutorial. There are, in essence, six different steps you can take to creating a search engine friendly website. We will outline each of those steps here, with later tutorials going into greater detail where need be.
NOTE: some of the material covered in this section is rather technical. You may need to consult your programmer to properly implement some of the ideas mentioned. Nonetheless, we will make every effort possible to keep this tutorial as non-technical as possible.
Six steps to designing a search engine friendly web site
- Use Well Formed, W3C Validated HTML Code
- Avoid (or at least limit) the use of flash technology, JavaScript & fames
- Proper use of URLs
- Proper website and directory structure
- Limiting the size of your web pages
- Proper use of the robots.txt file
Let’s understand each of these concerns a bit better.
Using Well Formed, W3C Validated HTML Code
The first step towards creating your search engine friendly web site is to make sure that your HTML code is correct and well-formed. Make sure that you have closed all tags which need to be closed and that you haven’t used source code which could “confuse†the search engine spider (such as old and outdated or which is proprietary to a
single browser or code which is so new that it has not yet been recognized by most search engine spiders).
To learn more about using proper code see our SEO Friendly HTML Code Tutorial –click here to access the tutorial.
Use of Flash Technology, JavaScript & Frames
While it is possible to optimize Flash, JavaScript and/or Frames pages, it is significantly more difficult to do so. As such, if you want to design a search engine friendly website it is best to avoid using these technologies if possible. With that said, if you feel that you have to use them, or you are optimizing a site which is already built around these technologies, then it is important to know what to do.
If you find yourself in this situation then you should check out our Flash, JavaScript, & Frames Tutorial. — click here to access the tutorial.
Proper use of URLs
When designing a search engine friendly site, you want to make sure that you use URLs that the search engine spiders can easily follow. Search engine spiders find and process web pages by following links from one site to another. This means that if your website’s URLs make it difficult for the search engine spider to download and/or process the pages found on your website it is possible that your web pages may never get processed. And, of course, a page which is not processed is also a page which will not show up in the search engine results. In other words, if you don’t use a search engine friendly URL you may find that all of your optimization efforts are for naught given that the search engines won’t even know about your web pages. On the other hand, the right URL may actually help your rankings in the search engine.
To learn more about the proper use of URLs see our SEO Friendly URLs Tutorial — click here to access the tutorial.
Note: you may want to first read our tutorial Introduction to URLs tutorial, which explains all of the various "sections" of a URL. You can read this tutorial by clicking here.
Website and Directory Structure
In general, the deeper a page is on your website (i.e,. the more links that a visitor has to click to get to a particular page), the harder it is for it to get indexed by the search engines. Unless your site happens to be exceptionally popular with a lot of incoming links you can assume that the search engines are not going to dig that deeply down into your site. Because of this problem many sites employ a sitemap which lists all of the pages on one’s web site. When done properly, this allows you to logically organize the material on your website while simultaneously ensuring that the search engines are able to reach (and thus process) all of the pages on your site.
To learn more about creating a site map for the search engines see our Site Map Tutorial. You can read this tutorial by clicking here.
Limiting the size of your web pages
Each of the major search engines has a general rule of how large a page they will index. This amount varies from search engine to search engine, and even from site to site (more popular sites, not surprisingly, have a larger limit than less popular sites). In general, it is best to limit the size of your pages to 100k or less (not including picture size, which does not effect how much text gets indexed). This limit (i.e., 100k) is based upon the fact that this may still be Google’s limit. Even though there have been some indications that Google may have raised that limit, it is best to be safe rather than sorry. Also, it is easier to optimize
smaller pages and when you split up your larger pages into smaller pages you provide the search engines with more opportunities to find and fully index your web pages.
Proper use of Robots.txt file
Robots.txt files are used to tell search engine spiders which pages not to index. The advantage of using a Robots.txt is that it focuses the search engines on indexing your “important†customer-development, product, and/or sales pages. Remember that the search engine spiders function with limited time and resources when indexing sites, it would be best to use that time wisely. For example, there is no need for the search engines to index your shopping cart (can you think of any reason why you would want your shopping cart checkout pages to show up in the search engines?). Other pages that you don’t want indexed by the search engines are directories that contain images or otherwise sensitive company data (for instance, do you want the search engine spiders to index your password files?). You will also want to use a Robots.txt file for anything that is in your cgi-bin folder.
Warning: don’t accidentally set up your robots.txt file in such a way as to prevent the search engine spiders from crawling your site. The following two lines of code are sufficient to prevent all of the major search engines from ever crawling your site:
User-agent: *
Disallow: /
There you have it, six important steps to help you create a search engine friendly website. If you find this information too technical, just concentrate on understanding the basics and make sure that your programmer understands what it is that he or she is suppose to do. While it is not crucial that you understand all of the details on this page, it is crucial that your programmer (or whoever it is that is building your site) does understand the information described above.
In the meantime, to learn more about creating a search engine friendly web site continue on to our next tutorial SEO Friendly HTML Code.
Next tutorial: SEO Friendly HTML Code.
This tutorial written by:
Moshe Morris
President of SEMBasics
Chief Research Analyst at Internet Marketing Initiative (www.internetmi.com)
The content on this page is based on SearchEngineNews.com’s e-book
Winning The Search Engine Wars.
To learn more about Search Engine News, click here.
How do the search engines determine relevance?
How do the search engines determine for any given search which websites to show, and how do they determine which order to show them in? As we mentioned in our previous tutorial, we are interested in these two questions because the answer will indicate to us what it is that we need to do in order that our site will show up in the search engines for relevant searches (and not just show up, but show up in the top spots also).
There are, in general, two factors which the search engines take into consideration when determining their search results. The first is the content of the actual website itself (in particular, the content of each and every page in and of itself). The second is the number and quality of links coming into the site in general, and each page on that site in particular. These two elements are generally known as on-page factors (i.e., the content of the site) and off-page factors. Let’s start with the on-page factors.
On-page factors
It is worth noting that for the most part, your web site never competes for a position in the search results. Only your individual pages compete for those coveted slots. This doesn’t mean that the overall design and structure of your site doesn’t have a role to play when it comes to optimizing your site. It has a very important role to play. Nonetheless, it is individual pages which the search engines results display and as such it is individual pages which compete to show up in those results. And, for the most part, each and every page on your site gets evaluated in-and-of itself (at least, it is helpful to look at it that way). As such, the search engines analyze the content on your page to try and determine what it is about. In particular, it looks at the text on your page and various elements in the code on your page to try and determine what that page is about. For instance, if a page has a great deal of content about fishing rods, and the phrase fishing rods appears in certain places in your HTML code then the search engine will label that page a fishing rod page. This doesn’t necessarily mean that that page will rank well for the term fishing rods, after all there may be many other pages which discuss fishing rods and have that term in the relevant places in the HTML code. What it does mean, though, is that the search engines will not consider this page for unrelated search terms such as as balloon rides. So, in essence, on-page factors tell the search engines what your page is about. It doesn’t, however, tell the search engines whether your page is the best source for that topic.
What this means is that, ultimately speaking, relevancy is not enough. There may be thousands, if not millions, of relevant pages for any given search insomuch as they relate in some way or other to the term being searched. What the search engines really want are the most relevant pages for any given search. The question is, then, what makes a site more relevant than another. The answer, for the most part, is links.
Off-page factors
The search engines look at links as a kind of vote. In theory, people link to sites that they are interested in, and any link that a page receives is an indication that some other site online finds that page worthwhile. In reality, links are much more complicated than this as we shall see in our Link Building Tutorials. For now, though, let it suffice to say that there are quality links, "bad" links and links which are somewhere in-between. The search engines goal (and they are getting increasingly good at this) is to determine how many quality, relevant links you have coming to your site in general, and each page in particular. A page which has a large number of high quality links from sites which relate to the topic of that page is considered to be a highly relevant page.
Of course, there are a number of factors which need to be clarified. For instance, what makes one link a quality link and not another? How do the search engines determine whether or not two sites are related so that a link from one site to another is considered more valuable? For now, though, we have enough information to continue with our tutorials. We know that the search engines determine relevancy based upon the content of our web pages and the number, quality, and relevancy of links coming to those pages. Our task now is to understand how we use that information to properly optimize our sites for the search engines. And that is the topic of our next tutorial.
Next tutorial: How to optimize your site for the search engines.
Previous tutorial: Taking the search engine point of view.
This tutorial written by:
Moshe Morris
President of SEMBasics
Chief Research Analyst at Internet Marketing Initiative (www.internetmi.com)
Taking the search engine point of view: why you want whatever the search engines want.
Let’s take a second and try and look at the internet from the search engines point of view. Why do the search engines provide us with this invaluable service of allowing us to search the internet? The answer, not surprisingly, is money. The more users that a search engine has, the more potential they have to make money. But how, and what does this have to do with search engine optimization?
The answer to the first question is simple enough, ads? Search engines make money by showing ads along side their search results. For instance, imagine that you did a search on Google for free cell phones. You would arrive at a page that looks like this:
On the left hand side of the screen there are what is known as the organic search results. These are the websites which Google thinks are the most relevant site for the term free cell phone (note: everything which we are about to say about Google also applies to the other major search engines: Yahoo, MSN, and Ask.com). Furthermore, Google does not take money for showing these results. They are displayed based on Google’s algorithm (a mathematical formula which Google uses to determine how to rank sites for any given search term). On the right hand side of the page are different results which also show up for the term free cell phone. These are ads, or more exactly Google AdWords ads, and the owner of that ad pays Google every time someone clicks on their ad (the amount varies according to various factors).
What is worth noting is that Google’s entire business plan revolves around people clicking on these ads. As such, the more people who use Google’s search engine, the more money Google will make (as more people will see Google’s ads with a certain percentage of those people clicking on them). Thus Google has a vested interested in providing the highest quality search engine that they possibly can (as do the other major search engines), for that is what drives people to their site. And the key ingredient to a quality search engine is relevance! After all, people are only interested in a search engine insomuch as it helps them find the results that they are looking for. And since that is what Google’s customers want, that is what Google wants. And since that is what Google wants, that is also what we want.
Here comes the answer to our second question. When we say that Google ranks the search results according to relevancy what we really mean is that Google has developed various criteria and methods for determining what is the most relevant site for any given search. What this means for us is that if we can discern what those criteria and/or methods are (Google doesn’t reveal them) then we can build our site accordingly for the terms that we want to rank well for. Put simply, Google sets the ranking rules. If we want to rank well then we best learn what those rules are and follow them. This, in a nutshell, is all that search engine optimization is about.
As simple as that may sound it’s actually a bit harder to do in real life. Particularly since the search engines are constantly trying to improve the results that they return. What that means is that the criteria and methods that the search engines use to rank sites are constantly changing. So not only is it important to know how it is that the search engines rank sites today, but it is equally important to get as clear a sense as possible as to how they plan to rank sites tomorrow. That way you can always be prepared (or at least try to be prepared) for whatever changes come tomorrow. Thus ensuring that your high rankings are as stable as can be.
Our first order, therefore, when it comes to optimizing our sites for the search engines is to understand as best as possible the criteria and methods that the search engines use to determine their search results. As such, that is the topic of the next tutorial.
Next tutorial: How do the search engines determine relevance?
Previous tutorial: What is search engine optimization?
This tutorial written by:
Moshe Morris
President of SEMBasics
Chief Research Analyst at Internet Marketing Initiative (www.internetmi.com)
What is search engine optimization?
Imagine that you sell cell phone and that at this very moment a potential customer is searching on Google for that very phrase. That customer will most likely reach a page like this:
There are three sections worth noting about on this page. the first are the three results shown in the upper left hand side of the page, which are highlighted in a light bluish background. Those are ads, which means that the owner of each ad pays Google every single time someone clicks on their ad (these ads are part of Google’s AdWords program and are generally known as pay-per-click ads or paid search).. Similarly, the results displayed on the right hand side of the page are also ads. Neither of these sections have anything to do with search engine optimization. Search engine optimization only deals with what are called organic (i.e., free) results, as shown in the results displayed on the left hand side of the page, below the three ads previously mentioned. In other words, the results displayed on the left hand side of the page are not there because someone paid Google to place them there, but rather because Google "decided" that they are the best pages on the Web to display for the query "cell phones". If you entered a different query you would see different results.
Now, with that said, let’s return to our imaginary cell phone site. Your goal, if you owned such a site, would be to have your web site show up in the top 10 - 20 search results for most (if not all) searches related to cell phones. Why? For starters because showing up in the organic (i.e., free) search results for the major search engines (Google, Yahoo, MSN, and Ask.com) can bring in large numbers, of targeted customers to your website for free! People generally do not scroll down beyond the top 10 - 20 search results for any given search. What this means is that if your site does not show up in the top 10 - 20 search results for relevant searches then your site is all but invisible to your potential customers. In other words, there is a tremendous amount of profitable business to be obtained by ranking well in the search engines for relevant search terms. This is why so many people spend a great deal of time, money, and effort trying to get their sites to rank well in the search engines for terms and phrases which relate to their products or services.
The obvious question, of course, is what do you have to do to rank well for those terms and phrases? The answer is that you have to optimize your website for the search engines. What that means in lay-man’s terms is that you have to figure out how it is that the search engines decide upon the order of their search results for any given search and then build your website accordingly. That’s what we mean by “optimization”, that we have built our site to perform well for given searches on the major search engines according to the rules which the search engines have already laid down. Thus, it should be clear that if we want to know how to best optimize our site for the search engines then we need to find answers to the following three questions:
- How do the major search engines determine the order of their search results?
- How can I take advantage of that knowledge so that my site will rank highly in those search engines
for relevant search terms? - Which terms and phrases (i.e., keywords) do my customers use when searching for my products and services?
Next tutorial: Taking the search engine point of view.
This tutorial written by:
Moshe Morris
President of SEMBasics
Chief Research Analyst at Internet Marketing Initiative (www.internetmi.com)