SEM: How to Find Keywords

In this part of our discussion of Search Engine Marketing we explore ways to locate the right keywords to attract search traffic including the use of several online tools and other useful techniques.

In the first two sections, we stressed that using keywords within a webpage title, page headings and page content are important for gaining traffic referrals through search engines.  Yet for many marketers unfamiliar with building strong content-rich websites the obvious question arises: How do I know which are the right keywords?  In this part of our on-going tutorial we address this question and offer several suggestions for finding the right keywords.

We stress that our goal is not to suggest tricks to attract search engines to a website but to offer sound, basic advice that makes a site more open to search engine evaluation.

We break down the ways of finding keywords into te following seven categories:

Generally Accepted Word Usage

Keyword development begins with an understanding of words and phrases most common to the marketer’s industry.  Most marketers should have no problem developing a list a terms, however, there are bound to be terms that do not easily come to mind.  A good tactic to dredge up words is to peruse the past twelve months of industry trade magazines (either print or online).  Look for words and phrases mentioned in several different issues of the same magazine and across different magazines since the more terms are repeated the more likely they are important terms.  Pay particularly close attention to terms found in the title of articles since titles are often written to attract attention and may include the most common form of the term. 

Once these words are discovered make sure the words are not strictly “supplier” jargon but are words used by customers.  A frequent mistake websites make is to focus on terms used internally by the marketing organization, rather than on words used by customers and media to refer to the same thing.  One method of checking this is to visit websites customers use to learn about industry products (e.g., consumer magazine site, consumer forums) and use the site’s search feature to enter search terms discovered in the trade publications.  You may find reference to the term within the context of “also called” or “sometimes referred to as” along with the terminology more common to the market.  If the site does not have a search feature or the search feature is not very robust, use the Advanced Search feature on one of the major search engines then limit your search to the web address of the consumer site.  A search engine’s strong indexing capability should locate most information on the site assuming it freely available when originally posted to the site.

Search Engine Supplied Keyword Suggestion Tools

Possibly the most cost effective method for determining a large number of keywords is to use tools offered by search engines within their advertising support toolbox.  The three major search engines in the U.S.  – Google, MSN and Yahoo – provide outstanding keyword suggestion features.  The search engines do this in support of their advertising programs, which allow marketers to place keyword generated ads (i.e., ads based on a user’s search string) on the search engine as well as on other websites within the search engine’s advertising network.  With the exception of Yahoo, access to these tools requires registration, which may include a small one time fee. 

The keyword suggestions offered through these site are based on searches performed through the main search engine, its international versions, and through searches performed through a search engine’s network of site.  These networks are made up of websites that include the search engine functionality on their site.  For example, Google supplies search functionality to America Online and Comcast and thousands of smaller sites while Yahoo supports search for CNN and USA Today.  Whether a site’s search is powered by a major search engine can be determined by examining the area around the search box since it generally contains information saying the site’s search function is “Powered by” or “Enhanced by” one of the top search engines.

The keyword suggestion tools offered by the leading search engines include:

  • Google – With Google’s advertising management program, called AdWords, users will find several tools for selecting the right keywords.  The basic option provides a list of related words to a user entered word.  Another option gives a listing of synonyms for an entered keyword.  Another option allows a user to enter a website address and Google will then perform an analysis and return appropriate keywords for the site.  And the URL to be evaluated is not limited to the marketer’s URL.  Entering the URL of a competitor’s site will also produce a list of keywords.  Finally, the Google’s keyword tool offers a glimpse of a keyword’s search volume, thus indicating which words are currently more important.
  • MSN – Microsoft’s MSN search engine’s advertising tool is a recent addition to search engine advertising.  In the past, all ads appearing on MSN were placed on the site through arrangements with other advertising providers including Google.  But in 2005 MSN began testing its own advertising program called AdCenter.  While still in its infancy, the keyword tools offered by MSN appear to be useful and worth a try.
  • Yahoo – Yahoo Keyword Selector tool (sometimes referred by its former name Overture) is the only one that can be used without a registration.  Unlike other tools, the Yahoo tool does not offer suggestions for related keywords.  The best feature of the Keyword Selector tool is the reported number of times the keywords was used in a search query within the Yahoo network of sites.  Generally this information is shown for the last full month.

For-Fee Services

While most of the search engine tools we’ve discussed are either free or available at low cost, there are other options for finding keywords that require payment.  While these may be costly, the website marketer should at least spend time evaluating them since these are popular among many search engine optimization professionals.

  • Subscription Services – Several Internet sites offer keyword help by analyzing searches in ways that differ from the methods used by the top three search engine tools.  The leader among these services is Wordtracker which compiles information from searches done through several lower level search engines.  Along the same lines as Wordtracker is KeywordDiscovery which claims to gather information from a large number of search engines, though most of what they do is based on deals they have made with Internet Service Provides, who are able to record search queries in an indirect way and then share this information with KeywordDiscovery.
  • Research Reports – At a cost that far exceeds the Subscription Services are the reports offered by Internet research companies.  These reports are not cheap and probably only of interest to the well-funded website marketer.  The leading option is Hitwise, which produces monthly reports by industry including analysis of search terms.  Like KeywordDiscovery, Hitwise gathers its data from relationships with Internet Service Providers but also utilizes software installed on many websites and computers.

Free Web Tools

There are many free tools on the Internet that offer help with keyword search.  Here are a few:

  • Google Suggest – In addition to the keyword tools offered in its advertising area, Google offers an interesting feature called Google Suggest, which offers suggested alternative words to what a users enters in the search box.  The fun part of this feature is that it works as the user types.
  • DigitalPoint – This site is mostly oriented to techie-types, but it does offer a number of tools that marketers will find interesting.  For keywords there is a nice tool that combines Yahoo’s Keyword Selector tool and Wordtracker to give keyword suggestions.  The tool not only lists suggested words but goes a step further and indicates per-day search frequency for each suggestion.

Search Engine Clustering Suggestions

So far we’ve discussed how search engines and other tools provide ideas for keywords using backend (i.e., not on search page) tools directed specifically to website marketers.  However, there is an additional front-end feature offered by a couple of search engines that could be helpful to website marketers.  The feature called clustering, is primarily designed to improve the user’s search experience.  Clustering helps users narrow their search by producing groups of similar search words within the same topic as the originally searched keyword.  The sites that do offer it include:

  • Vivisimo – The originator of clustering, this search engine is mostly marketed as a search tool for organizational sites and counts as clients the U.S. Government and several Fortune 500 companies.  Entering a search term will produce a list of Clustered Results presented as categories, which often appear as single words rather than phrases.  However, categories can be drilled down to find more narrow keyword suggestions including phrases.
  • Microsoft SCR – Among the major search engines only Microsoft has the clustering feature, and they only have it available in testing mode on an obscure website called SCR.  The SCR stands for Search Result Clustering and this site is a development site from Microsoft’s Asia research group.  While the look is not as clean as Vivisimo, SCR displays similar results, though SCR is more likely to produce keyword phrases on the initial display.

Trend Monitoring

Newly coined words and phrases are entering the lexicon nearly everyday.  While many of these never gain acceptance, a large number will eventually become widely used.  Some may even become used within an industry or culture as the key term to describe the industry (e.g., podcasting, viral marketing, crackberry). 

Staying on top of trends can give a website clues to keywords to include within their site.  Here are a few sites that may help:

  • Trendwatching – Interesting site that offers a monthly newsletter examining a consumer trend that appears to be occurring somewhere in the world. 
  • Wordspy – The creation of new words are often the sign of a trend and this site provides a regular updated list of new words including examples of usage. Site includes a subject index to quickly locate terms.

Analysis of Server Logs and Internal Search Logs

One of the most overlooked methods for determining important keywords is actually provided by website visitors.  Many sites, particularly smaller ones, have little knowledge of how visitors reach their site and probably know even less about what they do when they visit.  Yet each time someone visits a site they leave a trail of information, some of which can be quite useful for determining which keywords are important.  This information is contained within the web server log which records all user’s activity, such as how they arrived at the site, what pages they visited on the site, how long they spent and much more.

Using site logs can be useful in several ways. First, if a visitor is referred to a website by a search engine, server logs will record which search engine sent them AND what keyword search was used to locate the site.  Mining the server logs for this information shows these words.  But this raises the question: If visitors are already getting to the site using the keywords isn’t the site already successfully optimized for those keywords?  Maybe not.  Having visitors directed to a site doesn’t mean the site is doing as well as it can with the keywords.  It is more important to examine the keywords along with a site’s position in the search engine rankings.  To do this, enter the keyword string into the search engine from which traffic was received.  Next check the site’s ranking for that keyword.  If the site appears well down the list (e.g., below 10th place) then clearly the site is not receiving as many referrals as sites that show up on the first page.  This may suggest to the marketer that some adjustment to the site could improve their rankings. 

Second, an examination of the web server logs will tell the marketer which keywords are NOT helping drive traffic to the site.  If the logs report low or no referrals for keywords that the marketer expects to do well, then the marketer should be alarmed and needs to take steps to find out why the situation exists.

Third, there is another type of log that can also provide a great deal of information.  This log may exist on websites containing their own internal search function.  Sites that have programs to operate their own search generally also have the option of retaining all entries that were entered in the search box.  From this information a marketer can glean what search terms are entered within the site.  This may offer some guidance on terms that are of interest to website visitors but which the marketer was not aware.

SEM: Site Navigation
SEM: Page Content