How to Find Keywords

Analysis of Server Logs and Internal Search Logs

One of the most overlooked methods for determining important keywords is actually provided by website visitors.  Many sites, particularly smaller ones, have little knowledge of how visitors reach their site and probably know even less about what they do when they visit.  Yet each time someone visits a site they leave a trail of information, some of which can be quite useful for determining which keywords are important.  This information is contained within the web server log which records all user’s activity, such as how they arrived at the site, what pages they visited on the site, how long they spent and much more.

Using site logs can be useful in several ways. First, if a visitor is referred to a website by a search engine, server logs will record which search engine sent them AND what keyword search was used to locate the site.  Mining the server logs for this information shows these words.  But this raises the question: If visitors are already getting to the site using the keywords isn’t the site already successfully optimized for those keywords?  Maybe not.  Having visitors directed to a site doesn’t mean the site is doing as well as it can with the keywords.  It is more important to examine the keywords along with a site’s position in the search engine rankings.  To do this, enter the keyword string into the search engine from which traffic was received.  Next check the site’s ranking for that keyword.  If the site appears well down the list (e.g., below 10th place) then clearly the site is not receiving as many referrals as sites that show up on the first page.  This may suggest to the marketer that some adjustment to the site could improve their rankings. 

Second, an examination of the web server logs will tell the marketer which keywords are NOT helping drive traffic to the site.  If the logs report low or no referrals for keywords that the marketer expects to do well, then the marketer should be alarmed and needs to take steps to find out why the situation exists.

Third, there is another type of log that can also provide a great deal of information.  This log may exist on websites containing their own internal search function.  Sites that have programs to operate their own search generally also have the option of retaining all entries that were entered in the search box.  From this information a marketer can glean what search terms are entered within the site.  This may offer some guidance on terms that are of interest to website visitors but which the marketer was not aware.