Chasing the long tail with keyword research (SEM 101)

When a key opens a lock, it typically provides the key’s holder with a clear path to where he or she wants to go. Keywords and key phrases do the same for a website. They help direct searchers to content they wish to see on the Internet. But there is a key difference: whereas a lock key will typically match up with only one lock, keywords can lead a searcher down multiple paths to many matching, relevant websites. It is a filtering process that leads the holder to the destination to which they want to go. (At least that’s how it’s supposed to work – see my recent article on keyword web spam for times when this is not the case.)

Search engines are still heavily oriented toward text-based content. Even when other media types are indexed, it is typically done so using text-based descriptions. Search engine users separate the wheat from the chaff on the Internet by searching for words that are relevant to the information they seek. That is their key. Smart webmasters, anticipating those users who will employ search to find content similar to what they’ve published, can boost their chances of bringing searchers to their websites by using the same words in their content that searchers will in their searches. It’s simply matching keys to unlocking (revealing) the content you want.

Sometimes the keywords and key phrases searchers will use for a given field of interest are obvious, but that’s not always good news for webmasters. If these keywords are obvious to you, it’s likely that they are obvious to everyone, and if your site falls into one of those fields, all of your competitors’ websites will be using those same keywords.

The long tail of search

If this is the case for you, there’s no need to despair – there is hope. There’s an often-overlooked truism in our industry: search has a long tail. Most webmasters only work to identify their sites with the head, so there’s typically a lot of untapped value to be had in working on that long tail.

What do I mean by head and tail? Consider the form of a tadpole. Much of its mass is in the big head, but then its form flows into a long, tapering tail. Graphs of keyword search trends often look like a tadpole with a very long tail. A few primary keywords typically dominate a sizable percentage of the search traffic, but then there are secondary and even tertiary keywords. By themselves, they are clearly not as effective as the primary keywords, as fewer users search on them. But there are people who either search directly on them or use them as a part of longer queries, and those users are just as valuable as conversion opportunities as users of primary keywords. The key distinction here is that most webmasters do not bother to actively compete for those potential customers in the long tail.

If you are in an industry that has a few heavy-hitter, powerhouse websites as competitors, whose webmasters have worked hard to develop great content and earn authoritative backlinks, it can be as frustrating as chasing your own tail for a smaller upstart to compete with those sites using the same primary keywords. Competing in the long tail can be a great way to mop up some otherwise untapped business and begin to develop a name and reputation for your website. It’s always better to compete for a high rank for a few keywords in the tail than to merely settle for a middling or worse rank for the most popular keywords in the head (settling for mediocrity is what most webmasters do, and thus why there’s so often good opportunities for the taking).

And with the time you spend successfully targeting the long tail keyword opportunities, if you make the effort to simultaneously develop quality content and work to earn authoritative inbound links for that content, your site will only increase in stature. At that point, you can start thinking about getting more competitive for those primary keywords in the head as well.

Make it so

So all of that sounds fine in concept. But how do you execute on such a plan? You have to know what keywords are being used in your field. You need to know what keywords you need to use on your website. You need to make your website a legitimate target for searchers who use those keywords. To get such keyword intelligence, you need a great keyword tool. One that is easy to use, draws from strong industry data sources, and offers a variety of views of that data. Frankly, I suggest you take a look at Microsoft Advertising Intelligence.

Microsoft Advertising Intelligence is the successor to the 2009 beta tool called adCenter Excel Add-in Keyword Research Tool. As you might have inferred by its previous moniker, it installs as an add-in to Microsoft Office Excel 2007 (it won’t work with any previous versions of Excel, however). You’ll need an account with adCenter to gain access to the keyword data, but that’s easily enough done, and there’s no cost for setting up the account. Note that the tool was designed for users of search marketing (aka Pay Per Click [PPC] ads). However, the research needed to develop strong-performing keywords for PPC ads parallels that of keywords for search engine optimization (SEO), and thus the tool is easily repurposed for those efforts.

Once installed, Microsoft Advertising Intelligence is presented as a tab on the Excel ribbon named Ad Intelligence. Click that tab, and from there, you have access to a series of helpful tools that can help you perform the following tasks:

  • Extract current keywords from an existing site
  • Create new keywords by starting with an existing list, a webpage, or by selecting a vertical
  • Expand current list of keywords by examining advertiser bidding selections and analysis of search query data
  • Analyze keyword performance by query, time, demographics, geo-location, and more
  • Identify the categories using that keyword and drill down to common queries
  • Identify key performance indicators (KPIs) for keywords and compare yours against industry averages
  • Look up typical PPC keyword pricing for particular keywords
  • Learn the click-through-rate (CTR) and the cost per click (CPC) around your chosen match-type position
  • Learn about industry KPIs and learn more about your own particular vertical, including the average CTR and CPC, and then compare your performance against your vertical’s average

I recommend that, immediately after installation, you first configure the tool to work with your adCenter account. In the Options & Help section of the ribbon, click Options, and then fill in the User name and Password fields with your adCenter credentials. Click Test Connection to confirm everything is ready to go. Once you get a message box confirming the connection was good, click OK to close the open dialog boxes.

There are nine tool buttons on the ribbon, some containing multiple, related tools. Instead of me trying to explain all of the cool stuff that Microsoft Advertising Intelligence can do, I’ll simply refer you to the tool’s website for technical documentation, its active community forum, and the numerous video tutorials.

Identify the long tail

Once you’ve installed the tool, you can use it to pull a list of the current keywords used on your website today. Here’s how:

  1. In Excel’s Ad Intelligence tab, click the Keyword Wizards tool, select the option Extract from website, and then click Next.
  2. Type the URL you want to use, and then click Next.
  3. You can first review the keywords extracted by clicking Review, and then click Next to continue.
  4. Select the option Queries That Contain Your Keyword to see other keywords based on those extracted from your site, and then click Next.
  5. You can either change the setting Maximum suggested keywords or use the default. Click Next to continue.
  6. Click Review to see the updated list, and then click Next.
  7. To see historical data on the usage of the keywords in your list, click Monthly traffic, and then click Next.
  8. You can then modify the range of dates for historical usage performance data retrieved as well as for forward prediction usage or keep the defaults.
  9. Click Finish to get your report.

In the resulting report, you can change the sort order of any of the columns of data to see which keywords and key phrases had the highest CTR on any particular month or in aggregate.

If you want to be very specific in conducting your research and customizing your reports, you can skip the keyword wizard and instead use the other tools in Microsoft Advertising Intelligence to narrow down keywords for specific verticals, demographics (including age, gender, and location), and more. You’ll see which words are the highest performers, and how those words have performed recently.

This is powerful information, and you’ll learn which words are being used in your field at which frequency. Check your site’s keywords against those who are the movers and shakers in your field, and you may discover some under-utilized keywords in the long tail of search that may be a golden opportunity for your site.

Once you do, implement them wisely on your site, and then monitor your site’s progress over the coming weeks and months. For advice on implementing keywords wisely, check out our earlier blog articles on using keywords, including Put your keywords where the emphasis is (SEM 101) and The key to picking the right keywords (SEM 101). Whatever you do, don’t follow the examples of keyword abuse documented in the blog article The pernicious perfidy of page-level web spam (SEM 101). Remember that SEO is not an overnight quick fix. Time is needed for crawling and reindexing changed content from the search engine side and then for searchers to find you. Patience, along with hard, smart work, will pay off. (And don’t ignore other aspects of a thoughtful SEO plan that can improve ranking as well, such as creating great, unique content and earning authoritative, high-quality inbound links!)

So stop chasing your own tail. Instead, invest in chasing the long tail of search by using a keyword intelligence tool like Microsoft Advertising Intelligence. That is the key for unlocking success in search.

If you have any questions, comments, or suggestions, feel free to post them in our SEM forum. See you again soon…

— Rick DeJarnette, Bing Webmaster Center