A.I. & Optimization

Advanced Machine Learning, Data Mining, and Online Advertising Services

Exploring/Exploiting Long-Tail Keywords



In December 2014, we ran an ad campaign using Google Adwords targeting Ad Tech companies who are interested in developing advertising algorithms for buying/serving banner ads. In words, our pitch was that instead of using human to run, monitor and optimize advertising campaign, we should use machines' computational powers plus advanced machine learning algorithms to find the most profitable segment of audience and start exploiting it. Check the link below to view the landing page of AdWords campaign:

Using Online Learning Algorithms for Computational Advertising

Where to start?

To start the Adwords campaign we picked a set of keywords containing a combination of "advertising" and "machine learning" keys in them. We picked the exact match in order to test the keywords first before relaxing the match type (e.g. phrase match). Some examples of keywords we targeted were keywords like: "[machine learning online advertising]", "[multi armed bandit algorithms]", and "[machine learning for buying and serving ads]". As you see, the first keyword is not very specific in the sense that it targets two well known keywords "machine learning" and "online advertising". The second keyword is more specific to the promoted article. Finally, the last keywords seems very specific.

After running the campaign for a week, we went back to test the results. We found that the bid value for "[machine learning online advertising]" is $7 in order to appear on the first page results. This should not surprise us. We also found that the second keyword "multi armed ..." has a bid value of $14 which seems expensive CPC. The last keyword suffers from low search traffic. It's interesting to see there is a high competition for "multi armed bandit algorithms". One common issue with all of these keywords especially the first and second one is that they are to some extent general and not specifically targeted.

Long-Tail vs Generic Keywords

As mentioned earlier, keywords like "machine learning and online advertising" or "multi armed bandit algorithms" might not bring the right prospect to the landing page for the final conversion. This means that you waste your CPC $ on some people who are looking for something else. Also as we saw, both keywords are very expensive (due to high competition). The price of the keywords can be justified by considering the fact that both keywords are generic and many people/businesses are competing for them. So, what's the solution? It worth considering the probability distribution of the search volume of all possible keywords that people search on the web as a function of keyword length.

Power-Law Distribution for Web Keywords

We expect that there are lots of short keywords such as "advertising" or "machine learning" which drive a high volume of traffic impressions. On the other hand, we expect to see a very large number of longer keywords such as "optimize ads using machine learning" that each of which have lower volume but there are many of them due to combinatorial explosion. These long tail keywords that are more specific count for a significant portion of search engine traffic. So, the goal here is to take advantage of long tail of search engine market: Long Tail.

Generating Long Tail Keywords

We saw a few interesting properties in long tail keywords: they are more specific, they are more likely to be used when users have buying intention, and also there is less competition for them. Now the main question is how one can algorithmically find such keywords. Finding long tail keywords are hard where one needs to put herself in the customer position and see what keywords are likely to be used to search for a product or service. Obviously this process involves lots of testing where we need to test and see the performance of each keyword and make changes accordingly down the road.

In the future, we will discuss how one can programmatically generate long tail keywords.