Quality Score - Breaking Open Google’s Infamous Black Box
Ah, quality score. Whether you hate it or love it, every search marketer learns early on that having a high quality score (QS) for their keywords is essential to lowering cost per acquisition and increasing return on ad spend. In a recent post, we explained how quality scores are calculated and the impact they have on campaigns. At first glance, QS appears simple. It is a score out of 10 for each keyword, further broken down to three categories: expected click-through rate, ad relevance, and landing page experience. But digging deeper into the factors that determine a keyword’s quality score, we learn relatively little. Google is characteristically tight-lipped about the exact algorithms governing the system. It’s the proverbial black-box metric: at best vague and inconsistent, at worst hair-pullingly frustrating.
When dealing with a metric as fickle and opaque as quality score, data has to inform every decision. Realizing there was a lack of data to justify hours of work re-writing ad copy, we set out to quantify and answer this question: Does including a keyword in ad copy improve ad relevance and quality score?
Searching for a Relationship
Before starting the experiment, we compiled keyword data from existing accounts to see what they revealed about the relationship between quality score, ad relevance, and ad copy. Our goal was to determine if keywords included in ad copy text had consistently better ad relevance and quality scores. We examined 50 exact-match keywords across ten accounts. Each of these keywords had more than 200 impressions per month and included a good mix of branded and non-branded keywords, as well as a wide range of quality scores.
From there, we divided the keywords into groups. The two components of quality score that were not analyzed, expected CTR and landing page experience, were held constant within each group. This ensured any QS differences between keywords were due to ad relevance alone. Within each of these groups, the quality scores were divided based on whether or not the keyword was included in the ad copy and then compared against each other.
These ad changes had a significant impact. While holding the other components of QS constant, including a keyword in the ad copy, nearly a whole point was added to its quality score. Crucially, this increase was observed evenly across branded and non-branded divisions, and all levels of QS - meaning that non-branded keywords increased just as much as branded ones, and keywords with a quality score of eight improved just as much as those with a quality score of three.
Putting It Into Practice
It’s one thing to observe a relationship, but another to put it into practice. To prove the viability of keyword insertion, our team conducted a series of concurrent A/B tests in AdWords. We took 14 keywords which were not present in ad copy, then duplicated them and split the pairs into test and control ad groups. The test group ran an altered ad containing the keyword, while an unaltered ad ran in the control group. As you can see below, these changes were held to a minimum to keep the ads as similar as possible. Traffic was divided evenly for each pair of ad groups with the campaign experiments tool. The experiments ran for 30 days; then data was collected and analyzed for each pair of keywords.
The success here was even more resounding. None of the keywords included in their ad copy had a lower quality score than their unaltered counterparts without the keywords, and nine of the 14 keywords increased by an average of two points! The keywords which did not see a jump had a starting quality score of eight or ten, to begin with. Ultimately, these results confirmed keyword insertion’s effect on improving ad relevance and quality score.
How Marketers Can Use This to Increase Quality Score
Based on these tests, we determined a direct correlation between keyword insertion in ad copy and increased ad relevance and quality score. It’s important to note that a few keywords with a quality score of eight and below average ad relevance did not improve with keyword insertion. Even more perplexing is that several keywords were not present in their ad copy, yet had perfect 10's. These outliers demonstrate that quality score has much more going on under the hood, and that every tactic should therefore be applied under careful observation.
Ultimately, this research provides a strong, quantitative foundation for improving quality score. Incorporating keywords into your ad copy has a high likelihood to raise quality scores by 1-2 points, and has never been observed to decrease it. Digital marketers can leverage these results to not only take simple steps to improve their keywords’ quality scores but more importantly, to justify that work beforehand!
If you’d like to learn more about quality score and how to best optimize your search strategy, reach out to Rise.
You might be interested in:
July's Minute with a Marketer features Anthony Kim, director of mobile strategy for [...]
A few months ago, we shared highlights from our Millennial focus group at Chicago's  [...]