What is A/B testing?

A/B testing is a user experience research methodology. A/B tests consist of a randomized experiment with two variants, A and B. This is also known as two-sample hypothesis testing.

---

Does A/B testing have limitations?

Yes, A/B testing has limitations. The biggest thing A/B testing doesn’t account for is visitor behavior.

Let’s say 60% of people prefer ad set A and only 40% of people prefer ad set B. You then choose ad set A because it had the majority preference of 60%. How can you account for the other 40% of visitors who preferred something different?

Numbers lie. An example of this is that ad CPM/CTR is an average, which reveals very little about the underlying data and just how your impressions are priced.

Test groups must be random. When running A/B test, how do you segment your traffic? Who is really in each group? Does each user get assigned to a group? Each session? Each pageview?

The math is not easy or straightforward. How long to run for? What is the underlying distribution of data? What level of significance do I need? How big is the effect? How do I control for random bot traffic?

Not responsive to changes as they happen. You run your test in June, find a statistically significant effect, you’re confident in your results… but how confident are you that everything will be the same in December?

---

Do the results of A/B testing stay the same over time?

No. Ezoic’s data show the effects of A/B testing results seem to vanish over time.

Why is this?

• Emerging trends in user behavior
• Advertisers changing priorities
• Seasonality in user clicks
• False positives in data.

“Statistically significant” does not mean something is real or something that we can base all future decisions on.

---

What advantages does A.I. have over A/B testing?

A.I. has many advantages over A/B testing.

• Reduces data mining
• Eliminates hard coding
• Robust changes in data patterns over time
• Outsources the math

Take this example. How would you traditionally tell a computer system how to define what variables make a dog a dog?

• Animal
• Mammal
• Barks
• Walks on 4 legs

This is great until you get an outlier that still technically is a dog, but doesn’t fit all the variables you gave the computer—a dog with three legs that lost one in an accident.

What AI does is it flips traditional programming on its head: rather than defining the steps to get to an outcome (“take A, B, C, D… do this to them, make a decision), we can send in some data, and let the neural network learn what’s important. We could even add a completely irrelevant descriptor, like has red ball in mouth. It’ll learn quickly that this doesn’t describe dogs at all.

---

How do the neural networks behind A.I. work?

The learning process behind a neural network is a lot like how humans learn. But the math behind the learning is a lot like how our brains work at a cellular level.

Neurons take in information at their dendrites, weigh all of the inputs, and ultimately send an output based on the weighted signal.

If the weighted signal says “yes, dog”, the dog neuron fires. 

---

How does A.I. increase the value of ad inventory?

When it comes to ad placements, publishers are often met with dilemmas. Which ad do you show: top of page or sidebar?

Let’s say you look at your ad data and the CPM for “top of page” and “sidebar” are both $3.02. But is the amount the same for those two ad placements when more visitor behavior variables are introduced?

Instead of doing all of this investigating yourself, you can outsource this work to Ezoic’s artificial intelligence, powered by tensorflow.

Next, we tell tensorflow what features in our data we want it to learn from.

After that, we tell tensorflow what type of model to use: here, we’re using a deep neural network regressor (which is a good model to predict numbers, like CPMs)

Now, we tell tensorflow how to load and read the data

And lastly, we initiate training. 8 lines of code in total! It’s that easy. So what does it find?

Tensorflow figured out how to segment the data automatically, in a scalable way. This neural network found an extra 12% of ad revenue hidden in the data.

Summary statistics are uninformative, and you’re leaving money on the table by not digging deeper into your data.

---

What is Ezoic’s Ad Tester?

Ad Tester uses real artificial intelligence to increase ad earnings and improve visitor experiences. You control the experiments and Ezoic streamlines how ad partners, density, types, and locations are delivered to each visitor.

Ezoic learns from results automatically then makes intelligent decisions on how to display ads to each visitor. This increases ad revenue, improves UX, and causes advertisers to bid higher over time because of the improved website engagement.

---

What variables does Ezoic’s A.I. Test?

The ad variables tested are:

• Ad type (display, native, link, video)
• Ad location (where on the page)
• Ad size: e.g. 300×250 or 300×150
• Viewability metrics
• RTB landscape
• Presence of other ads and their location
• Navigation options present (sidebar, menu, etc.)
• Page load speed – critical path rendering
• Content location, e.g. recommendation widgets

The visitor variables tested are:

• OS or Device Type
• Browser type
• Time of day / day of week
• User connection speed, e.g. broadband or 3G
• Geographic location
• Upstream traffic source, e.g. Google
• UX metrics from other users visiting this page
• Subscriber or non-subscriber
• User history
• Many, many more…

---

Is Ezoic’s A.I. only used for Ad Tester?

No. The neural networks that powers the artificial intelligence behind Ezoic’s Ad Tester also powers the Site Speed Accelerator, Layout Tester, and more.

---