Intro
Ad is the major monetizing product for many companies. Here I’ll introduce how I build the simulator to evaluate and develop the ad system in indeed.
Here’s the slide:
The difference of DSP and “when you own the whole system”
When you’re just a DSP company, you could treat the external world like they’re static, and do your A/B test normally. Since your effort to the whole bidding world is tiny, your algorithm change cannot really change the whole bidding eco-sysyem hugely.
But when you own the system, things are very different: whatever your algorithm change, it will ramp-up to 100%, and it will change the world — which means what you see in 1% A/B test will be different as you ramp-up.
While we separate ads into control and treatment groups and bid with old and new bidding algorithm accordingly, they’re still bid together, so they will have network effect. Besides, ad product always has terrible seasonality concerns, making online test volatile.
