Mike On Ads

Mike,

Great to see someone post this. We have a unique view of the world due to the fact that we have both an Enterprise Hosted (SaaS) service and Download (free open-source) product. We get flooded with people that believe running a few billion ads through an open source download sitting on a $50 / month web hosting company is a piece of cake – until they can’t scale, have system / data issues, get hacked, etc. Then we usually get blamed for offering a product that doesn’t work for them!

Like AppNexus we have poured millions into our internal SaaS infrastructure and agree 100% with you that there could be a lot of “RTB” companies that by Q4 will definitely struggle to keep up and have already witnessed many blow ups with companies that believe they can handle the volumes.

One thing you didn’t mention is the compounding effect when not only dealing with scale but also the complexities of dealing with multiple geographic locations and redundancies, etc. Will definitely make for an interesting 2010!

Very relevant and timely discussion. With the inflation of ad startups which try to “throw in” ad serving one might be tempted to think there isn’t any sophistication to it. In fact, any serious company in this space can testify to the amount of investment going into infrastructure. As described above, we too learned that scaling up sometimes requires starting from scratch. The advantage is that once you have new infrastructure you can innovate much more quickly. When we developed MediaMind, half way through the process of rebuilding the infrastructure we found out that we were able to add new features twice as fast as we used to. It is risky to make such big investments in large projects but it can payoff big-time.

Fascinating article and discussion and it leaves me feeling the need to keep things as simple as possible.

I have a friend who has a newish network doing 85million impressions a day.

Just for discussions sake, lets say i wanted to start a similar network and being short sighted, i never intended to grow beyond 100million impressions per day (tops).

What would you recommend using for that that is ready made (nothing custom).

“….Take for example boolean segment targeting — the idea of having complex targeting logic. Eg, “this user is in both segments A and B OR this user is in segments C and D…..”

I completely agree, if an engineer/architect wants to do this in real-time, then it probably does not speak too highly of their technical skills-and I am not sure if I agree with the comment that this is actually what the better architected ad servers do and that this particular use case can be cited as the cause of increased latency. The trick here is to simply serve an ad based on the incoming cookie Id-which ad to serve, which segment to allocate this cookie to, which set of creatives to serve within the ad(if it is a dynamic ad)- these are all offline decisions and there are plenty of tools to do that kind of work(SAS Enterprise Miner, Hadoop et al.) Having engineers/architects who attempt doing these things in real time is a great way to head to the dead pool- That being said, what building scalable ad servers requires is not industry veterans who have spent their lives doing these things(and yes of course, they are far and few in between) but just smart, forward thinking coders and a lot of refactoring!- Hardly rocket science I say

New ideas in a stale product category are risky but often reap great rewards. There are a couple of things that innovative and pioneering companies should always do in order to ensure that the KABLOOOOOOOOEY moments are kept at bay.

First, they need to do comprehensive load testing. This isn’t difficult but takes expertise and a good QA team can set this up with automation. QA team, NOT the engineers that built the product. Pushing your production environment to the very edge of its capabilities and going through as many possible configuration and use scenarios as possible will not only show you the boundaries of your current capabilities but will usually open your eyes to other weaknesses and inefficiencies in your code. There are many companies that will drive dummy traffic to your system to help you determine your scalability and while it isn’t exactly the same as “real” real-time traffic this is a must.

The second thing that these ad technology companies should do is be realistic in what types of clients they’re willing to take on in the first year or so as they continue to build out their product. Yes, it would be fantastic to land a large client that will take you to break even sooner than expected but the safer, and smarter, play is to go with as many small to medium-sized clients as possible for the first year. This will place a burden on your client services staff but it’s easier to increase the size of that group than to let down a large partner and tarnish your name within the space.

The last suggestion I have, and it’s maybe more important than the first two, is you need to specialize. At least in the beginning. Don’t try to be everything to everyone in the ad world. It took DoubleClick, Atlas and others years and armies of engineers and experience to build their products. Why try to create a system to serve everyone? You’re just asking for the pain train to come rolling over you and your engineering staff and to take you in twenty different directions based on the needs of different types of marketers. Focus on one area that you think you can do well and dominate and then work outward from there.

Hi Steve,

If indeed your volumes will never exceed 100M/day I think in generally most solutions will work out alright. My general rule of thumb is that 1b/day is 30k/second at peak time — so 100M/day is about 3k/second.

Which solution to choose isn’t a question to be answered in a blog comment =) — that greatly depends on the type of network you are trying to build and what technology requirements you have!

-Mike

Mike, great article. I’ve had the pleasure (read sarcastically), of only working on next gen adserving platform teams my whole career. The one universal truth, we’ve made some scale mistake in everyone of them. From Juno to Quigo, to Microsoft… without fail, you never know all of what you needed to know. I would like to think we made new and interesting mistakes all along.
I’ve also had the pleasure of working with guys like Oded Itzhak, yaron galai Brian Burdick, Tarek Najm, Vikhas Jha and most recently Paul Martino and Rob Gryzwinski who have made careers out of big scale technologies. Even these guys with oodles of big scale experience run into problems. This is truly a space that seperates the “men from the boys.” I can tell you, I recommend looking for scale first, features second. I would rather have a guy that has experience with scale than a guy that knows every ad bell and whistle imaginable.

We’re already seeing companies fall down in scale all around us and I anticipate many more in 2010 alone. Exchanges are a great example. All this talk about RTB, and I’ve yet to see anyone in the ecosystem, thats right, no one do it for real. All I see today are trumped up re-targeting systems. I don’t think the real capabilities have yet hit the market… and likely won’t at scale until 2011. Show me the system processing 100,000 requests/sec with real logic sifting through inventory making dynamic decisions on multiple data points (yes that means more than 1) and I’ll show you a bunch of tired, over caffeinated, platform veterans on their 3rd or 4th platform.

Great article!

Mike, totally agree. The decline in CPMs over the past 24 months highlights your point: ad serving technology companies has reduced R&D in transitioning their platforms to higher scalability, and new media entrants have been lured by low-to-zero priced ad servers to maintain media margins. The challenge with both situations is latency, customer loss, and ultimately much higher marginal costs. Fortunately or unfortunately, AdJuggler hasn’t had the luxury of DoubleClick’s coffers and, as such, has built a highly scalable, yet small footprint platform. We have many publisher and network clients that suggest an impression volume in our cluster runs on 1/5 of the footprint of other providers. To Mike’s point, building the scale – at the OS, data, Java, and UI layers – is not trivial, and a key driver why such platform segments will consolidate like utility companies. Perhaps more importantly, as CPM have dropped, the sheer TCO – hardware, software, developers, backup, bandwidth, and other labor – makes a case that no media player should build or manage their own platform.

[...] WIth only 50 campaigns the total impact of the change is a 1ms increase in processing time — not n… [...]

[...] WIth only 50 campaigns the total impact of the change is a 1ms increase in processing time — not n… [...]

Good article Mike…not sure on the men from the boys tittle…is scale what is meant to separate or you going for a boyzIImen reunion tour?

My view on the industry is that what separates businesses that succeed vs. ones that don’t is maturity combined with execution. Good ideas are great but they don’t get sold easily and I agree scale is important.

Look forward to more posts!

[...] The Challenge of Scaling an Adserver [...]

[...] you haven’t done so already, read this post The Challenge of Scaling an Adserver [...]

Hi Mike,

Re: the (A^B)v(C^D) issue – to me it’s actually a question of on vs off-line processing.

The final actual logical (A && B) || (C && D) only going to end up one extra instruction anyway (assuming they’re all in registers) – far less than the time required to parse a cookie or HTTP header.

The most important thing I’ve found, and something that took me a while to convince others of, is that you can give up synchronisation. E.g. if you’ve got quotas you don’t want to be locking over something every request. It’s fine to have a delay in synchronisation, and soak up costs for the _tiny_ fraction of a percentage extra that may be delivered, at an incorrect price etc.

All that said, I’m working at what’s your low-end , >1K/s is a fairly big spike for me.

(I just came across your blog by the way – it’s a breath of fresh air to find so many people working on serving ads commenting on one blog)

Hi Mike,

Well written post.

I am going to go a bit math geeky on you for a bit!:-)

The problem in dynamic ad construction, as well as in optimization with algorithmic feedback loops at core boils down to “set computations” with a notion of relevance ranking and scoring. In general, the problem in real-time bidding is isomorphic to the dynamic ad selection problem – modulo some plumbing. The problem is of matching a high dimensional incoming vector to a set of high dimensional content/document vectors. The constraints for selection are often expressed in SQL like constructs with keyword search like semantics. In the literature, these are often called Non-Ordered Discrete Data Spaces, where standard Euclidean geometry does not work. And standard SQL oriented databases don’t work at all, in terms of scale of computation and the flexibility demands. A new mechanism that does the absolute minimum in terms of fast computations, where business rules for marketing are encoded offline, we found to be the best approach.

Having fallen down a few times implementing parametric search engines (in such discrete data environments) in past lives – our team has solved a substantial part of the retrieval scale problem in ad serving. We have had to run at volumes approaching 30K qps at latencies of less than 1 ms response times.

That was a surprising burst of unexpected volume that came from RightMedia – within the first 2 seconds at the top of the hour reaching such peak volumes. It was scary and fun.

It is also important to point out to the brave souls attempting this sort of build out that all of your systems – logging, data aggregation, analytics system scale are also not to be treated lightly. Everything gets stressed significantly. There are substantial challenges in horizontal scalability on those fronts as well.

There are some folks that appear to think that throwing hardware in a cluster will solve all problems. With battle scars to prove it, not really. You still have to look at your own computational pipeline and address bottlenecks one at a time. But nothing gives like a great algorithm. But you know that already. Long live D.E. Knuth!:-)

–pradeep

I spent a lot of time at Admeld discussing (erm, arguing) with the folks here about what seams at times like minute details of features and how they should be implemented (or how we need to go differently about things). Results of good designs can be scaled and grown rather then needing a rewrite in the future.

There’s a lot to be said for doing as much work ahead of time (offline) and sharing the upfront cost of it. But there’s absolutely no reason you should not be able to ridicules amount of user targeting in real time and not even having the cpu break a sweat.

I don’t think there’s anyone out there with well designed adserver who’s run into machine limits. Real machine limits where their bottle neck is page fault, cache misses. Including us here.

Lastly, it is my opinion that if can’t get faster at what you do as you grow up (request / and data wise) then you’re not doing it right. In fact as we’ve grown and added more features our systems acctually got faster. We now able to serve more qps (by order of magnitude) per machine then we use to.

[...] and I spent a lot of years building ad servers. So I can say this from experience: ad servers are hard to build. There’s a lot of expertise that goes into delivering the right ad to the right person where [...]

love the post, very interesting and informative. keep up the good work.