Mike On Ads

It’s been four years since I wrote one of my most popular series of blog posts of all time — “The Ad Exchange Model”. Since then a lot has happened. A whole slew of three letter acronyms has appeared: DSP, SSP, DSP, RTB… Venture capital investments have exploded, we have multiple blogs dedicated to ad-exchanges and it looks like the space has gotten a lot more complicated.

Or put another way… in 2007 I described the world with a simple diagram. Today Terry Kawaja has an industry “LUMAscape” that has logos so small I can’t even read them.

FROM…	TO

Wow. What the hell happened? We used to have this easy world… publishers sold to advertisers, there was one exchange and then a lot of ad-networks with different pitches. How the hell did we get from there to the above hodge podge “ecosystem” that nobody understands.

To help bring some clarity to this world I’d like to kick off a new series… “The RTB Display Ecosystem”. This first post is will primarily be musings on hype… as before we can talk about what’s really happening we all need to step back for a second and realize 90% of what we read is… well… bullshit.

How VCs and Bankers brought hype to the industry

I’m sure by now you’ve seen the below diagram. It’s confusing, cluttered and supposed to explain to the world how the new Display ecosystem works. 100s of companies have incorporated this slide in their presentation. I haven’t gone to a single conference where this hasn’t come up multiple times.

Here’s the hard truth people don’t like to hear. The display world is actually not that complicated. Yes the ecosystem has evolved. Exchanges are core tenets of display and there certainly has been a ton of innovation in the data space. But that doesn’t make for 20 boxes on a slide. What really happened is that online advertising captured the attention of silicon valley… complete with a massive influx of VCs, $, TechCrunch posts and of course… HYPE.

You see, Venture Capitalists make money off of home runs. The top companies in a category get great exits, and after that valuations drop off very quickly. To actually be able to justify an investment a VC has to be convinced that the company has a chance at being top in it’s category. Well, this is quite hard to do if the world were simply advertisers, publishers and ad-exchanges. So what do you do? Well… you create a new category, pump millions of dollars in a company marketed as such category, and then hype up this category on TechCrunch as the next greatest thing and rejoice.

Of course VC’s have coffee with each other, hype up their investments to their VC friends (ever heard of an “Echo Chamber”?), and now they’re all clamoring to invest money in other companies who could vie to be a winner in the category and a new slew of companies gets funded.

In 2006 it was impossible to differentiate yourself as an ad-network… and thus every ad-network rebranded as an exchange. In 2007/2008 nobody could raise money as an “ad exchange” that would compete head-to-head with Google and Yahoo. But “SSP” worked out quite well… even though it’s exactly the same business model (queue funding for Rubicon, Admeld & PubMatic, etc.). In 2007/2008 you also couldn’t raise money as an ad-network, but there were plenty of companies interested in helping advertisers spend their money… enter the “DSP” category (queue funding for MediaMath, Turn, Invite Media, etc.). What’s funny is that TMP was doing the SSP business before anybody else and Ad.com has been offering “DSP Services” since .. well, forever!

VCs are also obsessed with investing in “Technology Companies” that build “Scalable Platforms”. You see, Technology is supposed to be sticky. Platforms have ecosystem effects and become $1b companies. To adapt, companies have quickly adjusted their positioning to better reflect attributes that will attract high valuations from said Venture Capitalists. Again, ad-network isn’t sexy, but a technology “Demand Side Platform” is. The funny thing is… it’s hype yet again. Most companies on the LUMAscape slide receive the majority (if not all) of their revenue from media services and not technology fees. Now this line is blurring (more on that later) but what companies are doing is saying they are “technology providers” while behind the scenes they operate exactly like a media company. An “SSP” technology provider hands out tags to publishers and then pays them a check at the end of the month together with a nice excel sheet. This is exactly the same business model of many an ad-network.

Don’t get me wrong — I’m not saying that any of the aforementioned companies aren’t or can’t be great companies. Many of the companies I mention have built terrific technologies and great businesses and some have followed that up with successful exists. But, they did all capitalize on a great marketing opportunity, at the expense of some “old world” companies who were too slow to react.

And this is where VCs and Bankers are actually hurting the industry rather than helping. They are reinforcing the importance of new categories that in themselves shouldn’t necessarily exist. Rather than focusing truly on what a company does they repeat and hence validate what companies say they do.

So what’s next?

Well first, let’s stop the hype cycle and start celebrating real successful businesses for what they have accomplished. Give me more case studies of real results and less BS!

In the coming blog posts I’m going to lay out the new RTB ecosystem and how all the different parties are interacting. Your feedback is as always invaluable so please leave comments with specific topics you’d love covered.

If you haven’t done so already, read this post The Challenge of Scaling an Adserver first.

I received a lot of feedback on this post, both in comments and private emails — some good, some bad. Two things were pointed out to me in relation to innovation and established players that I thought were worth a follow-up post.

When real businesses depend on you

Most startups I know release as often as they can. On the balance of testing versus finding bugs in production, most err towards production. Hell, it can take days of testing to ensure a clean release, whereas only 30 minutes in production will point out pretty much every bug in the code.

Here’s the problem — that attitude doesn’t really work once there are large public companies that depend on your platform for 100% of their entire business — often the case when you are the true adserver of record. I know, crazy idea, there are people that pay us money to run these adservers! Once a certain amount of money starts flowing through a serving system suddenly the engineering and operations teams are held to a whole new standard than before. 30 minutes of “bad data” in production can mean hundreds of thousands of dollars of lost revenue — ouch.

Generally this pressure doesn’t come until after a bad release which costs somebody a lot of money. At this point, the CEO mandates that all production releases must go through rigorous testing to ensure that such a catastrophic event will absolutely never happen again. Immediately job postings will go out for more QA engineers and very quickly every release has to go through several steps of vetting and testing before it is deemed ok to release. Very quickly, cultures change to focus less on features and more on stability. As we all know… stability is the evil enemy of change. Change, sadly, is a requirement for innovation.

Invisible Features

The second thing I forgot to mention in my last post is the concept of “invisible features”. Running a simple adserver with little volume there are a very large number of “off the shelf” components that you can use that simply work at low scale.

Take for example server-side cookies. Imagine I’m the small startup discussed in our last post. I’m seeing about 100k unique users a day with my adserver, for a total of about 1M/month. I store about 1kb worth of data per user which means I have about a gigabyte of total data to store. I’m only doing about 3000 ads per second, so I buy myself a nice new shiny Dell server, install MySQL and use that as my server-side data store. Every few hours or so I use mysqldump to make a full copy of my 1GB of data and copy that to a central storage server. I’ve got gigabit ethernet so it only takes about 12.5 seconds to make a full backup of my data.

Compare that to an enterprise adserver… that’s got a few billion ads a day with 150M unique users/month, with 3kb of data each. Not only that, but enterprise active customers require redundant datacenters to meet SLAs — this means all data has to be available on both the east and west coasts — at the same time. This means the enterprise adserver has to store 450GB of data in two datacenters, and be able to read/write to this data 100s of thousands of times per second. Guess what MySQL will do in this situation?

KABLOOOEY!

I won’t dive into all the details, but suffice to say that just backing up 450GB of data will take over an hour and a half on a GigE connection — compare that to the 12.5 seconds it takes at small scale and you probably get an idea of how hard of a problem it is to keep 450GB of data in sync across the country. There is no off the shelf solution that addresses this problem. What that means is that instead the engineering teams now need to focus significant energy and resources into building a mass distributed primary-key value store.

Functionally there is little to no difference to the end-user of the adserver of the two different systems. Both store a certain amount of data on a user… the difference is one system scales and requires a significant amount of effort while the other doesn’t. This is what I would call an “invisible feature”. It’s one of the things that one has to do to be a viable scalable adserving company but one that goes utterly unnoticed by customers (assuming everything works).

Server side cookies aren’t the only one of these systems that can be found “off the shelf” at low volumes but break when we start talking billions. Data pipelines, data warehouses, ETL systems, operational systems that can manage 100s of servers at a time — and this isn’t even to mention all the supporting hardware an infrastructure — core switches, load-balancers, CDNs and ISPs — each items that start to fail at billions and require upgrades and custom integrations to function properly at scale.

How do you know?

One of the things that’s so challenging in this market is that few of the new technology companies has had to deal much with the scale problem. So as a buyer of an adserving system, how do you know that it’ll work? If both company A and company B have server-side cookies, how do you pick the one that’ll work at scale? That’s the problem… if neither is doing serious volume, who knows! More on that in another blog post.

PS: There’s a great paper on Amazon’s Dynamo Store — a similar technical problem to the one most adservers face on building server-side cookie stores. It’s a great read if you’re curious as to the problems one needs to solve at scale.

One of the things that boggles my mind is how massively fragmented and confusing the display world still is. It’s been over three years since the first ad-exchange launched yet the world hasn’t significantly changed. What makes matters more confusing is that there is no consistent terminology to describe what a company does. It seems everybody describes themselves as either a platform, marketplace or exchange — so what’s the difference?

A company can call itself a publisher, an agency, a network, a broker, a marketplace, an exchange, an optimizer — what does it all mean? What’s the difference between Right Media and Contextweb? Admeld and Rubicon? That’s really the problem — today’s commonly used labels are useless.

Instead of evaluating a company based on labels, evaluate it based on the services it provides, technology it has, the partners it works with, the revenue model and the media revenue it facilitates. Note — below I focus entirely on companies that in some shape or form touch an *impression* — either as a technology provider, buyer or seller. There are peripheral companies that provide a whole world of supporting services, but I’m leaving those out for now to avoid confusion.

Services

Each company provides certain core services to partners, customers and vendors. These primarily center around the relationship the company has with the media that flows through it.

Technologies

There are certain core technologies that define what a company does. Note that you will find technologies such as dynamic creative optimization, behavioral classification and contextualization missing from the below list as they are differentiators — they don’t define what a company does but provide a competitive advantage over the competition.

Size Matters

Last but not least, the size and the partnerships of a company matters. I’ve written before about the perils of building technology in a void. You can have the most amazing platform that provides great services, but if you’re only running a few thousand dollars a month it’s all moot in the grand scheme of things.

Size can be measured either in impressions or revenue, the latter being far more telling. Getting a billion impressions of traffic a day isn’t hard these days — between Facebook and Myspace alone you probably have close to fifteen billion impressions of traffic running daily.

There’s a huge difference between a partnership and a media relationship. If you’re willing to foot the minimum monthly bill, anybody can buy Yahoo’s inventory through the Right Media platform. That doesn’t say much about who you are as a company. A partnership is different — it might be deep API integrations tying two platforms together or co-selling and marketing a joint solution.

What does it all mean?

Phew… that was a long list, so what does it all mean? Well, the above provides a slightly less fuzzy framework than the classical “ad-network”, “marketplace” or “exchange” commonly used labels to describe a company. Let’s look at a few examples:

Rubicon Project provides publisher representation services through it’s network optimization platform, arbitrages inventory through it’s internal sales team has both an internal and has an externally available adserver (one for the sales team and one for publishers) and is rumored to be working on real time buying APIs. That’s a hell of a lot more descriptive than “publisher aggregator” or “network optimizer”. The one thing I always find confusing about rubicon is that it their incentives seem to be fundamentally misaligned. How can you both arbitrage inventory and serve as a publisher representative? Updated (5/3/09 @ 8pm EST) — I seem to be misinformed. Per comments, Rubicon does not sell inventory directly to agencies.

Compare this to AdMeld which provides publisher representation services through it’s network optimization platform — an externally available adserver — and provides buying APIs (currently via passback). So what’s the difference with Rubicon? Well, one has an internal ad-network and the other doesn’t — different incentives. Publishers are starting to treat Rubicon as another ad-network in the daisy chain whereas AdMeld sells all remnant inventory as a trusted partner.

ContextWeb has an internal adserver (with a self-service interface… I don’t count that as external), they arbitrage inventory, and provide buying APIs. Compare this to Right Media which has an externally available adserver, buying and selling APIs, internal trading, data aggregation and arbitrages media (through BlueLithium/Yahoo Network). Both are “exchanges”, but clearly there is a pretty big difference between the two!

Of course if you get to Google your mind starts to explode just a little bit — as they do everything. Seriously. They buy & sell, have multiple adservers, provide buying APIs, internal trading, data aggregation…

Final Thoughts

I hope this post has given you some ways to start thinking about companies in the online ad space. I’d love to hear your feedback in the comments — what core services & technologies am I missing?

Now — next time someone says — “I’m an exchange”, why not ask — “Ok, that’s great, but what do you really do.”

Just placed an order on Seamlessweb and received this nice warning after placing the order:

OOPS! Seems Yahoo forgot to renew it’s security certificates. Do you have a process in place to secure to ensure your domain names and associated security certificates are always up to date? How about 3rd party monitoring of your service? Now this little warning is a nuisance compared to what happened to perl.com when a domain that was used for serving on the site was registered by a hacker.

No, this isn’t another “OMG, the facebook API IS AWESOME” post. I mean, it is, it’s pretty damn cool, I’ve played with it a bit this month. The real revolution with the facebook API are the server-side requests.

Traditionally widgets & plugins interfaced with social networks by placing snippets of HTML on profile pages. In the Facebook world no content can show up on a user’s profile without passing through Facebook’s servers first. Even your actual application pages must either be within an IFRAME or pass through Facebook. This process provides Facebook with an extraordinary level of control over what can and cannot be displayed on a user’s page. FB can perform a virus scan on all content and analyze any scripts for vulnerabilities or exploits. By directly serving content Facebook also eliminates cookie access — making it far more difficult to track or distribute data about their users.

Yet, the approach has it’s limitations for application developers. I tried briefly to build a “stalker tracker” application which using cookies would tell the user how many people regularly checkout their profile page. No matter what I tried, I couldn’t get access to the cookie without somehow initiating a click — rendering my application completely useless.

Why should you care? Well — advertising isn’t that much different from a traditional social networking widget — both are delivered via a snippet of HTML. Online ads have also been plagued by security issues this past year and I wouldn’t be surprised if the bigger players (Myspace, Yahoo, MSN, etc.) start to ask for server-side ad-requests soon. Server-side requests are the only way that a seller can technically guarantee the safety of third-party ads. Of course this will open up a world of technical challenges — server-side cookies storage, strict global latency requirements and a need for increased capacity to only name a few.