The story of how Google could have killed Facebook with the flick of a switch

As we near the end of this decade, and more importantly the end of the hell that was 2020, I realised two things. First, I survived catching Covid-19, and secondly that I had a good story about the history of Silicon Valley that I’d never written down. I’m still functional, so here, for perpetuity, is my tale.

Back in 2013 I was working in the Ads Interfaces organisation at Facebook, building mostly front end products (other people did the AI, database etc). We had an application called Power Editor which was the kitchen sink of products, and 25% of all Facebook revenue depended on it working. Every single thing you could do with ads on Facebook was supported in Power Editor (we called it P.E. for short). This made it huge, hugely complex and pretty user hostile. However our big spending customers were forced to use it as it was the only way they could efficiently scale – PE had a lot of cool tools for duplication, permuting your ads and working in batches of thousands of changes.

By 2013, PE was creaking under its own weight (about 150k lines of front end JavaScript), and no one wanted to work on it. We had half of one (awesome) engineer supporting it, and she could just about keep it working. I was looking for an opportunity to become a manager, and my manager Brian and I decided that building a team to properly support PE would be a good idea.

I looked into the code base, and was shocked to find that the entire application depended on a technology called WebSQL. It only ran on Chrome, and Google had deprecated WebSQL over a year earlier. I kind of flopped back in my chair, dragged my manager to a room, and told him that Google could shut off 25% of Facebook’s revenue, and lose us all our large accounts, by turning off WebSQL in Chrome, and it could happen any time.

This became a closely held secret in Facebook Ads leadership. We didn’t want to take any chance that word our this vulnerability could get back to Google. They had already deprecated WebSQL, and other browsers had removed it. They would have been well within their rights to just flip a feature flag in Chrome and do the same.

There was a whispered joke among those who knew about it that Google never had to bother building Google +, they could shut us down by changing one boolean in a database.

We quickly put in place a team of 5 engineers and one PM to work on it, with me managing it and coding probably 75% of the time. The plan was fairly complex. There was no easy way to get us off of WebSQL without a full rewrite of 150k lines of JavaScript. We couldn’t just build a new application from the ground up, that would take years to support all the features, and Google could turn off WebSQL at any time. Also, PE was falling apart – it was blocking the entire company from shipping any new ads products at all.

So, we decided to first make it not fall apart with a lot of performance and reliability work. Next, we had a long running project called PE Live, where we took each subset of code and made it read from the live API rather than locally from the WebSQL database. To unblock the other teams we would rewrite the whole thing in ReactJS, whereas it was then built using two frameworks that we had sunset called UkiJS and BoltJS.

This whole process took over three years. By the time it was complete in 2016, we had improved Power Editor so much, with better features, more stability, speed and ease of development for partner teams, that over 50% of all Facebook’s revenue was spent through it. The team grew to 13 engineers, with lots of help from dozens more across the Ads organisation building new APIs and infrastructure to support our work.

Google could have killed it at any time, and there were no complete alternatives for our customers – some third party applications existed, but they were even buggier than PE, were always late with new features (we didn’t have to wait for a new API to be public, they did), and often each specialised in a subset of the features. 50% of our revenue disappearing over night could have happened. That it didn’t and that we moved mountains of code doing the horrible, inglorious work of rewriting hundreds of thousands of lines of spaghetti code in production while people used the product, while also building an infinitely better product, is the most satisfying period of professional work I’ve ever been fortunate enough to experience.

A huge thanks to all the amazing people I worked with on that crazy project, you’re the best team I ever worked with, and I’d hop into the trenches with you again any time!

Power Editor as I left it in 2016, after 6.5 years at Facebook.

[Edit: So this ended up on the front page of Hacker News, and there’s much more conversation about it over at https://news.ycombinator.com/item?id=26086056 ]

8 thoughts on “The story of how Google could have killed Facebook with the flick of a switch

  1. Damn, what a story! That’s a tough cookie, and a great success to launch a complete rewrite in just 3 years!
    Wondering were there any “high leverage” (Killing 10 birds with 1 stone type of stuff) things that increased the team productivity during that time, especially architecture-wise and organisation wise?
    Any “oh damn we didn’t think of that” moments where some obscure edge case from old version was forgotten about?
    Also what was the flow like? Find all hotspots, rewrite the DB code, then rewrite the same component in react and launch it live (was there interop with old version?) or did you wait for all chunks to be completed to be tested?

  2. Thanks for sharing this story, Shane! BrunoF here. At the time I was mostly focused on Ads Manager and didn’t realize the risk we were facing. Thanks for catching and driving it to success. Ads Manager nowadays is basically the evolution of PE Live. Your work continues to power large portions of Facebook revenue.

    1. Thanks Bruno! Yeah we kept the risk pretty quiet, if it had gotten back to Google it…. wouldn’t have been good for any of us 🙂 Thanks for all the awesome work over the years, hope you’re doing well

  3. Pingback: Efrens' Blog
  4. Thanks for the trip down memory lane! I joined FB in June 2012 on the Preferred Marketing Developer (PMD) team, and worked with the 3rd party developer ecosystem to build atop Facebook’s Ads, Pages, Insights, and Graph APIs.

    I actually recall sitting at a pod near yours on the 3rd floor in Hanover Quay in Dublin during one of your trips back! We were probably in some of the same meetings, or in the micro kitchen with the keg, but that’s so long ago it’s hard to recall properly. But definitely some of the best years for FB.

    A lot of the issues you mentioned were an opportunity for 3rd party developers – but many settled in thinking FB would not invest in fixing the most basic issues with ads interfaces. Those that thought beyond that, and innovated beyond that, still exist today. Many others fell by the wayside as FB’s own tools improved, but that was to be expected.

    I took a trip down memory lane a while back, which you might enjoy (https://www.developerecosystem.com/posts/how-facebook-created-their-mobile-advertising-business/). All those ad units, all those targeting options, wayyyyy tough for advertisers to figure out how to use them. Fun times for sure.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s