It's all about big data, cloud storage, and a million gigabytes per day

My name is Ed Harris, and I am a development manager on the Bing team.  I’ve been at Bing since 2004, and worked on a lot of different teams – on the front-end/consumer-facing bits, deep in the platform space, and now in our infrastructure team.

 

The infrastructure team’s mission is to provide a stable foundation for Bing and other online applications to run upon.  Want to install an operating system and deploy a build on 50,000 machines with a couple of keystrokes?  That’s us.  Need a place to store a petabyte or two of business-critical data?  That’s us too.  Want to analyze every page of the web?  We’ve got what you need. 

 

The team I work on is called Cosmos.  Cosmos is the cloud storage and computing environment that Microsoft’s online properties use for data storage, analytics, and “Big Data” computation.  Every day, we load or generate petabytes of new data per day – that’s a million gigabytes per day.  In the online business, data is the most precious commodity of all – whether it’s our relevance experiments, our copies of the web, or a thousand other data sets that get curated, all of it is mission critical. 

 

The thing I enjoy most about being in the infrastructure team is the diversity of what we get to work on.  To build an efficient storage engine, you need brilliant system programmers who can make the disks do amazing and unnatural things.  On top of that, you have to layer data integrity – at the scale we operate, we have random bits flip from zero to one or vice-versa on a regular basis. So besides being fast, the storage layer needs to constantly scrub itself.  We also have folks who are world-class compression experts – to be cost effective; we need to squeeze the most data possible onto the disks, but not spend too much CPU doing it.  Up further in the stack is our execution and computation team.  They are able to take a user query, optimize it, and schedule it to run on tens of thousands of PCs.  You can write a three line query in our language (called SCOPE), that actually turns into a map/reduce job across twenty thousand servers.

 

The best thing about our team is the caliber of people that work here.  Our problem-space is on the cutting edge of cloud computing, and has attracted a group of rock-star engineers.  It’s an infectiously collaborative environment, and not a day goes by when I don’t have the opportunity to learn a new technique or algorithm.

 

The team takes its responsibility very seriously – we know that we are the custodians of Bing and Microsoft’s data assets.  But we also have a lot of fun as well.  Besides a lot of hikes and flash-mob Frisbee, we built our very own infrastructure keg fridge this year.  We now have three different kinds of beer and home-made root beer on tap.  Because we are geeks, we intend to fully integrate our InfraKeggerator into our datacenter automation this fall – we want biometric access control so other teams don’t steal our beer, alerts to fire when the kegs run low, 24/7 temperature monitoring, and minable data on which microbrews are most popular.  We also have a Xbox/Kinect setup in our kitchen and a pool table nearby.  It’s fairly common to see team members cutting loose after work – the floor reverberates on a regular basis from aggressive Kinect volleyball and dance central contests.

 

Working in this kind of environment – with a great set of hard problems to solve, a world-class group of coworkers, and a team that knows how to cut loose, is what I love about being at Bing. 

 

Interested in working on Bing’s Cosmos team? Click here to view a list of openings with this team: http://bit.ly/bingcosmos