Welcome to Tech blog at Zynga
We’ve been heads down here at Zynga over the last few years but we’ve always been big fans of open source and open information. We’ve learned and leveraged a lot from others and we hope that this site will be found useful in turn. To those that have made their way here, a hearty hello– we welcome your comments, ideas and feedback. Zynga is a social game company and our mission is to connect the world through games — this has become quite broadly known. Perhaps less known is that Zynga is hugely focused on technology. The largest group of people at Zynga are software engineers and if you combine all the rest of the folks involved in software development, you’d get a hefty majority of the company. We’ve had to develop some interesting solutions along our growth path and we’ll be sharing some ideas and thoughts along the way. Here are some of the top areas we focus on — expect to hear more from us on these.
It is probably no surprise that we focus on games but perhaps less obvious is how much we research this area. We’ve developed our own game engine in Flash and we continually look at ways to experiment and eke out more performance, easier development or new game mechanics. We also try to keep pace with many other game engines in the space and we’re constantly looking at new innovations. While most of our games are simple to play, we spend considerable effort to make them fast to load, quick to respond and able to display rich artwork quickly within the confines of the browser. We also do heavy work on mobile game engines and we are big fans of Cocos2D— so much so that we now have two of the heavy contributors to that project working here at Zynga.
Games have traditionally been an area of study for computer science because of the challenges of multiple users interacting simultaneously. Many of the traditional solutions such as having separate servers to partition players don’t work for us. Social gamers can play across a broad graph of connected friends and trying to chop those graphs into particular servers doesn’t work. When you combine that problem with a massive workload, interesting technology comes out. Our workload differs from many of the largest websites on the planet — many of them are read oriented (search, shopping, news) or write oriented (photo or video upload) but our workload is almost entirely update oriented. Our players expect to interact with the game world and make changes as fast as they can click. This difference challenges many traditional databases, web servers and architectures and our solutions to these problems are unconventional.
Our business is based on virtual goods so we naturally focus considerable effort there ranging from management of massive economies of hundreds of millions of players to combating fraud. We’ve developed sophisticated models to predict demand and to understand how our players value virtual goods. We work tirelessly to create a safe and easy purchase experience with world class customer service to make sure our players feel comfortable transacting with us. When you add in trading and gifting, we often are managing economies that rival the scale of real world economies. We believe strongly that technology can help our players adopt virtual goods as a regular part of their entertainment.
We think there is considerable new ground in the interaction between virtual goods and real goods. Loyalty programs and crossover between real and virtual are fascinating business areas that we want to facilitate with technology.
Zynga has been written about as being a metrics driven culture and whenever someone joins Zynga, they often comment on how much the company lives and breathes metrics. As such our systems get heavy use and we’ve developed a system from very early on that relies upon deliberate structured instrumentation, massive data aggregation across multiple data centers and fast loading into one of the worlds largest columnar databases. All this means that we can get answers about events happening throughout our games across datacenters around the world within seconds or a few short minutes. Lest we come across as only driven by numbers, we also have significant investment in user surveys, focus groups and community forums. We’ve developed survey and experimentation systems that allow us to get feedback from our players based on a wide set of criteria.
Because our business can take off with a single feature or a single game, scaling our infrastructure presents out of the ordinary challenges. We are one of the world’s largest users of public and private cloud computing and we rely heavily on the elasticity of the cloud environment to handle sudden changes in demand. We’ve developed our architecture to work more effectively in a cloud environment and to be tolerant of a heavy dose of disruption. This robustness has enabled us to keep running even during significant cloud outages that more greatly impact other cloud businesses. Our toolchain for provisioning and managing this dynamic environment includes a mixture of open source, commercial tools and a good does of our own development. We continue to invest in this area but we’ve been able to leverage cloud computing to handle the extreme spikes that might require 1000 servers to be deployed in a single day. Without automation and a horizontally scalable architecture, we’d have now way to keep up with those spikes.
We use a linux, apache, memcached, mysql, php (LAMMP) stack throughout our games. We are heavy users of mysql, linux and apache but we focus our development efforts in particular on php and memcached and have contributed back to the open source communities in those areas. CouchBase, which is in our opinion an evolutionary successor to memcached, came about as a collaboration between a project we were developing combined with work that the CouchBase team was doing and we are running thousands of instances in production in place of memcached. We have done considerable work on PHP and PHP extensions in particular such as APC and AMF in order to improve backend efficiency.
That is brief overview and there is a lot more so poke around and let us know what you think. If you’re curious about something, shoot us a question. If you’re working on something that you think we should know about or just want to get more involved, let us know. We’re a pretty friendly bunch of folks. We play games all day.