Hi, I’m Dave
In late 2011, I joined the boys at Flyclops in order to help them complete their new game, Domino! A very polished take on the ages-old classic game dominoes, with a wonderful asynchronous multi-player twist, there was a glaring omission in the App Store that needed to be filled.
They had originally started the game in September, 2011, and expected the game to take a few weeks to a month to build. After a couple months, it became apparent that the game was much larger than they had estimated, and more revealing was how deficient Game Center actually was for building a complete, feature-rich multi-player game. Being friends with both of the founders – Jake O’Brien and Parker Whitney, via the storied coworking space IndyHall (where we still work) – I offered my assistance in building a backend API that would compliment the game in the way Game Center couldn’t.
What followed was a pretty amazing journey through 2012 – here’s our story.
For the impatient (yes, this is a lengthy post) the basics are this – we spent a plethora of time on a relatively simple game, making sure every piece was polished. We released before we felt it was ready (it will never be ready) and spent a lot of time iterating and making steady, constant improvements. We had an amazing first month in May of 2012, earning over $2000. We focus more of our efforts on customer service and engagement than anything else, bar none. By paying close attention to our customers, being incredibly personally involved in engagement, and providing a worthy experience, we topped $53,000 in revenue in December 2012, a mere 7 months later.
The revenue numbers are below, but here are some fun stats about play and our API server. We run our server stack on Amazon’s AWS offerings. Currently we employ:
- A single db.m1.large MySQL database, using Provisioned IOPS (currently, 2,000 IOPS/second of which we utilize around 1,200 and growing)
- One Elasticache cluster with a single cache.m1.small node
- A single Elastic Load Balancer
- Several Route 53 “alias” records against the load balancer
- Several CloudWatch alarms, all of which send SMS messages
- Anywhere between 3 – 7 app servers on c1.medium EC2 instances behind the load balancer
The API is:
- Written entirely in Python
- Utilizes the Flask web framework with the APY package for better class-based routing
- Uses SqlAlchemy as an ORM
On each API server, we run:
- nginx, with a memcached plugin that allows the iOS app to query the ElastiCache cluster directly through nginx for functions such as polling (skipping the web app entirely)
- pyapns (absolutely brilliantly rock-solid asynchronous push-notifications server – we’ve sent over 250,000,000 push notifications with it)
- Scale by hand when alarms go off. (OMG what a nightmare!! CloudFormation and AutoScale Groups are on the way, like, by the end of the week.)
Future engineering tasks:
- Autoscaling (did I mention??)
- Better caching
- Proactive push notifications (“Your game is about to expire!”)
- Real-time live gaming (we have “real-time” play, but it’s not a dedicated real-time platform)
- Blogging about it all
Our API currently handles:
- An daily average of 10,000 RPM (requests per minute) or ~165 requests/second
- Nightly averages of 15,000-20,000 RPM (300 requests/second)
- Spikes of over 28,000 RPM (over 460 requests/second)
- Around 14,400,000 API requests a day
- Upwards of 1/2 BILLION requests/month
(Note: Those spikes are when alarms go off. They will go away with autoscaling.) Despite that much traffic, we maintain an average of 53ms per request. I honestly believe this is extremely important in maintaining customer satisfaction. If we moved from 65,000 daily player to 650,000 players, however, maintaining this could be difficult.
Some totals for 2012
- 890,000+ downloads (97,000 active daily users)
- 3,957,000 matches (in the last week of December, we were adding ~43,500 a day)
- 176,700,000+ moves made
- 250,000,000+ push notifications sent
- Over 100,000,000 sessions since May, 2012 (905,000 daily sessions)
- 694 YEARS of aggregate play time as of December 31, 2012 (May 15, 2012, adding 6 years a day)
Launch: May, 2012
Nine months after development began, we launched.
We had zero idea what to expect. Sure, we studied and planned, but you can’t know until it’s out there. All of our friends dutifully played. We knew almost no one would have friends who had the app, so we expected a lot of people to try – and then abandon – the game. Having build random matching in to the game, we would take turns watching the DB and immediately matching back anyone who started a random-match game, so as to appear that we had plenty of active players. Phone calls went back and forth at all hours.
Then things began to pick up.
Any game launch will, currently, attract some attention on the App Store. The attention ours attracted was beyond our expectations. We had over 10,000 signups in the first 3 weeks. The first day was 2. Then 0. Then 4. Then on May 15th, we hit 116 new signups.
During the whole time, we were obsessing over random matches. We wanted anyone who played to feel like they almost instantly had someone to play against. On May 20 (my birthday, incidentally) we had 404 signups. Many people were buying the paid version of the app, and for the free version, people were just starting to look at ads in some consistent manner. Having launched our Facebook page, we began to interact with as many people as would watch, and hounded our friends on Twitter and Facebook to tell their friends.
By the end of May, we had culled over 10,000 multiplayer signups.
Half never played again.
We quickly learned that our attrition rate was going to be high. Only about 20% of signups became active players, and that number has stuck relatively consistently from May through today. We can say:
- 12% of signups never opened the app more than once
- 20% never played past the first day
- Another 15% never make it past a week
- Fully 45% of players never make it past a month
- Consistently, ~20% of all signups play each day
It was a bit of a pill to swallow. We quickly hit an average daily signup of around 1,000 “multiplayer” players, but it would take a week or more to add 1,000 active players to our daily total. We constantly sweated our signup stats. We’d have 1100 players sign up one day, followed by 700, which we immediately took as a sign that our time in the Sun had come to an end.
To boot (see the chart above) our active monthly signups spiked in August and have never come close to reaching that number again.
Quickly, we hit a bizarre trend.
Despite some minor daily fluctuations, our growth was constant. Amazingly constant. From July of 2012 through the middle of December, 2012, we would add between 1000-1500 multiplayer users. Every. Single. Day. Our daily-user trend is insane:
Our active daily user graph is so insanely normalized that it almost feels fake. I still have no good explanation for this. I expected we would eventually hit a critical mass that would cause daily signups to spike – but we never did.
Right, so… we shipped the game with a single-player mode. Apparently… people like that a LOT.
For 2012, we had:
- 891,148 downloads
- 331,221 multiplayer signups
- 96,375 active daily users, but
- only 62,380 daily multiplayer users
Despite being a heavily multiplayer-focused game, only 37% of people who download the app ever sign up for multiplayer play!
As for daily users, 35% – over 1/3 of our players – never play a single multiplayer game. But they play, a lot, and they look at ads. My point? If you’re going to develop a multiplayer game, never underestimate the power of including a single-player version. I never will again.
We never really expected Domino! to make much more money than might pay for some beer and perhaps a new laptop each at the end of the year. Boy were we wrong.
Our revenue stream is very ad-heavy. Over the months we’ve added and removed ad providers based on performance, and have fine-tuned our strategy. That has led to marked gains. Currently:
- We utilize 5 ad providers and 1 ad aggregator
- Domino! downloads a config file on every open that tells which ad providers to show, and how each should be weighted in priority and use
- We edit the config file several times a day based on current eCPM projections
- We split ads between traditional pay-per-view ad providers and pay-per-install ad providers, based on current payouts-per-impression calculations
This process has been fine-tuned, and will shortly be completely automated based on some optimization algorithms. By trending ads towards those paying a higher eCPM (and adjusting multiple times throughout the day) we maximize our profit across multiple providers.
We also have revenue from the ad-free version of the game. However, sales made up less than 10% of revenue in December, with ads making up over 90%.
By tweaking our ad providers constantly, our revenue growth has outpaced our user growth. By quarter:
Our expenses are minimal at this point in time. They are:
- Server costs. In December, the total was $1,561.29. We expect January to be around $1650.
- Office space: Less than $700/month
- Our time.
In December, our costs before paying ourselves were 2.9% of revenue. We also pay:
- Approximately $15,000/month in “salaries”, taxes, and health care
- Under $250 on meals and entertaining
The Fiscal Cliff
Something to note – the 4th quarter of any year is amazingly, deceptively good for any company. Usually. For us it was great. Advertisers pay huge amounts of money to attract Christmas season buyers. January, however, is for crap.
For instance, here’s a quick look at our daily iAd take. Notice that at the very end of the year, we jump from requesting ~600,000 ads a day to over 1 million, but our fill rate drops from the 80% range to 17%.
Our other ad providers look similar, however, we seem to be rebounding a bit.
That said, I would expect a 40-55% drop in revenue from December 2012 to January 2013.
Don’t get me wrong – $30,000/month during one of the worst months of the year is decent enough to run a small business. But now that we’ve tasted a $50,000+ month, we want it all the time, because we want to hire, and develop products faster. That said, it becomes glaringly obvious that it’s time for us to…
Diversify income sources
Our next update rolls out some sweet features, but after that, we have to concentrate on diversifying our income sources a bit. There will be in-app-purchase at some point in time (it is the way to make money on the App Store right now) and we plan to launch two more titles this year.
We also plan to port Domino! to Android. We’ll let you know how that goes, financially.
However, as we look to continue growing (which includes hiring a couple new people) and diversify and increase our revenue streams, we have to be extremely cognizant of user happiness. No matter what we do, we piss off some users with each update, but as long as the majority are happy, that’s good.
Adding something as seemingly a money-grab as in-app-purchase requires intense amounts of time being spent on making sure that extreme levels of value are being delivered to our customers.
So how did we do it?
How did a small game studio make good? We’re certainly not a million-dollar company… yet. But how did we get to be much more than self-sustaining?
I have no idea.
But it seems that these factors had something to do with it:
- Constant communication with our players. Constant. We engage users however and wherever we can – often directly in-app. Server hiccup? Here’s what happened. The entire app stops working for 30,000 people because we missed a change in Facebook’s API? A full-on customer service assault.
- Constant personal communication with players. Every regular player knows our first names (and perhaps our last names). Many people start matches with us, so they can play the developers. We have conversations with them through in-game chat, email, Facebook, Twitter, and more. Parker has over 80 ongoing matches right now, because he was silly enough to include his username in a Facebook post somewhere.
- The personal touch – which goes along with numbers 1 and 2. When Parker lost his first match against a player publicly, he posted a picture of his shame.
- Updates that don’t have to be constant, but contain both obvious features, as well as more subtle updates that make the game flow that much better. The “Next Game” button we recently added had a HUGE impact on player engagement, whereas tweaking our rematch process may have gone unnoticed, but suddenly people weren’t starting multiple matches against each other unnecessarily.
- Polish. Lots of polish.
- It’s a social game. Not social as in social media, but social as in fun between friends. We have a lot of people tell us they love playing family across the country and across the world as a way to keep in touch. That’s awesome.
- A snappy feel – helped by both snappy servers and good UX.
- Responsive, friendly, personalized customer service.
- The realization that without constant attention (and perhaps even with) this could all go away in an instant.
- Constantly making sure that we are, in our own honest opinions, the best dominoes game on the App Store.
If you took ANY theme out of that, it’s how important customer services is. We have rabid fans because of it, and what’s worse, we genuinely care about their happiness. I mean, that’s a good thing, but now we’re invested
So that’s a lot, but hopefully it’s let you in on a little of what’s been an incredible year for us. We hope to continue doing whatever it is that we’re doing well, while also fixing whatever we’re doing poorly – all while having just a stupid amount of fun.
It’s not impossible to create a profitable small business from iOS titles, but it’s certainly not easy (or fast) either. Moving from barely-sustaining to the next level is going to be like the blind leading the blind, so we’ll just keep our eye on customer satisfaction.
Here’s to a wonderful 2013 for Flyclops and for anyone who read down this far. You certainly deserve some good karma for that!