Hi, I’m Dave

In late 2011, I joined the boys at Flyclops in order to help them complete their new game, Domino! A very polished take on the ages-old classic game dominoes, with a wonderful asynchronous multi-player twist, there was a glaring omission in the App Store that needed to be filled.

They had originally started the game in September, 2011, and expected the game to take a few weeks to a month to build. After a couple months, it became apparent that the game was much larger than they had estimated, and more revealing was how deficient Game Center actually was for building a complete, feature-rich multi-player game. Being friends with both of the founders – Jake O’Brien and Parker Whitney, via the storied coworking space IndyHall (where we still work) – I offered my assistance in building a backend API that would compliment the game in the way Game Center couldn’t.

What followed was a pretty amazing journey through 2012 – here’s our story.

TL;DR

For the impatient (yes, this is a lengthy post) the basics are this – we spent a plethora of time on a relatively simple game, making sure every piece was polished. We released before we felt it was ready (it will never be ready) and spent a lot of time iterating and making steady, constant improvements. We had an amazing first month in May of 2012, earning over $2000. We focus more of our efforts on customer service and engagement than anything else, bar none. By paying close attention to our customers, being incredibly personally involved in engagement, and providing a worthy experience, we topped $53,000 in revenue in December 2012, a mere 7 months later.

Numbers Porn

The revenue numbers are below, but here are some fun stats about play and our API server. We run our server stack on Amazon’s AWS offerings. Currently we employ:

• A single db.m1.large MySQL database, using Provisioned IOPS (currently, 2,000 IOPS/second of which we utilize around 1,200 and growing)
• One Elasticache cluster with a single cache.m1.small node
• A single Elastic Load Balancer
• Several Route 53 “alias” records against the load balancer
• Several CloudWatch alarms, all of which send SMS messages
• Anywhere between 3 – 7 app servers on c1.medium EC2 instances behind the load balancer

The API is:

• Written entirely in Python
• Utilizes the Flask web framework with the APY package for better class-based routing
• Uses SqlAlchemy as an ORM

On each API server, we run:

• nginx, with a memcached plugin that allows the iOS app to query the ElastiCache cluster directly through nginx for functions such as polling (skipping the web app entirely)
• uwsgi
• pyapns (absolutely brilliantly rock-solid asynchronous push-notifications server – we’ve sent over 250,000,000 push notifications with it)

We currently:

• Scale by hand when alarms go off. (OMG what a nightmare!! CloudFormation and AutoScale Groups are on the way, like, by the end of the week.)

Future engineering tasks:

• Autoscaling (did I mention??)
• Better caching
• Proactive push notifications (“Your game is about to expire!”)
• Real-time live gaming (we have “real-time” play, but it’s not a dedicated real-time platform)
• Blogging about it all

Our API currently handles:

• An daily average of 10,000 RPM (requests per minute) or ~165 requests/second
• Nightly averages of 15,000-20,000 RPM (300 requests/second)
• Spikes of over 28,000 RPM (over 460 requests/second)
• Around 14,400,000 API requests a day
• Upwards of 1/2 BILLION requests/month

An average night (busy time)

(Note: Those spikes are when alarms go off. They will go away with autoscaling.) Despite that much traffic, we maintain an average of 53ms per request. I honestly believe this is extremely important in maintaining customer satisfaction. If we moved from 65,000 daily player to 650,000 players, however, maintaining this could be difficult.

API server response times

Some totals for 2012

• 890,000+ downloads (97,000 active daily users)
• 3,957,000 matches (in the last week of December, we were adding ~43,500 a day)
• 176,700,000+ moves made
• 250,000,000+ push notifications sent
• Over 100,000,000 sessions since May, 2012 (905,000 daily sessions)
• 694 YEARS of aggregate play time as of December 31, 2012 (May 15, 2012, adding 6 years a day)

Total time spent in-app every day

Launch: May, 2012

Nine months after development began, we launched.

We had zero idea what to expect. Sure, we studied and planned, but you can’t know until it’s out there. All of our friends dutifully played. We knew almost no one would have friends who had the app, so we expected a lot of people to try – and then abandon – the game. Having build random matching in to the game, we would take turns watching the DB and immediately matching back anyone who started a random-match game, so as to appear that we had plenty of active players. Phone calls went back and forth at all hours.

Then things began to pick up.

Players!

Any game launch will, currently, attract some attention on the App Store. The attention ours attracted was beyond our expectations. We had over 10,000 signups in the first 3 weeks. The first day was 2. Then 0. Then 4. Then on May 15th, we hit 116 new signups.

Multiplayer signups by month

During the whole time, we were obsessing over random matches. We wanted anyone who played to feel like they almost instantly had someone to play against. On May 20 (my birthday, incidentally) we had 404 signups. Many people were buying the paid version of the app, and for the free version, people were just starting to look at ads in some consistent manner. Having launched our Facebook page, we began to interact with as many people as would watch, and hounded our friends on Twitter and Facebook to tell their friends.

By the end of May, we had culled over 10,000 multiplayer signups.

Half never played again.

Momentum

We quickly learned that our attrition rate was going to be high. Only about 20% of signups became active players, and that number has stuck relatively consistently from May through today. We can say:

• 12% of signups never opened the app more than once
• 20% never played past the first day
• Another 15% never make it past a week
• Fully 45% of players never make it past a month
• Consistently, ~20% of all signups play each day

It was a bit of a pill to swallow. We quickly hit an average daily signup of around 1,000 “multiplayer” players, but it would take a week or more to add 1,000 active players to our daily total. We constantly sweated our signup stats. We’d have 1100 players sign up one day, followed by 700, which we immediately took as a sign that our time in the Sun had come to an end.

To boot (see the chart above) our active monthly signups spiked in August and have never come close to reaching that number again.

Plodding growth

Quickly, we hit a bizarre trend.

Growth

Despite some minor daily fluctuations, our growth was constant. Amazingly constant. From July of 2012 through the middle of December, 2012, we would add between 1000-1500 multiplayer users. Every. Single. Day. Our daily-user trend is insane:

Holy straight line, Batman. (Active daily users)

Our active daily user graph is so insanely normalized that it almost feels fake. I still have no good explanation for this. I expected we would eventually hit a critical mass that would cause daily signups to spike – but we never did.

Multiplayer?

Right, so… we shipped the game with a single-player mode. Apparently… people like that a LOT.

For 2012, we had:

• 891,148 downloads
• 331,221 multiplayer signups
• 96,375 active daily users, but
• only 62,380 daily multiplayer users

Despite being a heavily multiplayer-focused game, only 37% of people who download the app ever sign up for multiplayer play!

As for daily users, 35% – over 1/3 of our players – never play a single multiplayer game. But they play, a lot, and they look at ads. My point? If you’re going to develop a multiplayer game, never underestimate the power of including a single-player version. I never will again.

The Money

We never really expected Domino! to make much more money than might pay for some beer and perhaps a new laptop each at the end of the year. Boy were we wrong.

Our revenue stream is very ad-heavy. Over the months we’ve added and removed ad providers based on performance, and have fine-tuned our strategy. That has led to marked gains. Currently:

• We utilize 5 ad providers and 1 ad aggregator
• Domino! downloads a config file on every open that tells which ad providers to show, and how each should be weighted in priority and use
• We edit the config file several times a day based on current eCPM projections
• We split ads between traditional pay-per-view ad providers and pay-per-install ad providers, based on current payouts-per-impression calculations

This process has been fine-tuned, and will shortly be completely automated based on some optimization algorithms. By trending ads towards those paying a higher eCPM (and adjusting multiple times throughout the day) we maximize our profit across multiple providers.

We also have revenue from the ad-free version of the game. However, sales made up less than 10% of revenue in December, with ads making up over 90%. 

By tweaking our ad providers constantly, our revenue growth has outpaced our user growth. By quarter:

Expenses

Our expenses are minimal at this point in time. They are:

• Server costs. In December, the total was $1,561.29. We expect January to be around $1650.
• Office space: Less than $700/month
• Our time.

In December, our costs before paying ourselves were 2.9% of revenue. We also pay:

• Approximately $15,000/month in “salaries”, taxes, and health care
• Under $250 on meals and entertaining

The Fiscal Cliff

Something to note – the 4th quarter of any year is amazingly, deceptively good for any company. Usually. For us it was great. Advertisers pay huge amounts of money to attract Christmas season buyers. January, however, is for crap.

For instance, here’s a quick look at our daily iAd take. Notice that at the very end of the year, we jump from requesting ~600,000 ads a day to over 1 million, but our fill rate drops from the 80% range to 17%.

The fiscal cliff

Our other ad providers look similar, however, we seem to be rebounding a bit.

That said, I would expect a 40-55% drop in revenue from December 2012 to January 2013.

Don’t get me wrong – $30,000/month during one of the worst months of the year is decent enough to run a small business. But now that we’ve tasted a $50,000+ month, we want it all the time, because we want to hire, and develop products faster. That said, it becomes glaringly obvious that it’s time for us to…

Diversify income sources

Our next update rolls out some sweet features, but after that, we have to concentrate on diversifying our income sources a bit. There will be in-app-purchase at some point in time (it is the way to make money on the App Store right now) and we plan to launch two more titles this year.

We also plan to port Domino! to Android. We’ll let you know how that goes, financially.

However, as we look to continue growing (which includes hiring a couple new people) and diversify and increase our revenue streams, we have to be extremely cognizant of user happiness. No matter what we do, we piss off some users with each update, but as long as the majority are happy, that’s good.

Adding something as seemingly a money-grab as in-app-purchase requires intense amounts of time being spent on making sure that extreme levels of value are being delivered to our customers.

So how did we do it?

How did a small game studio make good? We’re certainly not a million-dollar company… yet. But how did we get to be much more than self-sustaining?

I have no idea.

But it seems that these factors had something to do with it:

1. Constant communication with our players. Constant. We engage users however and wherever we can – often directly in-app. Server hiccup? Here’s what happened. The entire app stops working for 30,000 people because we missed a change in Facebook’s API? A full-on customer service assault.
2. Constant personal communication with players. Every regular player knows our first names (and perhaps our last names). Many people start matches with us, so they can play the developers. We have conversations with them through in-game chat, email, Facebook, Twitter, and more. Parker has over 80 ongoing matches right now, because he was silly enough to include his username in a Facebook post somewhere.
3. The personal touch – which goes along with numbers 1 and 2. When Parker lost his first match against a player publicly, he posted a picture of his shame.
   

   Parker’s Shame

4. Updates that don’t have to be constant, but contain both obvious features, as well as more subtle updates that make the game flow that much better. The “Next Game” button we recently added had a HUGE impact on player engagement, whereas tweaking our rematch process may have gone unnoticed, but suddenly people weren’t starting multiple matches against each other unnecessarily.
5. Polish. Lots of polish.
6. It’s a social game. Not social as in social media, but social as in fun between friends. We have a lot of people tell us they love playing family across the country and across the world as a way to keep in touch. That’s awesome.
7. A snappy feel – helped by both snappy servers and good UX.
8. Responsive, friendly, personalized customer service.
9. The realization that without constant attention (and perhaps even with) this could all go away in an instant.
10. Constantly making sure that we are, in our own honest opinions, the best dominoes game on the App Store.

If you took ANY theme out of that, it’s how important customer services is. We have rabid fans because of it, and what’s worse, we genuinely care about their happiness. I mean, that’s a good thing, but now we’re invested

Conclusion…?

So that’s a lot, but hopefully it’s let you in on a little of what’s been an incredible year for us. We hope to continue doing whatever it is that we’re doing well, while also fixing whatever we’re doing poorly – all while having just a stupid amount of fun.

It’s not impossible to create a profitable small business from iOS titles, but it’s certainly not easy (or fast) either. Moving from barely-sustaining to the next level is going to be like the blind leading the blind, so we’ll just keep our eye on customer satisfaction.

Here’s to a wonderful 2013 for Flyclops and for anyone who read down this far. You certainly deserve some good karma for that!

The common way Google presents is as follows:

There is one glaring problem with this – for me anyway. bash and sh don’t work with fractions of a second (or floating point/decimal numbers at all) so the difference will always be a whole number.

Seeing “2 seconds” is fine, but I’m interested in it being “2.255322 seconds.” A better solution:

First, we change the date format to

date +%s%N

Adding the %N gets us nanoseconds instead of seconds – as whole numbers, that is. (Note: %N doesn’t work on BSD variants.) So we divide by 1 billion, and pipe the calculation string through bc, a unix calculator, and it will return our seconds difference to 8 decimal places.

This changes my output from:

Backup complete in 2 seconds

to:

Backup complete in 2.83732828 seconds

Much better

The API server is write-heavy. It’s constantly saving state, setting people as online or offline, writing data about games, etc. There are more API writes than reads. I’m using SQLAlchemy and have highly optimized read queries… but writes… well…

I was finding that I was severely IO bound on writes. InnoDB, it seems, likes to write to the transaction log on disk every operation. Amazon’s RDS instances use EBS backing stores, which are nice and write-slow. At the lightest 3 updates per second, I was seeing per-operation disk writes spike up to 600ms and beyond.

Wut. Slow.

I tweaked two settings:

The big one was innodb_flush_log_at_trx_commit – it seems that InnoDB synchronously writes to the transaction log (on disk) for every transaction commit – which would have been several times a second. Turning off innodb_flush_log_at_trx_commit stopped that from happening. Is that OK to do? Yup – because InnoDB will flush the log EVERY SECOND anyway. And it will do it asynchronously.

Now, I can handle the loss of data from 1 second. Some apps can’t. Your mileage will vary.

Turning off innodb_support_xa also moves from a 2-stage commit to a 1-stage commit. Again, it’s giving up a little transactional security, but I don’t particularly need it.

After… well, I’ll let this 24 hour log do my ‘splainin.

Much better.

See? Much better. Since then, I’ve been trailing this sort of graph, where DB access is almost constant time:

So letting InnoDB write to the transaction log asynchronously is HUGE in an RDS, write-heavy environment. Or so it would seem.

UPDATE: As some commenters have pointed out, the GM release of Mountain Lion did enable the Notification Center APIs for non-App-Store apps. While this is a positive move, there are still a few restricted APIs (iCloud, and the argument can be made that this is a positive or negative thing) but sandboxing and OS-refusal to run unsigned apps continue to be hard pills to swallow.

UPDATE 2: It looks like sandboxing is so restrictive, that Apple is using private exceptions and security scoping in their apps that aren’t available to developers. It’s easy to play by a different set of rules if you’re Apple, but it’s further indication that you can’t play on par in the OS anymore.

Gruber did a nice writeup about his experience with the Apple Event For One and new features in 10.8, but makes the following statement that I find to be relatively uninformed:

This default setting benefits users by increasing practical security, and also benefits developers, preserving the freedom to ship whatever software they want for the Mac, with no approval process.

It’s true, you know, that there is a security interest here. However, it is crippling to development. Here are a couple reasons why:

1. Unsigned software won’t run unless you hop in to the OS settings. For practical purposes, it makes signing almost necessary. The problem is that any old signing won’t do – I can’t get a Verisign cert and use that. Apple holds all the keys here.
2. Signed software can’t load unsigned (compiled) plugins. There are a plethora of apps that you probably use on a daily basis that load plugins. They range from launchers (like LaunchBar, Alfred, etc.), to IM clients, professional tools for design, music, and movie creation and editing, and so on. Many apps make use of plugins to extend functionality and separate features in a logical fashion.
1. The flip side to this is that you can make all plugins Javascript or Python or Ruby, but then not only are they slower, but they’re completely open-source. The license may not be open source, but code is definitely available for all to see.
2. While developers may be able to statically link to their own plugins, any software that encourages plugin development from the community will suffer.

Apple’s strategy has been termed, in our office, “boiling the frog.” It’s the old adage of a frog tossed in to boiling water will hop out, but put it in room-temperature water with the water temperature turned up slowly will not notice and will eventually boil to death. Apple’s strategy is easy to see.

I can understand people thinking I’m overreacting, but here’s where Apple is boiling the frog – now it’s signed apps, but you can still run unsigned ones with effort. If Apple is willing to give out signing certificates for free, requiring all apps to be signed is easily imagined in 10.9. But oh, there’s more – some of the more interesting new frameworks in 10.8, like Notification Center and iCloud, are only available to apps sold through the App Store.

The nail in the coffin is this – starting March 1, all apps submitted to the Mac App Store will have to be sandboxed. To quickly review what that means, apps will only have access to their install directory and perhaps their preferences directory, and will be unable to do things like access the general file system, hook in to OS-level functions, and so on. This effects the above-listed apps, as well renders useless apps like Little Snitch, Dropbox, Transmit, Git and SVN apps, across-file search functions in text editors like TextMate and Sublime Text, and so-on.

If all apps have to be signed, and in order to sign them Apple holds the keys, and in order to use all the frameworks you have to use the app store, and if you sell your app in the app store it is required to be sandboxed, well, you can see where this is going. The water is getting a lot hotter for Mr. Frog.

If you’re still a little confused about sandboxing, you can think of it like iOS on the iPhone and iPad – each app runs in its own space, can’t communicate with other apps, doesn’t have access to lower level OS services, and so on.

Of course, you can, when submitting an app to the App Store, write an essay (not kidding) about why your app needs to be able to step out of the sandbox in certain ways. However, you’re at the mercy of Apple also thinking you should have such permissions, that your app should be allowed to do what you developed it to do… And even if you do get approval, by the way, it can be revoked at any time.

There is going to be an awful lot of talking about these new policies in the coming months. I’m hoping things end up getting hashed out in the favor of developers – and the users who rely on their apps daily.

Scientific Revolution by ~pod-tanwen on deviantART

In terms of economic impact, the Information Revolution makes the Industrial Revolution look like child’s play. The political machines of individual nations are still trying to grasp the full force of the barely-regulated globalization impact of the Information Revolution. (It may be this barely-there-regulation that has gotten us so far so quickly.) All of this, and the Information Revolution is likely still in the early stages of what future generations will look back upon as our awakening.

Update: Video of me giving a talk on this post at Ignite Philly 9 after the jump. (Thanks to Technically Philly!)

But it’s not alone. We get to live through two more revolutions as well – slower starters, but that may have a bigger impact on humanity than the Information Revolution will ever have. (Granted, we will see that none of these three are mutually exclusive – nonetheless, for the purposes of this essay, we’ll define them individually.)

The Energy Revolution

Unlike the Information Revolution, which very few were able to foresee, and therefore try to stifle, the Energy Revolution is well within sight. It will fight an uphill battle against entrenched global energy corporations, but progress can only be slowed – it cannot be stopped. The Energy Revolution is interestingly quiet in its first stages, but so was the Information Revolution. Here is the future that awaits us.

There is a tectonic shift in the very fabric of energy production. We currently produce most of our energy at great cost to the environment which surrounds us – either by burning the finite resources of fossil fuels and coal, or by creating highly radioactive waste that can melt down and have a murderous impact that lasts generations. The future will harness the seemingly limitless energy provided to us on a constant and ongoing basis by nature itself. Instead of fighting against nature, we will embrace it.

I will skip wind energy in this conversation. Despite it being more efficient than nuclear and fossil fuel generators (not to mention cheaper), it will be relegated to small tasks in the future. It will always have a place (think of a wind turbine on top of a sailboat) but its place will be small.

The future is in solar energy. We are just tapping the well of this almost limitless power. Solar energy is already cheaper (PDF warning) than nuclear energy, and will be cheaper than coal-fired plants in under 10 years. If you take in to account the economic impact of the pollution and health issues caused by burning coal, solar energy is already cheaper.

Here are some quick numbers according to Scientific America: the Sun hits the earth with 89 petawatts of power every day. That’s 6,000 times more energy than all of civilization uses. Enough energy hits the Earth in 14 seconds to power the entire world for a day. In 88 minutes, as much as the world consumes in a full year. “In 112 hours – less than five days – it provides 36 zettajoules of energy – as much energy as is contained in all proven reserves of oil, coal, and natural gas on this planet. If humanity could capture one tenth of one percent of the solar energy striking the earth – one part in one thousand – we would have access to six times as much energy as we consume in all forms today, with almost no greenhouse gas emissions.”

Scientific America went ahead and did the math and discovered that the cost of solar power collection is following Moore’s Law – and has been for over 31 years. By 2030, solar energy production will cost half of coal production – with virtually zero carbon emissions. We currently emit 35 gigatons of carbon emissions each year by burning fossil fuels for energy – by one study, that’s enough to fill the balloons of the Macy’s Thanksgiving Day Parade over 2 billion times. With solar costing half of what fossil fuels do, we’ll be able to eliminate a huge chunk of that 35 gigatons of annual carbon pollution.

Of course the biggest factor sitting between us and energy independence is not solar technology, but battery technology. It takes a long time to charge batteries, and they have limited capacity. They’re also toxic to produce. Never fear – insane strides are being made to battery technology too. For now, it’s Lithium-ion. Northwestern University just discovered that a small change in the way we produce Lithium-ion batteries – basically drilling holes in-between the layers – increases the storage capacity of a battery by a factor of 10, and decreases the charge time again by a factor of 10. They expect the production changes to take at most 5 years to reach market, which means our iPhones will take 5 minutes to charge and last for 5-10 days of constant use. To put it another way, this simple change makes electric cars 100% viable. A Nissan Leaf would take 3 minutes to charge on a Level 3 charger, and have a range of 730 miles. What comes after the Lithium-ion is anyone’s guess, but at least we know smart people are working on it.

The potential of Northwestern’s Lithium-ion breakthrough is huge. Aside from electric cars, it means home-based batteries that store solar-collected energy could withstand a few days of low sunlight. It means that even a small break in a huge storm-system could mean fully-recharged batteries. This is huge, as solar cells only collect energy during the day, and only if it’s sunny out. Better batteries mean less direct sunlight is required. Solar power on every building starts to become viable.

But hey – just in case you still think this direct-sunlight thing is a huge detractor of solar cells, fear not. Idaho National Labs have created a thin solar film that actually holds nano-coils that vibrate at the natural frequency of a type of radiation that floods the Earth from the Sun both day and night, sunny or cloudy. To boot, they’re 80% efficient (as opposed to current cells that are 40% efficient). Solar cells that can collect energy efficiently both night and day? It’s not a pipe-dream. It’s the future.

As Moore’s Law continues to act on the price of collecting energy from the Sun that hits the Earth whether we use it or not, we mathematically head towards a point where the cost of producing energy nears zero. Since energy is nine-tenths of production, the ability to produce food and goods opens up to even the poorest among us. Where desalination is currently inefficient and energy-hungry, near-limitless near-free energy would make desalination (in current technology) affordable to those who desperately need it. Limitless cheap energy and worldwide access to clean water changes not “the game”, but “every game.”

The best part is, we will be alive to witness its birth.

The Medical Revolution

The third in our series of revolutions is medicine. What we take for medical technology now will look antiquated and even barbaric within our lifetimes. Mapping the human genome was more important than any of us could have imagined. Still in its infancy, understanding what the 25-40,000 genes in our bodies do is just starting to emerge. The human body has about 3-4 billion base pairs in its DNA, but only around 4 percent of those effect gene function, which is where the future of medicine lies.

Viruses and bacterial infections are basically epic battles between genes. Right now we flail in the dark trying to combat viruses, and our ability to beat back a bacterial infection results in mutated, drug-resistant bacteria capable of withstanding the best of the fight we have. But if we can understand the actual battle happening between our genes and those of viruses and bacteria, we can build targeted weapons against disease that nothing can combat.

Let’s start with genetic mapping. When we started mapping the human genome in 2001, we thought it would take 30 years and cost more money than we could imagine. The initial costs were astronomical – the cost of mapping one Megabase (one million base pairs) was over $10,000. But unlike the cost of solar energy which is following Moore’s Law, the cost of genetic sequencing and mapping has fallen far, far faster. The big change happened in 2008, and by July of 2011, the cost per Megabase had fallen to $0.10. The cost of mapping a Megabase of DNA has dropped by 100,000%. The cost of mapping an entire genome? It’s down from $100,000,000 to well below $30,000.

Of course, in order to truly understand a genome, you have to understand patterns in genetic sequences. That requires a lot of data. Some genetic banks have stores of DNA samples in the hundreds of thousands but more will be required. Still, this hasn’t stopped science, and genetic pattern matching has already begun.

But lets back up for a second. Imagine the cost of mapping a genome persists on the same path. The cost of mapping a genome would drop to 30 cents in 3 years. It’s likely to not continue on this trend, but the price will drop significantly, and in the near future it will become cost effective to map everyone’s DNA. Sample sizes will increase to the millions if not billions, and pattern matching will be able to pinpoint the exact genetic sequences of everything from cancer to the common cold. Several companies are already working on mapping the genetic profiles of cancers. What this means is targeted therapies that replace nuclear options like chemotherapy. Not only will such mapping lead doctors to be able to tell what drugs a cancer might be resistant to and which would be incredibly effective, but will even eventually lead to custom-made, custom-tailored drugs that are specific to not a kind of cancer, but to a particular cancer in a particular individual. Tailor-made drugs with DNA-specific targeting mechanisms mean cancer and other diseases become a nuisance rather than a death sentence – if we just don’t prevent them in the first place.

Then there’s the insanely cool and barely understood stem cell. Much speculation has been made of its potential, but the debate has been bogged down in the politics of abortion. Not to fear, we’ve already moved past embryonic stem cells in many therapies. Not only can we use your own stem cells now, but we can use pig cells too, why not?

Recently the Pentagon has been dabbing in regenerating muscle lost by soldiers to IEDs and other battlefield injuries. They’ve been testing regenerative muscle replacement on soldiers suffering “massive loss” of tissue – defined by a 25% tissue loss (but including up to 90% loss) – preventing the need for amputation and giving soldiers 25% or more improvement in physical function. This isn’t the future – it’s happening now. A mere 4 years after a company was able to regenerate a fingertip in an experiment, the first trials are happening on soldiers. Certain proteins and cells from a pig’s bladder can create the scaffolding that will cause your own stem cells to flood a particular area and start regenerating lost tissue. It only takes a single surgery and some fairly intense physical therapy to strengthen the newly regenerated muscle cells, but the procedure has already been a success on 4 soldiers. The trial wraps up in less than 2 years, and baring any unforeseen circumstances (none of which have raised their heads since this began in 2007) this will become the standard treatment for traumatic loss. Consider the consequences: 50% of battlefield injuries in Iraq and Afganistan are these types of “massive loss” injuries. Amputations could become a thing of the past. Bring that technology in to the civilian world and amputations from car accidents to diabetes will all but go away.

But enough about muscles. What about the rest of your body? Got you covered there too. In what can only be described as a blurring of the lines between traditionally held beliefs about the fabric of life itself, we can now use your own stem cells to regrow replacement body parts. The basics are this – create a scaffold, dip it in some stem cells, program it to be a thing (your choice – a heart? bladder? How about a urethra, windpipe, kidney…?) and then once it’s grown in a lab, transplant it back in to your body. Your second, growing heart will actually start to beat before it is ever implanted in your body.

The benefits are obvious. First there is the enormous shortage of donated organs – and in order to receive one, often someone else has to die. Transplanted organs require a lifetime of massive drug therapy to keep you from rejecting the implanted organs. Even with drugs, many transplants fail, and the patient ends up dying anyway. Now take an organ built with your own stem cells – it’s yours, free and clear. Science already has the ability to build your own replacement organs, which will one day do away with the need for organ donors. Your stem cells will become life-regenerating factories, capable of regrowing anything broken. Born blind? New eyes. Heart failure? New heart. And for one guy who had throat cancer, a new windpipe. The best part is that since the body part is grown of your own body, it will never be rejected – no post-transplant drugs, just a whole new part that’s fresh and ready to take on the role of its previous, failing brother-in-arms.

This isn’t the future. It’s now. Within a decade, it will be common-place. Within 25 years, we may no longer have a need for a donor registry at all.

So whether you get a genetically hand-tailored drug to destroy the cancer in your kidney, or you simply replace a kidney that’s beyond repair… you can see how mind-bending the near-future of medicine is – and only imagine what the not-too-distant future will hold.

Besides, it will be nice to be able to grab some replacement body parts, even if we may have already found a way to slow aging drastically. 300 years is a long time to live with all the same organs.

What a time to be alive!

We get to live through not one, but three concurrent revolutions, each of which will have a more changing effect on humanity over the course of a few decades than the rest of human history put together. This is an insane time to be alive. At 33 years of age, I imagine what the scientific landscape will look like when I’m 60 is almost impossible to imagine. While there is still political and religious strife, extreme poverty, starvation, war, and genocide, each will be impacted positively by the three revolutions. When the world has access to information, energy, and medical breakthroughs beyond comprehension, the landscape of humanity will change forever – for the better. I guarantee it.

Nina Garcia on WhoSay

You’ll find me, Flyclops, and so much more!

1. Cloudfiles images are insanely easy to manage, and any server can be spun up from any snapshot. This is so much easier than trying to create a custom AMI that it’s not even worth discussing.
2. The initial buy-in is incredibly low – a single small server will run you about $12 on Rackspace, while running ~$70 on EC2. Yes, I know that the hardware is basically apples-to-oranges, but we’re talking about initial buy-in here, not what $70 will get you. I can run a web server box and a PostgreSQL box separately for well under a single small instance on EC2.
3. The support – oh the support. Zip, zilch, nada on EC2… while on Rackspace – a human in under 20 seconds. It’s glorious.

However, I grew to love the idea of throw-away servers with EC2. Alestic, Ubuntu, and other well-known institutions provide ready-to-go AMIs, and we spent a little bit of time writing code that would take a vanilla API, run a shell script, and turn it in to whatever it needed to be in no time flat. We plugged this scripting in to our CI server, and would have a successful build of our production branch of code spin up new servers with new code, drop the old ones, and no one would be the wiser.

Since my new business is fledgling, I didn’t have the money to run multiple EC2 instances with RDS as another expense. Rackspace has always had a place in my heart since my days of running $10k/month in servers from one of my old jobs. While their cloud offerings are certainly less mature than Amazon’s, the price was right, and it was pretty easy to get going.

I knew there had to be a way to get the kind of flexibility I had with EC2 on Rackspace, even though they built their offerings completely oppositely – servers are supposed to be long-running, save state, etc.

Enter Cloud Load Balancers

As soon as Rackspace opened their Cloud Load Balancer product to beta (and I discovered a Python API to it) I knew I had found what I needed. The idea was this – write a script against their API that would spin up new servers, drop them behind static load balancers, drop out the old servers, and no one would be the wiser. Rackspace’s load balancers even had a huge advantage over Amazon’s Elastic Load Balancers – I could point an A record at them.

There were, however, a few hurdles I hadn’t expected.

Amazon has this neat little feature where you can pass in data to a newly booting server. You simply have your rc.local file call a special IP address, download the data, and in our case, execute said data as a script. That would get you in the door and your code running.

Rackspace was more difficult – much more. They allow you to pass in files to a newly booting server, but those files are then marked as read-only with no execute permissions. So passing in an rc.local file would mark that file as un-executable, and it wouldn’t run.

Hey cron, can you help me out?

The solution is an ugly one, but here it is. There are three files I pass in to a newly booting web server.

/etc/rc.local

/etc/rc.local.stage1

And the crux of the whole thing:

/etc/cron.d/initialrunner

Let’t take a look at these files. rc.local does as it should – it has some scripting that runs whenever the server reboots – in this case launching django-ztask – our async backgrounding app build on top of 0MQ (but that’s another story for another day).

rc.local.stage1 is run once and only once, and is the script that sets up our whole server. I’ve only put the first few lines, but you can see what they do – update the server, install git, and as you can imagine, do a bunch of other things (load in our git public keys, install apache, side-load our apache conf files, etc.).

But like I said – all three of these files will be marked as read-only and neither script will be executable. So we toss in a script in to /etc/cron.d/initialrunner that does 4 things the first time you reboot the server:

1. Echo “hello world” in to a log file as an indication to me that it was run
2. Change the permissions of /etc/rc.local and /etc/rc.local.stage1
3. Execute /etc/rc.local.stage1 to set up the server, sending output to a log file

Simple enough.

Putting it all together

The build script (written in Python) is somewhat long, but I’m going to put it here, and explain below what it’s doing. This is the script that my CI server runs after a successful unittest build of code committed to our production branch.

build.py

So what’s going on here?

The if __name__ == '__main__': section at the bottom of the script is what gets run when the script is called from a shell. This is what it does in order:

1. It instantiates a cloudservers.CloudServers object instance, (from the official python-cloudservers Python API package, available here) passing in our username and API key
2. Calls create_server, passing in the name of the initial image to use and the size of the server.
3. Calls wait_for_server which waits for the server to be done setting up. This takes the keyword argument with_url_ping that we’ll use later to not only wait for the server to respond, but acknowledge our site is up as well.
4. Sleeps for 20 seconds (Rackspace seems to have a few places where it needs a couple seconds to catch up with itself)
5. Calls the the update method on the server instance, which has the happy side effect of REBOOTING the server. That – guess what – causes our /etc/cron.d/initialrunner script to be called, which starts building our server out!
6. Wait for the server again, waiting for the ping to return true this time, which means that Apache is up, mods have been enabled, it’s configured properly, and serving our content.
7. … At this point in time, our REAL script also uses paramkio (SSH library) to run server migrations and run our unittest suite against the installed code, but I left that out.
8. Creates an instance of cloudlb.CloudLoadBalancer object, (from the official python-cloudlb Python API package, available here) passing in our username, API key, and which of the data centers to use. (Hint: there are only two.)
9. Call replace_servers, which takes the new servers, appends them to the load balancer, waits 20 seconds, and removes the old servers. It then deletes the old servers from existence.

It may seem somewhat complicated, but it’s not much different than using BOTO to do the same thing on EC2. There’s also nothing stopping you from modifying this script slightly to create multiple servers each time. The replace_servers function is already aware of replacing all servers with all new servers (notice it takes our new server in an array, expecting to load as many new servers behind our load balancers as we ask it to).

Any downsides?

Yeah, a couple. Every now and again the server build will just die. Seems to be a Rackspace issue and I need to get in touch with support about it, but since the final wait_for_server call will fail over by deleting our hanging server without touching existing servers, it’s not a big deal and the build script is just re-run.

Also, Rackspace load balancers work in an odd fashion – they can only run a single port. They can share IP addresses (see the replace_servers function which, if finding no existing load balancers, will create them for you) so your DNS doesn’t have to do anything fancy, but to run a website with port 80 and 443, you have to pay for two load balancers. Still relatively cheap, but an annoying downside.

Conclusion, maybe?

With a little love from the official API packages and Rackspace’s new Cloud Load Balancers, I was able to achieve the same immutable-server effect that I like from EC2, for a fraction of the price. While many people like long-running servers and using things like Capistrano or Fabric for deployment, I prefer that servers build new each time, test themselves, then simply replace outdated servers with not a clue that it’s happening to the end user. The beauty of this is that with a single line of code change, I can update what image new servers are built with, instantly taking advantage of an updated OS release.

This method isn’t for everyone, and it’s a lot more popular with EC2, but it’s possible with Rackspace, and I’d prefer to stick with Rackspace (and reap the benefits of their amazing support) and still get what I want.

(Image courtesy of clix on sxc.hu)