Why is it that open source wins in infrastructure software? Why do websites run on Cassandra and MongoDB? Why are the biggest clouds in the world built on Linux and open source hypervisors? How come the leading cloud management platforms are open source?
The answer is surprisingly simple. The solution has emerged over the past two decades thanks to Apache, Linux, MySQL and other enormously popular software technologies. When you provide software to people who can and want to develop or maintain software themselves, the code needs to be open. If you provide software to a consumer or to a business, it’s about the service and the business benefits, not about openness per se.
It turns out that the openness requirement by professionals is a generic phenomenon. Anyone can love chicken biryani, but those who enjoy cooking or who are professional chefs will want to know the specifics of how it is prepared and what it contains. The connoisseur and enthusiast require access to the details — not just to have the ability to do the same, but for the ability to verify the quality and modify to fit their needs. The casual consumer will just focus on the experience and the benefits.
As the world goes wild on mobile apps, social networks and Big Data deployments, the underlying infrastructure is changing. From what we had in the client/server and the web era, we are now moving to cloud infrastructures. In our existing app deployments, software is the brittle part and resilience is built into the hardware infrastructure.
In the new world, this changes. It must be assumed that infrastructure will fail every now and then, and it’s up to the app developer to build resilience into the software layer. A very concrete example of this is Netflix, where they have a deliberate process called the Chaos Monkey that brings down compute instances to push the apps (and their developers) to make sure they can survive such failures. The result is higher uptime and greater resiliency.
The shift to a cloud paradigm for hardware and software infrastructure is causing disruption across the entire landscape. We see new development frameworks, configuration management tools, databases, cloud platforms, storage solutions, networking solutions, and so on. Nearly everything is up for grabs. And nearly everything is being snatched up by open source software.
Linux and the KVM hypervisors are becoming the industry norm. Cloud platforms such as Eucalyptus, OpenStack, CloudStack and OpenNebula are winning the IaaS battle. Storage and file systems such as Riak, Ceph and Swift are expanding their footprint. Cassandra, MongoDB, Couchbase, Neo4J and other databases are replacing closed source rivals. In this situation, it is very difficult to bring to market a closed source product. No matter how good your technology may be, it is not exciting for the targeted users if they can’t “see under the hood,” so to speak.
But open source is also not a business panacea. Although the forefathers who wrote the open source licenses did not intend to stifle business opportunities, it so happened that business models built around open source software are a particularly tricky subject. There are numerous open source projects that reached impressive popularity among users but failed to provide a significant revenue opportunity for its creators. If that happens, it helps little that your code is open. Free code exists, but free lunches don’t. Open source developers need to be paid just like closed source developers.
Many attempts have been made to create the perfect business model for open source software. Two main categories have emerged: foundation-based open source projects with multiple productizers (think Linux, Eclipse, and OpenStack) and company-based open source projects with a singular vendor (think JBoss, MySQL, MongoDB, and Eucalyptus). Both models can work wonderfully for vendors and customers, as well as users. The foundation model tends to not provide financial reward to the original innovator nor the software developers. It rewards the corporation that becomes the leading distributor. The singular model tends to reward the original innovator and the key employees of that company.
But in the modern cloud world, it’s not enough to have open source code and a functional business model. We must also take into account interfaces. We must learn to quickly couple and decouple software, and to move workloads from one infrastructure to another. To do this successfully, we need well-defined and commonly accepted APIs. You could say that in 80s, the common standard was the x86 architecture, in 90s open source software became the common standard, and now an additional 15 years later, it is APIs that form the common standard. Young software developers develop on EC2. EC2 is of course a service of Amazon Web Services, but more importantly it represents a new API way of looking at things. It’s the new frontier of openness. And for that reason, the leading software innovations of our times will be EC2 compatible.