Why is it that open source initiatives win in the infrastructure software space? Why are the biggest clouds in the world built on Linux and open source hypervisors? Why are the leading cloud management platforms open source?
The answer is surprisingly simple. The solution has emerged over the past two decades thanks to Apache, Linux, MySQL and other enormously popular software open source initiative technologies. When you provide software to people who can and want to develop or maintain software themselves, the code needs to be open. If you provide software to a consumer or to a business, it’s about the service and the business benefits, not about openness per se. For a great read on open source business models, check out Cloudera's Mike Olson on how you can no longer win with a closed-source platform, and you can't build a successful stand-alone company purely on open source.
The openness requirement is a general phenomenon. Anyone can love chicken biryani, but those who enjoy cooking or who are professional chefs will want to know the specifics of how it is prepared and what it contains. The connoisseur and enthusiast require access to the details — not just to have the ability to do the same, but for the ability to verify the quality and to modify it to fit their needs. The casual consumer will simply focus on the experience and the benefits.
As the world embraces mobile apps, social networks and Big Data deployments, the underlying infrastructure is changing. We are now moving to cloud infrastructures from what we had in the client/server and the web era. In existing deployments, app software is the brittle part and resilience is a fundamental part of the hardware infrastructure.
This is now changing. We can assume that infrastructure will fail every now and then, so it’s up to the app developer to build resilience into the software layer. A very concrete example of this is Netflix. They have a deliberate process called the Chaos Monkey that brings down compute instances to push the apps (and their developers) to make sure they can survive such failures. The result is more uptime and better resilience.
The shift to a cloud paradigm for hardware and software infrastructure is disrupting the entire landscape. We see new development frameworks, configuration management tools, databases, cloud platforms, storage solutions, networking solutions, and so on. Nearly everything is up for grabs. And nearly everything is being snatched up by open source software initiatives.
Linux and the KVM hypervisors are becoming the industry norm. Cloud platforms such as Eucalyptus, OpenStack, CloudStack and OpenNebula are winning the IaaS battle. Storage and file systems such as Riak, Ceph and Swift are expanding their footprint. Cassandra, MongoDB, Couchbase, Neo4J and other databases are replacing closed source rivals. Today it's very difficult to bring a closed source product to market. No matter how good your technology may be, it is not interesting for the targeted users if they can’t "see under the hood," so to speak.
But open source initiatives are not a business panacea. Although the forefathers who wrote the open source licenses did not intend to stifle business opportunities, business models built around open source software initiatives are particularly tricky. There are numerous open source projects that reached immense popularity among users but failed to provide a significant revenue opportunity for its creators. If that happens, it helps little that your code is open. Free code exists, but free lunches don’t. Open source developers need to be paid just like closed source developers.
Many attempts have been made to create the perfect business model for open source initiatves. Two main categories have emerged: foundation-based open source projects with multiple productizers (think Linux, Eclipse, and OpenStack) and company-based open source projects with a singular vendor (think JBoss, MySQL, MongoDB, and Eucalyptus). Both open source initiative models can work wonderfully for vendors and customers, as well as users. The foundation model tends to not provide financial reward to the original innovator nor the software developers. It rewards the corporation that becomes the leading distributor. The singular model tends to reward the original innovator and the key employees of that company.
But in the modern cloud world, it’s not enough to have open source code and a functional business model. We must also take into account interfaces. We must learn to quickly couple and decouple software, and to move workloads from one infrastructure to another. To do this successfully, we need well-defined and commonly accepted APIs. You could say that in 80s, the common standard was the x86 architecture, in 90s open source software became the common standard, and now an additional 15 years later, it is APIs that form the common standard. Young software developers develop on EC2. EC2 is of course a service of Amazon Web Services, but more importantly it represents a new API way of looking at things. It’s the new frontier of openness. And for that reason, the leading software innovations of our times will be EC2 compatible.