The evolution of computing has been marked by a series of paradigm shifts, from mainframes to personal computers and now to the cloud. Currently, the most common and reliable way to deploy web infrastructure is to become a paying customer of a large company like Amazon or Google and write infrastructure as code to spin up a constellation of interoperable services on their machines. This approach ensures that the infrastructure you create is secure, scalable, and efficient, and it is the preferred choice of many organizations worldwide.
The current dominance of cloud hyper scalers, who hold the lion's share of the world's computation, is not a natural monopoly but rather a temporary state. The open-source community is working tirelessly to develop a decentralized solution to revolutionize the industry again. A decentralized solution will enable users to move freely between service providers, breaking the chains of lock-in. This will lead to increased price competition and allow businesses to negotiate on their terms with their cloud providers. The current state of deep entanglement and dependence on a single entity will be a thing of the past.
The Lessons of Linux
The potential of a decentralized cloud is best understood by looking at the history of the Linux operating system. It is a well-known fact that it took a decade for the Bells Labs developed software Unix to evolve into Minix, and another decade for Minix to become Linux. And finally, after yet another decade, Linux saw widespread enterprise adoption, becoming the obvious choice for developers.
It is indisputable that Linux open-source software currently dominates the market, running on an impressive 80% of public servers and a staggering 100% of supercomputers. It's the go-to choice for all developers when they spin up an EC2 (Elastic Compute Cloud), an exceptional service provided by Amazon that enables users to run applications in AWS’s public cloud effortlessly.
The success of Linux can be attributed to the power of compounding gains achieved through shared and open development practices. Developing an operating system from scratch is a monumental and intricate task that requires millions of hours of specialized labor, research, and testing. However, after years of incubation fueled by government funding and hobbyist contributions, Linux finally reached a usable state, and the benefits began to multiply.
Open-source operating systems have become an indispensable part of modern technology. The Linux kernel, which forms the core of a computer's operating system, serves as a common ground for people to collaborate and continually improve upon. Every user of the Linux kernel is responsible for identifying and reporting any bugs they find, and some even contribute patches to fix them. It is safe to say that open-source codebases are incredibly resilient and nearly impossible to eliminate.
Open-source software support is undeniably cheaper than closed-source software support plans. This is simply because creating an open-source software product costs much smaller than building and maintaining a custom operating system. The cost of Linux to a business is nominal, as it can be denominated in fractional developers lending a hand or in support plans, which is particularly beneficial for companies without in-house kernel expertise.
On the other hand, the cost of building and maintaining a custom operating system is exorbitant, requiring the controlling entity to cover the entire lifetime cost. This means a high consumption cost is necessary to make the endeavor economically viable. Multiple closed operating systems competing with each other must fund their own development and find sufficient market share to recoup their substantial investments, which explains the current state of the market to a substantial extent.
The cloud is essentially another operating system
The "cloud" is much more than just a tool for abstracting away when and where jobs run, their resources, and how they interact - it is another operating system in itself. Like operating systems, clouds are highly sophisticated and complex resource managers, schedulers, and security providers. It's not just about running a program on your computer or setting up an open-source Kubernetes cluster; the cloud operates on a much larger scale. AWS, GCP, Azure, and DigitalOcean are the mainframe computers of our time, and they are independently maintained within each company. This is the reality of the cloud - a technology that has revolutionized how we think about computing and data management.
It is crucial to remember that open-source codebases are nearly indestructible, as their knowledge and power are stored in git repositories rather than in human processes that can fail. OpenStack, developed by Rackspace and NASA in the 2010s as an open-source cloud stack that could transform any data center into its own AWS, is still under active development by companies, primarily in China. While some claim that it's dead, it is far from it. As a result, open-source contenders have a significant advantage and a high probability of success since they can always be reborn, even when their original business models have collapsed.
Correcting the mistakes made in our previous attempt
The failure of OpenStack was inevitable due to the self-serving interests of companies who insisted on creating their distributions of the software, leading to a fragmented and competitive marketplace rather than a collaborative one. The major vendors released their OpenStack distributions, causing further fragmentation and fierce competition, which hindered progress and development. The lack of standardization across OpenStack distributions made deploying and managing clouds consistently challenging for users. In contrast, the complexity of OpenStack made it even more challenging to install and operate, especially when compared to more polished and user-friendly public cloud offerings from Amazon, Microsoft, and Google.
Without a strong central governing body, driving a cohesive vision and rallying the community around shared priorities becomes daunting. Unfortunately, many companies that initially supported OpenStack reduced their investments or pulled out entirely. However, this time, we have a secret weapon: nodes, consensus, block rewards, public goods, and ecosystem alignment research. With 15 years of running blockchains, node distribution is a mostly solved problem, and we can certainly overcome standardization.
We must determine the right public goods mechanisms and align users to unite people around protocols. This will put an end to endless forking, competition, and defection. Governance and shared roadmaps are problems that we have been continuously hacking into, and we are making great strides. The shared state and shared value of blockchains bind communities together and force them to collaborate in ways that easily forkable open-source repositories simply cannot. With the financial primitives and collaboration tooling nearly ready to work, we already have promising starts. However, we must continue to work hard and push forward to achieve our goals.
A journey towards a successful outcome
Decentralized cloud infrastructure is not a question of if but when. We are closer than we realize to a decentralized cloud go-to-market, which could bring hundreds of billions of dollars to the ecosystem if we capture the market like Linux did. The cloud markets are growing at 11% annually, and AI adoption can only accelerate that. Bitcoin ETFs look like lunch money in comparison.
However, some gaps are holding us back from winning right now, and the area we're most sorely lacking is sensible product decisions. The current state of Web3's attempts at decentralized cloud services is questionable, with limited product-market fit. Most people use these protocols only to game their token incentives, which is unfortunate.
We must step up and compete with a mature, intimidatingly well-funded industry. Technically or theoretically, doing storage, computing, or CDN-speed delivery is not enough. This will not attract paying users, and token inflation will run out, leading to failure.
We need to design hybrid centralized/decentralized SLAs. We need to secure it with reliability incentives, security attestations, zero-knowledge proofs, fully homomorphic encryption, computation fraud proofs, governance protocols, and more. Blending our home-grown incentive structures with Web2 and cloud hyperscaler monitoring, compliance, and security technologies is crucial.
Most of us in this segment have prototypes and MVPs, but we need to do the hard thing: rival the incumbents by working with users and adapting to their needs thoroughly. The thing that attracts paying users will be the product iteration cycle, evolving our offerings to meet their needs and going to market like a Web2 cloud company.
We have to listen to user feedback and use reward mechanisms with care. Fortunately, the Web3 community is almost infinitely patient to test and give feedback, as we all want (need?) this to work. Look closely at projects like Spheron Network, Estuary, and ArDrive, and ask yourself if you'd choose them over Dropbox, especially if your job depended on it. Now, think about how much work needs to be done to get there.