The AI sector has made tremendous progress regarding AI model capabilities. Large AI models have revolutionized scaling by enabling narrow, general AI systems that can handle multiple tasks. As AI models grow exponentially in parameter size, they are expected to improve continually, leading to increased demand for computational resources in model training. Amongst these resources, Graphics Processing Units (GPUs) have emerged as strategic and competitive resources amongst AI companies and even countries.
GPUs are specialized processors that efficiently handle graphics manipulations and parallel data processing. Their ability to handle large amounts of processed data in parallel makes them more favorable than Central Processing Units (CPUs) in AI applications. While CPUs are well-suited for tasks that involve sequential processing and single-threaded applications, GPUs excel in accelerating high-performance computing tasks, especially those involving parallel processing.
Navigating the Surge in Demand for AI Compute
The exponential growth and widespread acceptance of AI technologies have resulted in an unprecedented demand for GPUs. Chat GPT, which has been in existence for a mere two months, has already garnered an estimated 100 million monthly active users. As a result of this, along with the tremendous compute power required for training advanced models, the availability of high-end GPUs has been significantly impacted.
High-performance GPUs have become indispensable assets in various industries. It is critical to note that the current lead time for Nvidia's H100-based servers is between 36 to 52 weeks, indicating a severe supply shortage. Companies are compelled to sign long-term contracts for GPUs, which often results in more computing power than necessary and wasted resources. This situation emphasizes the urgent need for decentralized computing solutions facilitated by blockchain technology, which can offer a more efficient and adaptable compute resource market.
The rapid growth of large artificial intelligence (AI) models has resulted in an exponential demand for Graphics Processing Units (GPUs). This demand is driven by supply chain and geopolitical issues and has made the cost of AI training immense. Typically, developers use NVIDIA's advanced GPUs, such as enterprise-grade GPUs, to train their models, which can cost up to $10,000-$12,000 per unit. OpenAI's GPT-3 required a whopping 1,000 GPUs, while Stability AI needed 4,000 GPUs. This has cost these companies millions just to train their models on their dataset. It's safe to say that the high cost of GPU usage for AI training is a significant challenge for developers.
The competition to develop AI is being driven by the industry's workhorses and GPUs such as NVIDIA (document number 1) and (document number 2) clusters. To fully capitalize on the revenue potential of AI, centralized cloud technology platforms have been actively accumulating GPUs. This is done to gather computing power for their own use and, in some instances, to offer computing power for rent to customers who are willing to pay the price.
Developers face a harsh reality when it comes to acquiring the computational power required for training machine learning models. They are forced to either invest in expensive hardware or pay exorbitant prices to cloud providers. Due to the sheer expense, the development of such models has been monopolized by tech giants who can afford it, leaving smaller developers out of the game.
Compute is the new oil in the Gulf States?
King Abdullah University of Science and Technology (KAUST) in Saudi Arabia reportedly plans to purchase over 3,000 H100s to construct a supercomputer named Shaheen III, expected to be operational by the end of 2023. The university's researchers, mainly focusing on LLM, are Chinese nationals facing restrictions accessing the US due to their universities' limitations. Meanwhile, the Technology Innovation Institute in Masdar City, UAE, which developed the Falcon LLM, is also acquiring computing resources from NVIDIA. Additionally, Abu Dhabi-based G42 has teamed up with US-based Cerebras to acquire up to $900 million worth of the company's Wafer-scale compute systems to construct nine interconnected AI supercomputers. This marks just the beginning, with significant future spending anticipated in this field.
Decentralized Compute Networks
The AI industry is constantly racing to develop advanced AI systems, significantly increasing the demand for computing resources. Therefore, it is imperative to have scalable computing power to meet the ever-growing demand for AI. Decentralised compute networks offer a sustainable and robust solution to the challenges of centralisation and security in this field.
Decentralised compute networks use blockchain technology to offer computing services in a secure and decentralised manner. Unlike conventional centralised systems, these networks distribute the workload among multiple nodes, allowing parallel processing, improved scalability, and optimal resource utilisation.
Decentralized compute systems effectively tackle the inefficiencies in the GPU market by establishing a secondary market where computing resource owners can sublease their excess capacity in real-time. It is a viable solution for individuals with high-performance gaming PCs or small businesses with underutilized server capabilities to offer their surplus processing power to others, thereby minimizing resource wastage and mitigating supply constraints.
Using blockchain technology to coordinate this excess supply ensures that compute resources become almost cost-free. The pooling of resources provides competitive pricing and enhances accessibility to computational power while compensating network participants for sharing their excess compute.
Apart from offering competitive pricing and improved accessibility, decentralized compute has a significant advantage in preventing censorship. By distributing control over computational resources, these systems guarantee that no single entity can impose its norms and values on AI models. In turn, this prevents large corporations from monopolizing the AI landscape.
Decentralized compute networks offer many advantages for developing AI solutions.
Cost Efficiency: Decentralized networks should utilize low-cost and underutilized computing and storage resources to reduce costs, even if it requires adding more machines.
Resilience Against Censorship: Decentralised computing offers a significant advantage in its ability to withstand censorship, providing a much-needed counterbalance to the alarming trend of concentrated AI development among a select few dominant technology firms.
Resource Utilisation: Decentralized computing protocols are leveraging untapped computing power to construct cutting-edge open-source models through incentivization.
Decentralized compute networks have witnessed remarkable growth since 2023, with surging interest in AI and groundbreaking developments in the AI sector. The market capitalization of decentralized compute network tokens has surged by a staggering 507% in the past year, reflecting this growth. During this period, Render Network has emerged as the clear market leader, boasting a market cap of more than US$4.19 billion. Notably, Akash Network has seen impressive growth, skyrocketing from $487 million to $1.3 billion today, representing a whopping 1,217% increase.
Landscape of Decentralised Compute Networks
As the world is rapidly realizing the increasing significance of AI in our daily lives and workflows, the crypto industry is also acknowledging the need for AI to play a more crucial role. Projects are being launched quickly to compete with the centralized and closed-source AI companies that dominate the traditional AI landscape. Despite OpenAI being the most well-known AI team, several other teams, organizations, and research labs are working to build the most efficient and performant large language model (LLM) possible with today’s technology. Crypto + AI projects are determined to create decentralized and open-source AI that can operate on the global distributed ledgers we know as blockchains.
High-Level Overview
The AI computing landscape is transforming with the rise of decentralized compute solutions. AI models necessitate vast computational resources for both model training and inference execution. The demand for these resources has escalated sharply as AI models become increasingly complex. OpenAI, for example, witnessed its compute requirements doubling every three and a half months between 2012 and 2018. This exponential growth in demand has intensified the competition for computational resources and driven up costs, compelling some crypto-mining companies to repurpose their GPUs for cloud computing services.
Nevertheless, the scarcity of state-of-the-art GPUs, particularly Nvidia's offerings, remains a significant challenge for AI training. The high demand for these GPUs has led to long wait times, and lengthy contracts may lock companies into more compute capacity than they require, exacerbating market inefficiencies. Decentralized compute platforms emerge as a solution to these challenges, creating a secondary market that allows for the immediate leasing of excess compute capacity, thereby increasing supply and accessibility while also providing competitive pricing.
Source: State of AI Report
Decentralized compute offers the crucial advantage of being resilient against censorship, which serves as a counterbalance to the increasing concentration of AI development among a few large technology firms. This concentration raises concerns about these entities potential to dictate the norms and values embedded in AI models. Moreover, the push for regulations by these firms may stifle innovation outside their control. Decentralized compute platforms such as Akash democratize access to computational resources, thereby ensuring a more equitable playing field for AI development.
Conclusion
Decentralized computing networks are making significant strides. However, one of the biggest challenges they face is the effective management of GPU supply and demand. Despite the ongoing GPU arms race, as demonstrated by OpenAI's recent limitations due to shortages, these networks have the chance to demonstrate their superiority in terms of cost-effectiveness, resistance to censorship, and better accessibility.
The approaching year is absolutely pivotal for decentralized computing platforms to establish an unwavering user base and unequivocally underscore their superiority over conventional data centers. In light of the increasingly evident and tangible benefits of decentralized computing, there is an ever-growing confidence in the blockchain's potential to revolutionize AI development, making it more accessible and equitable for all.