Decentralized storage and commercial scenarios in Web3

Written by CAM @ Zee Prime Capital

翻译：Scarlett LI @ Contributor of PermaDAO

审阅：Xiaosong Hu @ Contributor of PermaDAO

Storage is a critical part of any computing stack. (Note: A machine stack refers to nested layers of software, where each layer depends on the previous layer and provides different features and services.) The machine stack includes layers such as operating systems, applications, libraries, frameworks, and hardware. In computer programming, handling and managing the computer stack is important for developing efficient, reliable, and secure software applications); Nothing would be possible without this essential element. Through the continuous advancement of computing resources, a large amount of excess and underutilized storage has been generated. Distributed storage networks (DSNs) provide a way to coordinate, leverage, and turn these potential resources into productive assets. These networks have the potential to bring the first true vertical commerce into the Web3 ecosystem.

History of P2P**

With the advent of Napster, the history of true peer-to-peer file sharing really began to go mainstream. While there were early methods of file sharing on the internet before that, the eventual mainstream was MP3 file sharing brought by Napster. From this initial starting point, activity in the world of distributed systems exploded. The centralization of the Napster model (for indexing) makes it easy to shut down for breaking the law, however, it lays the foundation for a more robust approach to file sharing.

(Note: Napster is an online music-sharing service founded in 1999.) It allows users to share and download music files over a peer-to-peer (P2P) network. With Napster, anyone can upload and download MP3 files and share their music file library with other users. At the time, Napster was seen as a revolutionary platform because it lowered the barrier to entry for sharing and downloading digital music. However, Napster also faces problems such as copyright infringement and fair remuneration for artists. Although Napster was shut down by order of a competent judge in 2001, it had a profound impact on music sharing and the digital music industry. ）

The Gnutella protocol follows this groundbreaking and has many different effective front ends that leverage the network in different ways. As a more decentralized version of the Napstereqsue query network, it is more robust to censorship. Even then, it went through censorship. AOL acquired development company Nullsoft and quickly realized its potential and almost immediately stopped distribution. However, it had already spread to the outside and was quickly reverse-engineered. Bearshare, Limewire, and Frostwire are probably the most compelling of these front-end applications you've probably come across. Where it ultimately failed was bandwidth requirements (resources were very limited at the time) and a lack of dynamism and content guarantees.

! [Decentralized storage and commercial scenarios in Web3] (https://img-cdn.gateio.im/webp-social/moments-69a80767fe-e915eb3580-dd1a6f-69ad2a.webp)

Source: Zee Prime Capital

Remember this? If not, don't worry, it's been reborn as an NFT marketplace...

! [Decentralized storage and commercial scenarios in Web3] (https://img-cdn.gateio.im/webp-social/moments-69a80767fe-98a27d9eb6-dd1a6f-69ad2a.webp)

Source: Zee Prime Capital

Next is BitTorrent. This brings an upgrade due to the bidirectional nature of the protocol and its ability to maintain distributed hash tables (DHTs). DHTs are important because they serve as decentralized versions of the ledger, storing the location of files and making them discoverable by other participating nodes in the network.

With the advent of Bitcoin and blockchain, people began to think about how to use this novel coordination mechanism to connect potentially unused resources and commodity networks. This was followed by the development of DSNs.

Perhaps to the surprise of many, the history of tokens and P2P networks goes back much further than Bitcoin and blockchain. The pioneers of these networks quickly realized the following:

• It is difficult to monetize the useful protocols you build due to forks. Even if you monetize through the frontend and serve ads or take advantage of other forms of monetization, forking may weaken your advantage.
• Not all usages are created equal. In the case of Gnutella, 70% of users don't share files, and 50% of requests are for files hosted by the top 1%.

Power-law

(Note: Power law, also known as "Stirling's law", is a concept in statistics and mathematics that describes phenomena in which there is a nonlinear relationship between intensity and frequency in certain phenomena.) For example, when we look at the comments on a blog post, we see that most of the comments have few replies, but a very small number of comments have many replies. This phenomenon can be observed in many areas, such as the number of followers in social networks, the distribution of the population in cities, the volume of goods in the transportation of goods, etc. Many real-world phenomena and distributions have been found to be well described using power laws. For example, Pareto's theorem is a common power law that describes the phenomenon that 90% of wealth is in the hands of 10% of people. Power laws are often closely related to research in network science, complex systems, biology, economics, sociology, and other fields, which are important for understanding and predicting many real-world phenomena. ）

How to solve these problems? For BitTorrent, it is the torrent ratio (download/upload ratio), and for others, it is the introduction of the original token system. Most commonly referred to as credits or points, they are allocated to incentivize good behavior (promoting the health of protocols) and network management (such as regulating content in the form of trust ratings). For a deeper look at the history of it all, I highly recommend these articles by John Backus (now deleted and available from the web archive):

• The FAT protocol is not new
• What if BitTorrent has a token?

Interestingly, DSN was part of Ethereum's original vision. The so-called "Holy Trinity" aims to provide the necessary suite of tools for the flourishing of the world's computers. Legend has it that this was actually the idea for the concept of Swarm, which was proposed by Gavin Wood, which is the storage layer of Ethereum and Whisper is the messaging layer.

Mainstream DSNs follow, and the rest is history.

Decentralized Storage Network Landscape

Decentralized storage environments are the most interesting because of the huge difference in scale between Filecoin and other newer storage networks. While many consider Filecoin and Arweave to be the two giants in storage, most might be surprised to learn that Arweave ranks fourth in terms of usage, behind Storj and Sia (although Sia's usage seems to be declining). While we can easily question the legitimacy of stored FIL data, even if we limit it to 90%, FIL usage is still about 400 times that of Arweave.

What can we deduce from this?

There is a clear dominance in the market today, but the sustainability of this dominance depends on the usefulness of these storage resources. DSNs all use roughly the same architecture, and node operators have a bunch of unused storage assets (hard drives) that they can stake to the network, mine blocks, and earn miner rewards for storing data. While pricing and permanence methods may vary, the most important thing is that the retrieval and computation of stored data is simple and affordable.

Figure 1. Storage Networking by Capacity and Usage

! [Decentralized storage and commercial scenarios in Web3] (https://img-cdn.gateio.im/webp-social/moments-69a80767fe-f4b28e10e6-dd1a6f-69ad2a.webp)

Source: Zee Prime Capital

1. Arweave capacity cannot be measured directly; Instead, node operators are always incentivized to have enough buffers and increase supply to meet demand. How big is the buffer? Given its immeasurability, we have no way of knowing.

2. Swarm's actual network usage is not known, we can only see how much storage has been paid. Whether to use Unknown.

While this is a live project table, there are other DSNs in development. These include ETH storage, Maidsafe, and more.

FVM

Before going any further, it's worth noting that Filecoin recently launched the Filecoin Ethereum Virtual Machine (FEVM). The FVM is a WASM virtual machine that can support many different run times through a hypervisor. For example, the recently launched FEVM is an Ethereum virtual machine running on top of the FVM/FIL network. The reason worth highlighting is that it has facilitated a surge in smart contract (i.e., storing content – stuff) activity on top of FIL. Prior to its release in March, there were essentially 11 active smart contracts on FIL, and this number exploded after the FVM launch. It benefits from composability, i.e. building new businesses on top of FIL with all the work done reliably. That means innovations like quasi-liquid staking type primitives from teams like GLIF, and all sorts of additional financialization of these marketplaces that you can build on such platforms. We believe this will accelerate the growth of storage providers due to increased capital efficiency (SPs need FIL to aggressively mine/archive storage transactions). This differs from a typical LSD because it contains factors that assess the credit risk of individual storage providers.

(Note: Filecoin Ethereum Virtual Machine (FEVM) is a smart contract execution environment used by the Filecoin network, which is built on Ethereum Virtual Machine (EVM) technology.) It has a smart contract language and toolchain similar to Ethereum, allowing developers to build distributed applications (dApps) and implement decentralized file storage and interaction on the Filecoin network. Using smart contracts, users can create storage contracts, lease storage space, access stored content, and more on the Filecoin network. FEVM emerged to take advantage of the compactness, versatility, and reliability of EVM and to provide a cadence for file storage and retrieval suitable for Filecoin networks. In addition, FEVM is not limited to the Filecoin network, it can be freely extended to other blockchain platforms, spawning various interoperability standards and tools. ）

Persistent Storage

I believe Arweave has received the most attention in this regard, and it has a gorgeous slogan that appeals to the deepest desires of Web 3 participants:

Permanent storage.

But what does this really mean? It's a very desirable property, but in reality, execution is everything, and ultimately execution depends on end-user sustainability and cost. Arweave's model is based on a one-time payment, perpetual storage (200 years upfront + storage value crunch assumption) model. This pricing model works well in the deflationary pricing environment of the underlying asset, because there is a protocol called "Constant Goodwill Accrual"** (Note: Constant Goodwill Accrual is a blockchain protocol designed to provide a stable and continuous return mechanism for nodes participating in consensus. It uses a concept called "goodwill," which represents a certain amount of coins being continuously injected into the protocol and distributed to participants. The basic idea of this mechanism is to provide a stable reward mechanism for all nodes participating in consensus, thereby encouraging more nodes to join the network. i.e. old transactions subsidize new transactions)**, but the opposite is true in inflationary environments. History tells us that this should not be a problem, since the cost of computer storage has fallen more or less since its inception, but the cost of hard drives by itself is not everything.

! [Decentralized storage and commercial scenarios in Web3] (https://img-cdn.gateio.im/webp-social/moments-69a80767fe-7080a417f8-dd1a6f-69ad2a.webp)

Source: Zee Prime Capital

Arweave creates permanent storage through incentives from the Pure Random Access Proof (SPoRA) algorithm, which incentivizes miners to store all data and proves that they can randomly generate historical blocks. Doing so gives them a higher probability of being selected to create the next block (and rewarded accordingly).

While this model does a good job of getting node runners to want to store all the data, that doesn't mean it has to happen. Even if you set up ultra-high redundancy and use conservative heuristics to determine the parameters of the model, you will basically never be able to escape this potential risk of loss.

Please visit the link.

! [Decentralized storage and commercial scenarios in Web3] (https://img-cdn.gateio.im/webp-social/moments-69a80767fe-1ab56b74ec-dd1a6f-69ad2a.webp)

Please visit the link.

! [Decentralized storage and commercial scenarios in Web3] (https://img-cdn.gateio.im/webp-social/moments-69a80767fe-e40ba8ad5a-dd1a6f-69ad2a.webp)

Fundamentally, the only way to really perform permanent storage is to deterministically force someone (everyone?) Throw them into the Gulag concentration camp when they screw up. How do you appropriately motivate personal responsibility to achieve this? There is nothing wrong with the heuristic, but we need to determine the best way to implement and price persistent storage.

! [Decentralized storage and commercial scenarios in Web3] (https://img-cdn.gateio.im/webp-social/moments-69a80767fe-8e780ccd91-dd1a6f-69ad2a.webp)

Having said that, what I want to express is what kind of price we want to achieve a reasonable level of security for a certain period of time for permanent storage. In fact, consumer preferences decline along the replication (permanent) range, so they should be able to decide what that level is and get pricing accordingly.

In traditional investment literature and research, knowledge about how the benefits of diversification affect a portfolio's overall risk is notorious. While adding stocks will initially reduce the risk of your portfolio, soon the diversification gains of adding additional stocks become more or less worthless.

! [Decentralized storage and commercial scenarios in Web3] (https://img-cdn.gateio.im/webp-social/moments-69a80767fe-8f417d22db-dd1a6f-69ad2a.webp)

Source: Zee Prime Capital

I believe that storage pricing other than certain default replication criteria on DSNs should follow a similar curve, but the cost and security of storage as replication volumes increase should also be considered.

! [Decentralized storage and commercial scenarios in Web3] (https://img-cdn.gateio.im/webp-social/moments-69a80767fe-a06314aa2d-dd1a6f-69ad2a.webp)

Source: Zee Prime Capital

For the future, I'm most excited about what more DSNs with easily accessible smart contracts can bring to the perpetual storage market. I think the overall consumer will benefit the most as the market opens up this persistent range.

For example, in the image above, we can consider the green area as an experimental area. With minimal changes to the number of replications and durability levels, exponential reductions in storage costs can be achieved.

Other ways to build persistence can come from replication across different storage networks, not just within a single network. Such routes are more ambitious, but naturally lead to more differentiated levels of persistence. The big question here is, can we achieve some kind of "free lunch forever" by spreading it across DSNs, much like we diversify market risk in a portfolio of publicly traded stocks?

The answer may be yes, but it depends on the overlap of node providers and other complicating factors. It can also be built in the form of insurance, possibly by node operators putting themselves under higher levels of cutting conditions in exchange for these guarantees. Maintaining such a system would also be extremely complex, as multiple code bases and coordination between them would be required. Nonetheless, we expect this design landscape to expand significantly and come up with an overarching idea of permanent storage for our industry.

First commercial marketplace for Web 3

Matti recently tweeted about the promise of storage as a use case that brings true business to Web 3. I believe it is possible.

Recently, I had a conversation with a Layer 1 project and I told them that it was their moral responsibility to fill the block space at Layer 1, but more importantly, it was through economic activity. The industry often forgets the second part of its name.

** The entire currency section. **

Any agreement to launch a token that does not want to depreciate requires some kind of economic activity in that currency. For Layer 1, this is their native token that processes payments (performs calculations) and charges gas fees for them. The more economic activity, the more gas fees are used, and the greater the demand for its tokens. This is the cryptoeconomic model. For other protocols, it might be some sort of middleware SaaS service.

The most interesting thing about this model is that when it is paired with some kind of commercial commodity, in the case of classic Layer 1, it is computation. The problem is that since it is related to financial transactions, variable pricing at execution is a poor user experience. Execution costs should be the least important part of financial transactions such as swaps.

Faced with this poor user experience, the difficulty is to fill this block space with economic activity. While scaling solutions are in development that will help stabilize this (I highly recommend this white paper on interstellar consensus, note that this is in PDF format), the flood of Layer 1 markets makes it difficult to find enough activity for a given market.

This problem is easier to solve when this computing power is combined with some additional commercial commodity. For DSNs, this is obviously storage. The economic activity of the stored data and the related elements such as financing and securitization of these storage providers are a direct filler.

But this storage also needs to be a functional solution for traditional enterprises. Especially those who deal with regulations on the way data is stored. The most common forms are audit criteria, geographical restrictions, and making the user experience simple enough to use.

We discussed Banyan earlier in Part 2 of the middleware paper, but their product is on the right track in this regard. Work with node operators in your DSN to ensure that the storage provided is SOC certified while providing a simple UX to facilitate file uploads.

But that alone is not enough.

Stored content also needs to be easily accessible through an efficient retrieval market. One of the things we're really excited about is creating a content delivery network (CDN) on top of a DSN. Basically, a CDN is a tool that caches content close to users and improves latency when retrieving content.

We believe this is the next key component in enabling widespread adoption of DSNs, as this allows videos to load quickly (see Building decentralized Netflix, Youtube, Tiktok, etc.). One of the representatives in this space is our portfolio company, Glitter, which specializes in indexing DSNs. This is important because it is critical infrastructure to improve the efficiency of the retrieval market and facilitate these more exciting use cases.

We are excited about the potential of such products because they prove that PMF is in high demand in Web 2. (Note: PMF here should be short for Product-Market Fit, i.e. "Product Market Fit") Despite the adoption of this product, many people face friction that may benefit from the permissionless nature of leveraging Web 3 solutions.

Impact of composability

Interestingly, we think some of the best alpha versions on DSN are hidden in plain sight. In these two articles by Jnthnvctr.eth, he shares some great ideas for how these markets will evolve and upcoming products (on Filecoin's side):

• The status and direction of Filecoin
• Business model on FVM

The most interesting ** takeaway** is the potential for pairing off-chain computing in addition to storage and on-chain computing. This is because the natural compute needs that provide storage resources in the first place. This natural pairing can add additional commerce to DSN while opening up new use cases.

The introduction of FEVM made many of these upgrades possible and made storage more interesting and competitive. For founders seeking to build new products, there's even a resource with all the products that Protocol Labs asks people to build with potential funding.

In Web 2, we learned that data has a certain gravitational pull and that companies that collect/create large amounts of data can be rewarded and are therefore incentivized to shut down data in a way that protects it.

If our dream of user-controlled data solutions becomes mainstream, we can ask ourselves, how will this accumulation of value change? When users become the primary beneficiaries, trading data for cash flow, there is no doubt that monetization tools that unlock this potential will also benefit, but where and how this data is stored and accessed has also changed dramatically. Of course, this data can reside on a DSN, which benefits from using it through a powerful query marketplace. This is a shift from exploitation to mobility.

What happens next can be very exciting.

When we think about the future of decentralized storage, it's interesting to consider how it will interact with future operating systems like Urbit. For those unfamiliar, Urbit is a personal server built using open-source software that allows you to participate in peer-to-peer networks. A truly decentralized operating system can be self-hosted and interact with the internet, which is a P2P approach.

If the future goes the way Urbit maximizers want, decentralized storage solutions will undoubtedly become a critical part of a single stack. One can easily imagine that all user-related data is encrypted hosted on one of the DSNs and coordinated operations through the Urbit operating system. In addition to this, we can expect further integration with the rest of Web 3 and Urbit, especially projects like Uqbar Network, which brings smart contracts to your Nook environment.

These are all effects of composability, and the slow burn continues to grow exponentially until it produces something truly exciting. What felt like a thumb clumsy turned into a revolution, an alternative path to a hyper-connected world. While Urbit may not be the ultimate solution in this regard (it has its negative reviews), it does show us how these pieces fit together to open up a new river of exploration.

! [Decentralized storage and commercial scenarios in Web3] (https://img-cdn.gateio.im/webp-social/moments-69a80767fe-bddf41a3dc-dd1a6f-69ad2a.webp)

Source: Zee Prime Capital