CN117785999A

CN117785999A - Verifiable account book database storage expansion method and system

Info

Publication number: CN117785999A
Application number: CN202311604142.4A
Authority: CN
Inventors: 曲强; 陈蕾竹
Original assignee: Shenzhen Institute of Advanced Technology of CAS
Current assignee: Shenzhen Institute of Advanced Technology of CAS
Priority date: 2023-11-27
Filing date: 2023-11-27
Publication date: 2024-03-29

Abstract

The application relates to the technical field of computers, in particular to a method and a system for expanding the storage of a verifiable ledger database, which can solve the problems of how to improve a verifiable data structure to improve query efficiency and balance storage and calculation overhead caused by security requirements to a certain extent. The method comprises the following steps: introducing a fragmentation expansion protocol, wherein the fragmentation expansion protocol abstracts a database layer interface to provide transparent service for lower-layer applications; introducing a synchronous control protocol, improving a storage engine interface to realize read-write separated two-layer storage, and defining a cross-piece transaction snapshot format based on a merck tree and a derived verifiable index structure; and designing a storage engine to optimize the locality of index data.

Description

Verifiable account book database storage expansion method and system

Technical Field

The application relates to the technical field of computers, in particular to a method and a system for expanding a verifiable ledger database storage.

Background

Ledger databases can be validated, directly from innovative applications and engineering practices of blockchain systems in data management and storage. The block chain system applies the distributed account book technology, brings new characteristics for the traditional data service, opens up an application scene of the trusted account book data service, and is applied to scenes of data tracing and interactive authentication, such as finance, medical treatment, internet of things and the like. Nevertheless, the need for industry applications to select more robust solutions depending on the particular scenario, blockchain systems are limited to performance and scalability, and are not yet adequate to complete revolution of conventional data storage systems. Traditional databases and distributed computing, cloud service architectures remain the dominant technical forces, also spawning data service schemes for blockchain database hybrid architectures, and independent verifiable ledger database systems.

The blockchain system realizes the decentralized data management from networking to application, and a plurality of nodes in the network participate in management and verification of data independently and synchronously, so that the correctness and consistency of the data can be independently verified even under the conditions of malicious attack, single-point fault and the like. The verifiable account database is abstracted in the blockchain storage layer, and monotonically increasing account data storage requirements further promote account modes to serve as trusted storage evidence of interactive application in a scene where data calculation and storage are separated, provide data verification and audit characteristics, serve as independent storage components, and can be customized to serve more flexibly and efficiently for different application scenes. Both types of systems face expandability bottlenecks, and along with the growth of account book data, storage architecture design is a main difficulty of system expansion, and storage performance is a key optimization point.

The blockchain defines a basic blockchain data structure, comprises a cryptography technology and a distributed consensus mechanism, and can realize data synchronization and consistency verification under a distributed network as the basis of the distributed ledger wall technology. The account book data service provided by the blockchain database hybrid architecture needs to be realized: 1) The verifiable query interface enables the client to verify the validity of the query results of the untrusted node and requires the storage node to traverse the entire blockchain ledger to perform the query; 2) The distributed consensus interface enables consistency of a data synchronization mechanism between logic units of the distributed network. Thus, verifiable ledger databases typically include two major components, an abstract database layer and a ledger storage engine, providing transparent ledger data storage and query interfaces to various upper-level applications, requiring large amounts of data to be processed and persisted.

The existing main blockchain database hybrid architecture system is difficult to establish transverse comparison and compatibility assessment due to non-unification of abstract database layers, which means that the system is difficult to apply to cross-system heterogeneous computation, and the verifiable ledger database can be combined with the abstract database layers and a storage engine layer in an expansion scheme to realize linear expansion and usability assurance, and generally comprises under-chain storage expansion and fragmented storage expansion.

However, how to improve the authenticatable data structure increases query efficiency, balances storage and computational overhead with security requirements, and pays less attention to underlying storage protocol extensibility and storage engine performance optimization.

Disclosure of Invention

In order to solve the problems of how to improve the authenticatable data structure, improve the query efficiency, balance the storage and calculation overhead caused by security requirements and pay less attention to the expansibility of the underlying storage protocol and the optimization of the performance of the storage engine, the application provides a method and a system for expanding the storage of a verifiable ledger-book database.

Embodiments of the present application are implemented as follows:

in a first aspect, the present application provides a verifiable ledger database storage extension method, including:

introducing a fragmentation expansion protocol, wherein the fragmentation expansion protocol is used for providing transparent service for lower-layer application by an abstract database layer interface, a specific fragmentation scheme and a specific form are not required to be considered, and the abstract database layer interface can be adapted to a storage module of a current arbitrary blockchain system;

introducing a synchronous control protocol, improving a storage engine interface to realize read-write separated two-layer storage, defining a cross-piece transaction snapshot format based on a merck tree and a derived verifiable index structure, supporting concurrent lock-free realization through snapshot isolation, and guaranteeing the integrity of distributed transactions;

the storage engine is designed to optimize locality of index data to match the sharding policy and to ensure that data within each shard can be compactly stored in physical storage to reduce latency.

In one possible implementation, the method further comprises optimizing index structure encoding and data page drop strategies by modifying a storage engine of the LSM structure.

In one possible implementation, the synchronization control protocol also introduces the concept of a merck tree-based and verifiable index structure for defining the snapshot format of the cross-slice transaction.

In one possible implementation, the shards divide the storage network nodes into groups, each of which may process transactions in parallel and reduce the storage burden of each node.

In one possible implementation, the slicing includes:

fragmentation based on data copy: data consensus is carried out through network grouping, so that data redundancy processed by a storage engine is reduced overall, and a fragmentation algorithm and an aggregation algorithm are required to be realized;

fragmentation based on data index: by separating the ability of state data to optimize parallel computing, data service performance is generally improved, dynamic load balancing and transactional control need to be achieved, the shards are isolated, and each operation can be provided by a single shard.

In one possible implementation, the slice expansion protocol includes a slice routing protocol and a synchronization control protocol, which can adapt the slice expansion of the data copy-based slice and the data index-based slice at the same time, and customize a verifiable query interface, and a storage engine component supporting transactions and synchronization.

In one possible implementation, for a cross-slice transaction, the method includes the steps of:

a transaction is started: after execution, generating the serial number of the cross-fragment transaction according to the block and the position of the transaction, updating the serial number into a current active transaction list, generating a snapshot corresponding to the transaction and storing the snapshot in a transaction context;

continuing the transaction: according to the last intra-chain transaction id executed on the chain, inheriting the serial number and snapshot of the cross-fragment transaction, and continuing the transaction operation;

commit transaction: and inheriting the number and snapshot of the cross-slice transaction according to the last intra-chain transaction id executed on the chain, and completely submitting the state related to the cross-slice transaction. The specific operation is to remove the transaction from the system active transaction list, then the state of the transaction change can be read by other transactions;

rollback transactions: and inheriting the serial number and snapshot of the cross-chain transaction according to the last intra-chain transaction id executed on the chain, and rolling back all the states related to the cross-fragment transaction.

In a second aspect, the present application provides a verifiable ledger database storage expansion system, including an abstract database layer, a plurality of storage engine layers, and a plurality of data pages;

the abstract database layer is connected with a plurality of storage engine layers, and each data page is respectively connected with a different storage engine page.

In one possible implementation, the abstract database layer includes a plurality of tasks, a slicing router, and a synchronization controller, wherein the slicing router is connected with the synchronization controller, and a plurality of tasks are connected with the slicing router and the synchronization controller.

In one possible implementation, the storage engine layer includes a disk buffer, concurrency control, a codec, a verifiable index, and a block chained log, the verifiable index being connected with the block chained log.

The technical scheme provided by the application can at least achieve the following beneficial effects:

the method for expanding the storage of the verifiable ledger database is suitable for application scenes such as data tracing and auditing by providing a general distributed storage architecture expansion implementation scheme for the verifiable ledger database, is applicable to a block chain system and a distributed ledger technology architecture, meets two modes of fragmentation expansion and under-chain storage expansion, ensures the reliability of cross-fragmentation transaction service under the background of the fragmentation technology, and provides an LSM storage and synchronous optimization scheme for a verifiable data structure. In the partition storage scheme of the verifiable ledger database, the storage and synchronization performance is improved, the design of an LSM storage engine is analyzed from the data angle, and a semantic bridge is built between a semantic layer of the verifiable ledger database and a bottom storage layer based on a verifiable data structure and a data access mode, so that collaborative optimization is realized by combining the data layer and the storage layer, the storage protocol of the partition scheme of the verifiable ledger database is unified, good expandability exists on a data interface, and different distributed consensus algorithms can be adapted; meanwhile, the invention integrates the semantic characteristics of the verifiable data layer and the storage layer, and optimizes the storage and synchronization performance.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief description will be given below of the drawings that are needed in the embodiments or the prior art descriptions, it being obvious that the drawings in the following description are some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort to a person skilled in the art.

FIG. 1 is a flow diagram of a verifiable ledger database storage expansion method according to an exemplary embodiment of the present application;

FIG. 2 is a schematic diagram of a system architecture of a verifiable ledger database storage expansion system, as shown in an exemplary embodiment of the present application;

FIG. 3 is an algorithmic schematic diagram of a verifiable query interface shown in accordance with an exemplary embodiment of the present application;

FIG. 4 is a diagram illustrating a data isolation control based on a version chain-block structure according to an exemplary embodiment of the present application;

FIG. 5 is a flow diagram of a cross-slice transaction process as illustrated in an exemplary embodiment of the present application;

FIG. 6 is a flow diagram of transactional control as illustrated in an exemplary embodiment of the present application;

fig. 7 is a schematic diagram of an architecture of an ledger storage engine as illustrated in an exemplary embodiment of the present application.

Detailed Description

For purposes of making the objects, embodiments and advantages of the present application more apparent, the exemplary embodiments of the present application will be described in detail and fully in connection with the accompanying drawings in which exemplary embodiments of the present application are shown, it being understood that the exemplary embodiments described are only some, but not all, of the examples of the present application, and it is to be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the application.

It should be noted that the brief description of the terms in the present application is only for convenience in understanding the embodiments described below, and is not intended to limit the embodiments of the present application. Unless otherwise indicated, these terms should be construed in their ordinary and customary meaning.

The terms "first," second, "" third and the like in the description and in the claims and in the above-described figures are used for distinguishing between similar or similar objects or entities and not necessarily for limiting a particular order or sequence, unless otherwise indicated. It is to be understood that the terms so used are interchangeable under appropriate circumstances.

The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a product or apparatus that comprises a list of elements is not necessarily limited to all elements explicitly listed, but may include other elements not expressly listed or inherent to such product or apparatus.

In order to facilitate the technical solution of the application, some concepts related to the present application will be described below first.

Verifiable ledger database (Verifiable Ledger Database): the verifiable account book database is a data management and storage system oriented to data objects and historical transaction records thereof, and by maintaining a verifiable data structure and utilizing a cryptography interaction authentication technology, the existence and integrity certification of a data query result is constructed, so that effective data tracing and auditing functions are realized.

Distributed ledger technology (Distributed Ledger Technology): the distributed account book technology is a basic framework for constructing a shared account book by serving a distributed network and comprises a data transmission protocol, an account book synchronization mechanism, a distributed consistency algorithm and the like, wherein the blockchain technology is one of representative forms. The distributed architecture is the solution foundation for the verifiable ledger database storage extension.

Ledger storage engine (Ledger Storage Engine): the account book storage engine is an implementation based on an account book storage protocol, defines an organization structure, a storage mode and an access interface of account book data, maintains the integrity, reliability and safety of the data, provides high-efficiency data access and query capability, and is a general basic unit of a distributed system.

Authentication data structure (Authenticated Data Structure, ADS): the authentication data structure is the core of account book data storage, interactive authentication and distributed synchronization, and is the key design of storage protocol and distributed expansion.

Before explaining the verifiable ledger database storage expansion method provided by the embodiment of the application, an application scenario and an implementation environment of the embodiment of the application are described.

The current main blockchain database hybrid architecture system is difficult to establish transverse contrast and compatibility assessment due to non-uniformity of abstract database layers, which also means that the system is difficult to apply to cross-system heterogeneous computation. The verifiable ledger database combines an abstract database layer and a storage engine layer in an expansion scheme to realize linear expansion and usability assurance, and can be divided into two types in general:

1. under-chain storage extension: by separating the computing tasks by an offline storage protocol, each transaction is avoided from being processed by a consensus mechanism, but only the consensus is used to perform critical tasks (e.g., settlement and dispute resolution).

2. Fragment storage extension: the method is characterized in that the read-write loads of calculation and storage are horizontally separated, the bottleneck of system throughput caused by crowding of calculation tasks of a large number of transactions is avoided, and better concurrency performance is realized through data isolation.

The technical proposal has larger limitation.

1. Network fragmentation mechanism: data slicing is a main means of blockchain and database expansion, and in a distributed environment, an account book storage engine is required to maintain the consistency of account book data, and complex coordination and synchronization mechanisms are required to be adopted, so that the account book storage engine is required to provide concurrency control and a distributed service interface.

2. Concurrent execution mechanism: when multiple users or nodes access the ledger storage engine simultaneously, it is necessary to ensure performance and efficiency of concurrent access. Concurrent read and write operations may cause contention and conflict, requiring the adoption of appropriate concurrency control mechanisms to ensure data consistency and concurrency performance.

3. Storage index optimization: ledger storage engines are often required to support indexing and querying operations on ledger data, and to meet the batch access needs of users to ledger data, with increasing data volume and expanding data storage across segments, the efficiency of indexing and querying may decrease, resulting in increased query latency.

Based on the method, the application provides a verifiable ledger database storage expansion method, which defines a general ledger storage protocol, decouples and abstracts an authentication data service module and an ledger storage engine interface, and supports data fragment expansion and distributed synchronous storage; on the basis of storage expansion, the performance is further optimized aiming at the design of the storage engine, the storage efficiency and the performance are balanced, and certain load balancing flexibility is kept.

Next, the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be specifically described by way of examples with reference to the accompanying drawings. Embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. It will be apparent that the described embodiments are some, but not all, of the embodiments of the present application.

The application scene is as follows: the basic scene of the verifiable account database application is to ensure the integrity, auditability and safety of data in a distributed system, and the important application and the blockchain system are used for storing transactions and data, so that each node can verify the consistency of the data, thereby realizing the decentralised trusted database. In addition, the technical breakthrough of the method is helpful to promote the development of application fields such as centralized financial transactions, medical records, supply chain management, data storage of the Internet of things, digital identity verification, value interaction of private chains of the vertical industry and the like.

Fig. 1 is a flow chart illustrating a method for expanding a verifiable ledger database storage according to an exemplary embodiment of the present application.

In an exemplary embodiment, as shown in fig. 1, there is provided a verifiable ledger database storage expansion method, which in this embodiment may include:

step 100: the method comprises the steps of introducing a fragmentation expansion protocol, wherein the fragmentation expansion protocol is used for providing transparent service for lower-layer application by an abstract database layer interface, a specific fragmentation scheme and a specific form are not required to be considered, and the method can be adapted to a storage module of a current arbitrary blockchain system.

Step 200: and introducing a synchronous control protocol, improving a storage engine interface to realize read-write separated two-layer storage, defining a cross-piece transaction snapshot format based on a merck tree and a derived verifiable index structure, supporting concurrent lock-free realization through snapshot isolation, and guaranteeing the integrity of distributed transactions.

Step 300: the storage engine is designed to optimize locality of index data to match the sharding policy and to ensure that data within each shard can be compactly stored in physical storage to reduce latency.

In one possible implementation, a sharding expansion protocol is introduced, and the core objective of the sharding expansion protocol is to improve the scalability of the verifiable ledger database, and a data storage layer expansion protocol based on a sharding mechanism is provided to provide a transparent service interface for upper-layer applications. Such partitioning can be adapted to different blockchain systems, such as ethernet, without modifying the memory modules of the underlying blockchain system, and the slicing mechanism allows transactions and data to be processed in parallel on different slices, thereby improving throughput and performance of the overall system, the design of which is not limited to a particular slicing scheme, but can accommodate a variety of different slicing strategies and forms.

In one possible implementation, a synchronization control protocol is introduced, and in order to support the collaborative work between multiple slices, the scheme introduces a distributed synchronization protocol that improves the interface of the storage engine to implement a read-write separated two-layer storage structure based on the concept of a Merkle Tree (Merkle Tree) and a verifiable index structure for defining the snapshot format of the cross-slice transaction. This allows for more efficient isolation of cross-slice transactions and supports concurrent operations without using conventional lock mechanisms. Through the improvements, the scheme can ensure the integrity of distributed transactions and ensure the consistency of data among different fragments.

In one possible implementation, a storage engine design is introduced, the storage engine being responsible for managing the storage and retrieval of data. The scheme optimizes the locality of index data to ensure matching with a slicing strategy, ensures that data in each slicing can be stored in a physical storage medium in a compact mode to reduce IO delay, and some embodiments of the application improve a storage engine of an LSM (Low-Structured Merge) structure, further improve the performance and efficiency of the storage engine by improving the coding mode of the index structure and the data page landing strategy, and are beneficial to reducing the time overhead of data reading and writing.

In some embodiments of the present application, by deconstructing verifiable ledger databases into an abstract database layer and a storage engine layer, a sharding protocol and a synchronization protocol are designed as extensible basic components, and a reusable ledger storage engine is customized for a critical data model, completing corresponding performance optimization.

Fig. 3 is an algorithm schematic diagram of a verifiable query interface according to an exemplary embodiment of the present application, fig. 4 is a data isolation control schematic diagram based on a version chain-block structure according to an exemplary embodiment of the present application, fig. 5 is a flow schematic diagram of a cross-slice transaction according to an exemplary embodiment of the present application, fig. 6 is a flow schematic diagram of a transaction control according to an exemplary embodiment of the present application, and fig. 7 is an architecture schematic diagram of an account book storage engine according to an exemplary embodiment of the present application.

In one possible implementation, some embodiments of the present application include abstract storage protocol design to ledger storage engine design and optimization, whose core logic unit includes the following three parts:

1. storage expansion protocol

The segmentation scheme is an important technical means capable of verifying the expansion of the account book database: shards, which divide storage network nodes into groups, called shards, each shard can process transactions in parallel and relieve storage burden of each node, introduce a new requirement, namely, cross-sharded database service, namely, data aggregation for query and workload balancing for management, which is a transparent storage expansion protocol for account data service, and can provide a unified application interface for the following common shard expansion schemes:

1) Fragmentation based on data copy: the method is characterized in that data consensus is carried out through network grouping, so that data redundancy processed by a storage engine is reduced overall, and a fragmentation algorithm and an aggregation algorithm are required to be realized, wherein the schemes comprise account-based fragmentation, side chain, offline transaction, under-chain storage and the like in the Ethernet 2.0.

2) Fragmentation based on data index: by the ability to optimize parallel computing by separating state data, data service performance is generally improved, dynamic load balancing and transactional control are required, as is common in many blockchain database hybrid architectures, shards are isolated, and each operation may be provided by a single shard.

The present solution decouples the storage expansion protocol into the partition routing protocol and the synchronization control protocol, which can adapt the partition expansion in both modes at the same time, and customize the verifiable query interface as shown in fig. 3, and the storage engine components supporting transactions and synchronization.

2. Synchronous control protocol

In account/balance based state sharding, there is a natural problem in distributing a large number of transactions, namely how to achieve load balancing among all shards, workload imbalance in hot shards may lead to long transaction validation delays, threatens to the final atomicity of cross-sharded transactions, and when most transactions are cross-sharded transactions, this problem becomes more challenging, ensures that the final atomicity of cross-sharded transactions becomes a challenge, impeding widespread use of state sharding mechanisms in practice.

The common MVCC in a blockchain system in combination with the block structure provides data versioning to support conventional concurrent operations, but does not provide data isolation guarantees, resulting in transaction integrity issues in a fragmented environment.

In some embodiments of the present application, as shown in FIG. 4, the blockchain underlying store does not take full advantage of the characteristics of the blockchain structure, providing characteristic support for online and transactional mode evolution by adjusting the MVCC concurrency control protocol in the ledger memory engine.

The block chain structure commonly uses a merck tree related index structure, such as MPT, in which each block contains a merck tree, where historical state data of the block chain is stored, and each leaf node has a pointer pointing to a front state of the state, so as to effectively support snapshot generation and state rollback in multi-version concurrency control, where a leaf node in the merck tree is L, and an intermediate node is M. Each leaf node contains a specific value of the state it manages and a pointer to the state's leading state, which can be expressed asWherein k is _i For the state key value corresponding to the leaf node, v _j For the version number of the state,t _n to update this state to the transaction number of the current value, its value is +.>The value of the intermediate node of the merck tree is k _M ＝hash(L _i ，f(L _i ) =m), the function obtains L _i The hash is the hash of the sequence after the value concatenation of all nodes in the set is calculated.

For a cross-slice transaction, as shown in fig. 5, the specific operation of the transaction method of each stage is as follows:

1. begin transaction begin tx: after execution, generating the serial number of the cross-fragment transaction according to the block and the position of the transaction, updating the serial number into a current active transaction list, generating a snapshot corresponding to the transaction and storing the snapshot in a transaction context;

2. continuing the transaction ContinueTx: according to the last intra-chain transaction id executed on the chain, inheriting the serial number and snapshot of the cross-fragment transaction, and continuing the transaction operation;

3. commit transaction CommittTx: and inheriting the number and snapshot of the cross-slice transaction according to the last intra-chain transaction id executed on the chain, and completely submitting the state related to the cross-slice transaction. The specific operation is to remove the transaction from the system active transaction list, then the state of the transaction change can be read by other transactions;

4. rollback transaction rollback tx: and inheriting the serial number and snapshot of the cross-chain transaction according to the last intra-chain transaction id executed on the chain, and rolling back all the states related to the cross-fragment transaction. The method specifically comprises the steps of reading a write set of the object from the snapshot, acquiring a version before all states in the write set are changed by the transaction, updating the states by using the version, and removing the states from a system active transaction list.

In the basic read method, as shown in fig. 6, the state and its corresponding version number are first read.

And judging whether the version number of the state is larger than the maximum version number of the transaction in the active list, if so, indicating that the state is updated by other transactions after the transaction snapshot is generated, and the state is invisible to the transaction, so that the state is continuously returned to the last version until the version of the state is smaller than the maximum number of the active transaction list.

And secondly, judging whether the state version is smaller than the minimum value of the versions in the active transaction list, if so, indicating that the transaction for updating the state is completed, wherein the state of the version is visible to all transactions including the transaction, and returning the state value of the version. If not, the state version number is between the minimum and maximum of the active list transaction version numbers, and whether the transaction updating the state is the current transaction is continuously judged.

If yes, the update is visible to the current transaction, and the state value of the version is returned; if not, a determination is continued as to whether the transaction updating the state is in the active list.

If the state update is not submitted in the active list, the state is invisible to other transactions including the current transaction, and the state needs to be continuously returned to the last version until the update of the version has completed the submission; if not in the active list, the transaction that illustrates updating the state has committed to completion, the version is visible to all transactions, including the present transaction, and the state value for the version is returned.

3. Storage engine optimization

Some embodiments of the present application observe that the ledger-store engine, as shown in fig. 7, designs an asynchronous non-blocking migration method, with possible service interruption and performance bottlenecks during the data synchronization process. The main idea is to design a data aggregation technology based on a multi-level storage structure of an LSM storage engine and adopting a dual mode of cross-slice synchronization and concurrency control to minimize the interrupt influence of the migration process on the migrating table.

The scheme deeply researches the process of node serialization storage and the influence of the process on the system query performance in a secure merck patricia tree (Merkle Patricia Trie, MPT) structure realized by the Ethernet. The serialized storage implemented by the hash function results in the key values being mapped randomly to different key ranges, causing the data layout to be unplanned, losing the original location semantics. This introduces multiple reads of ordered string (Sorted String Table, SST) files in the state data request, severely degrading the query performance of the system. In order to solve the problem, we propose a strategy based on semantic mapping, which aggregates key value pairs in the same query path into adjacent SST files in a prefix manner, so as to reduce extra disk IOs in the query process. In connection with the data construction supported by the transaction in the foregoing, the stored procedure encoding scheme is as follows:

k _M ＝prefix+hash(L _i ，f(L _i )＝M)

for each access of the MPT, one complete branch from the root node to the leaf node is involved. Due to the index design of the tree structure, adjacent access nodes are necessarily parent-child nodes, and certain time locality is generated.

Therefore, some embodiments of the present application aggregate nodes that are parent-child relationships to each other in adjacent locations of a storage device to ensure sequential reading of data and improve storage performance, in this respect, the LSM-Tree based storage component exhibits great potential, and the ordered SSTtable and multi-level data cache structure thereof provides adjacent storage locations for keys of similar word order, and further improves storage performance by exerting a spatial locality effect through a block cache.

It can be seen that transparent extensions to verifiable ledger databases do have certain requirements to the storage engine to support efficient, extensible, and reliable database sharding mechanisms, some embodiments of the present application focus on the following optimizable points:

1. concurrent transaction control: the storage engine needs to handle high concurrency random reads and writes, even coordination and management across sharded transactions, in conjunction with an index structure.

2. High-efficiency read-write capability: the storage engine should optimize the locality of the data to match the slicing policy, ensuring that the data within each slice can be compactly stored in physical storage to reduce IO delay.

3. Partial synchronization mechanism: the storage engine should support efficient migration of data, and as the data access pattern changes, the sharding policy may need to be readjusted without causing significant data migration or downtime

It should be understood that, although the steps in the flowcharts relating to the above embodiments are shown in order as indicated, these steps are not necessarily performed in order as indicated. Unless explicitly stated otherwise herein, the steps are not strictly limited to the order in which they may be performed, and at least some of the steps in the flowcharts described in the above embodiments may include steps or stages which are not necessarily performed at the same time but may be performed at different times, and the order in which the steps or stages are performed may not necessarily be sequential, but may be performed in turn or alternately with at least some of the other steps or stages.

Corresponding to the embodiment of the verifiable ledger database storage expansion method, the application also provides an embodiment of the verifiable ledger database storage expansion system by adopting the same technical conception.

FIG. 2 is a system architecture diagram of a verifiable ledger database storage expansion system, as shown in an exemplary embodiment of the present application.

In one exemplary embodiment, as shown in FIG. 2, the verifiable ledger database storage expansion system includes an abstract database layer, a plurality of storage engine layers, and a plurality of data pages;

For specific limitations on the verifiable ledger database storage expansion system, reference is made to the above limitations on the verifiable ledger database storage expansion method, which are not repeated here. The modules in the verifiable ledger database storage expansion system described above may be implemented in whole or in part in software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples represent only a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application is to be determined by the claims appended hereto.

Claims

1. A method for expanding the storage of a verifiable ledger database, comprising:

2. The verifiable ledger database storage expansion method of claim 1, further comprising optimizing index structure coding and data page drop policies by modifying a storage engine of the LSM structure.

3. The verifiable ledger-book database storage extension method of claim 1, wherein the synchronization control protocol further introduces concepts based on a merck tree and a verifiable index structure for defining snapshot formats of cross-slice transactions.

4. The verifiable ledger database storage expansion method of claim 1, wherein the shards divide storage network nodes into groups, each of which can process transactions in parallel and reduce the storage burden of each node.

5. The verifiable ledger database storage expansion method of claim 4, wherein the sharding comprises:

6. The verifiable ledger database storage expansion method of claim 5, wherein the shard expansion protocol comprises a shard routing protocol and a synchronization control protocol that can adapt shard expansion of the data copy-based shard and the data index-based shard simultaneously, and customize a verifiable query interface, and a storage engine component that supports transactions and synchronization.

7. The verifiable ledger database storage expansion method of claim 3, comprising the steps of, for a cross-sharded transaction:

8. The system is characterized by comprising an abstract database layer, a plurality of storage engine layers and a plurality of data pages;

9. The verifiable ledger database storage expansion system of claim 8, wherein the abstract database layer comprises a plurality of tasks, a shard router, and a synchronization controller, the shard router being coupled to the synchronization controller, the plurality of tasks being coupled to the shard router and the synchronization controller.

10. The verifiable ledger database storage expansion system of claim 8, wherein the storage engine layer comprises a disk buffer, concurrency control, a codec, a verifiable index, and a block chaining log, the verifiable index being connected with the block chaining log.