CN114168685B - Novel database architecture based on blockchain system and operation method - Google Patents
Novel database architecture based on blockchain system and operation method Download PDFInfo
- Publication number
- CN114168685B CN114168685B CN202111531166.2A CN202111531166A CN114168685B CN 114168685 B CN114168685 B CN 114168685B CN 202111531166 A CN202111531166 A CN 202111531166A CN 114168685 B CN114168685 B CN 114168685B
- Authority
- CN
- China
- Prior art keywords
- database
- metadata
- blockchain
- blockchain system
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a novel database architecture based on a blockchain system, which comprises the following components: a built-in blockchain system, a legacy database system, supports an external blockchain system. An external blockchain system typically has multiple nodes, each with its own independent database system, each containing a built-in blockchain system. Such an external blockchain system typically includes at least 16 built-in blockchain sub-nodes, each of which can independently operate to maintain the functionality and performance of the database system; all metadata is stored by the built-in blockchain system. In this way, an external blockchain system stores data in a traditional database system, but metadata exists in an internal blockchain system, so that high security that the metadata is not damaged and is not tampered and deleted is ensured.
Description
Technical Field
The invention belongs to the technical field of block chain database technology, block chain systems and related database systems, and particularly relates to a novel database architecture based on a block chain system and an operation method.
Background
Blockchains are considered to be a special database, but their data structures are quite far apart from traditional databases, and their performance has been problematic due to the redundancy mechanism, layer-by-layer encryption mechanism of blockchains. Conventional database systems may store large amounts of data, may be fragmented, may be scalable, and may be fault tolerant, but the data stored on the database may be altered. And for the blockchain system, the expandability is difficult, a special structure is required for expandability, the blockchain system has an encryption mechanism, and the data is difficult to change. Both micro-soft and IBM have proposed their own blockchain databases, but each has its drawbacks. The microsoft blockchain db blockchain database adopts partial consensus (consensus is performed by only nodes participating in transactions), thereby improving efficiency and reducing security. The original IBM blockchain database was a virtually centralized design, placing the ordering service on a centralized server to order the transaction sequence, the centralized and blockchain number characteristics being violated.
The block chain system is greatly improved, great progress is shown in the system architecture and the application flow, and the situation that the block chain system and the database system are gradually fused is presented. In particular, new blockchain systems and architectures are continuously presented, the architecture gap between the architecture and the super ledger is very large, and the transaction flow based on the new blockchain is also different from the transaction mode of the traditional blockchain system. However, currently, the blockchain database system or cloud platform, metadata (metadata) is the most important component subsystem. If the metadata system has problems, the system needs to stop the operation, and the metadata can be processed before continuing.
Thus, there is a need for a high performance, highly fault tolerant database system, particularly one that is capable of effectively protecting metadata and that is scalable.
Disclosure of Invention
The invention aims to solve one or more technical problems in the prior art, and provides a high-performance and high-fault-tolerance database system, in particular to a novel database architecture based on a blockchain system, which can effectively protect metadata and can be expanded, wherein the novel database system stores metadata on the blockchain system and stores data in a traditional database structure, so that a high-performance unstructured and fused blockchain distributed database is formed. The method is characterized in that the characteristics of MySQL relational lines are reserved, the unstructured storage structure of MongoDB is used for reference, the method has excellent performance like Redis, metadata is stored by a blockchain, and high security that the data is not damaged and is not tampered and deleted is ensured.
The invention aims to provide a novel database architecture based on a blockchain system, which can support the operation of a blockchain system and comprises the following components:
a built-in blockchain system, a database system, the blockchain system supported by the database system being an external blockchain system; wherein a plurality of nodes are arranged in the external block chain system, each node comprises the database system, and the database system comprises the built-in block chain system;
the external blockchain system is provided with a plurality of nodes, each node is provided with an independent sub-database, and each independent database is provided with an independent built-in blockchain subsystem, so that the external blockchain system comprises at least 16 built-in blockchain nodes, and each built-in blockchain subsystem can independently operate to maintain the functions and the performances of the database system;
all metadata in the novel database architecture based on the blockchain system are managed by the built-in blockchain system and the external blockchain system, the metadata are stored in the built-in blockchain system and the external blockchain system, the data are stored in the database system, a Connection SDK is externally provided, the Connection SDK is provided with an SDK Api, a user uses the SDK Api to perform CRUD operation on a table of the database system based on an application program, and the CRUD operation represents creation, reading, updating and deleting;
the metadata includes:
DB: data information describing the entire database system;
name: describing the name of a database or the name of a certain table, wherein the tables with the same name cannot exist in the same database;
BaseDir: the path of the database on disk;
hash and preHash: marking a current metadata version, if the current metadata version is updated, assigning a current hash to a pre-hash, updating the current hash to a latest version hash, and when the pre-hash is not equal to the hash, indicating that the metadata version is modified, and the metadata version needs to be serialized to a block chain, wherein when serialization is completed, the hash is equal to the pre-hash;
writer: the mark has the written user authority;
reader: marking the user authority which is read;
TableList: describing all table information under the current database;
BlockList: describing all block information under a current table;
TxList: recording all business data information on the current block, wherein the data information is serialized on a magnetic disk;
ActiviTime: recording the last modification time of the current block, if the database is larger than a pool, indicating that not all blocks can be cached in the pool, and serializing the blocks with the farthest activiTime distance to a disk for reservation;
table: describing basic information of a database system table;
block (Block): information describing the sub-table chunks;
index: describing all index data on the table;
tx: the most original transaction data of the user is saved, and the transaction data is a JSON character string type.
Preferably, among the plurality of nodes of the external blockchain system, some nodes use a traditional database system, no built-in blockchain system is used for storing metadata, and some or all nodes use the novel database architecture based on the blockchain system for storing metadata. Since each node operates independently, it is a preferred embodiment that can be selected as long as the functionality and performance are acceptable.
Preferably, each of the database systems uses a different kind of database, including a relational database.
Preferably, the database system forms a cluster, a plurality of Master processes, a backup process and a Connection control are arranged in the cluster, the Master processes are database processes for storing data, the backup process is a process responsible for backing up the masters, when the Master processes fail, the backup process replaces the Master processes to complete work, process information of the Master processes and process information of the backup process are concentrated together and stored on the blockchain system to be managed, and after the Connection space obtains global process information through the blockchain system and establishes Connection, commands of the CRUD operation are distributed to each cluster to be executed; the Master process is provided with a plurality of Master sub-processes which are independently operated, and the number of the Master sub-processes can be increased or reduced according to the requirements of different database systems; the backup process is provided with a plurality of backup sub-processes which are independently operated, and the number of the backup sub-processes can be increased or decreased according to the requirements of different database systems; the processes have different relations, wherein the included processes can be a Master, and the included processes are a Worker which listens to the Master; the number of processes increases or decreases depending on the application environment, there may be multiple processes on a large machine to process and run, and a corresponding decrease on a small system.
Preferably, in one of the external blockchain systems, the database system in each node may use a different database, and even if the same database is used, the data may be stored in a different or mixed mode, and the data includes structured data and unstructured data.
Preferably, the metadata and data information stored includes 4 layers:
layer 1 layer data information: storing basic principles of the database system, thereby maintaining basic information of the database system, including database names, paths of the database on the disk, list information, as part of metadata;
layer 2 layer data information: the list of layer 1 data information points to data information of a Table structure of layer 2 data information as a part of metadata;
layer 3 data information: the table structure of layer 2 data information points to Block and Index data structure information as part of metadata;
layer 4 layer data information: actual transaction information, these are data.
Preferably, the new database architecture based on blockchain system uses simple and easily understood object-oriented query statement OQL (Object Query Language) or uses traditional database query language (e.g., SQL) for the CRUD operation of data; the novel database architecture based on the blockchain system has a database expansibility mechanism and a metadata fault tolerance mechanism.
The invention also aims to provide an operation method of the novel database architecture based on the blockchain system, which is realized by adopting a Response mode of a Request-Response, and comprises the following steps:
step 1, a client initiates a request through connection, and the request is handed to a right manager to verify the right;
step 2, submitting the request to other auditing tasks for verification after passing the verification;
step 3, after verification by the other auditing tasks, the information is submitted to a Table, and the Table analyzes whether metadata of an Index needs to be modified;
step 4, calculating the blockId of the metadata to be operated, and sending an operation result to a corresponding BlockManager according to the blockId after performing the CRUD operation on the metadata;
step 5, the information is sent to Pool, a Block entity is obtained according to the Block Id, and the obtaining is achieved through caching or a magnetic disk;
step 6, executing corresponding CRUD operation according to the obtained Block entity and the request of the request;
and 7, writing the result after the operation into a response object and returning to connection.
Preferably, the step 4 adopts different BlockManager algorithms, including: table processes multiple tables at a time; the Blockmanager creates a plurality of blocks each time; unused blocks can be stored first and used after waiting for later use.
Preferably, the method further comprises: metadata update, comprising: all metadata are stored in a memory to ensure performance except on a blockchain system, when each process is started, whether the pre Hash of the metadata is consistent with the Hash of the metadata or not is firstly inquired, and if so, the process is executed; if not, reading the updated metadata back from the blockchain system and then executing 'PreHash=Hash'; each time the database needs to update metadata, the metadata is changed, the Hash is updated, and the updated metadata is stored in the built-in blockchain system.
The invention has the beneficial effects that:
a high-performance and high-fault-tolerance database system, particularly a database system which can effectively protect metadata and can be expanded, namely a novel database architecture based on a blockchain system, wherein the novel database system stores metadata on the blockchain system and stores data in a traditional database structure, so that a high-performance unstructured and fused blockchain distributed database is formed. The method is characterized in that the characteristics of MySQL relational lines are reserved, the unstructured storage structure of MongoDB is used for reference, the method has excellent performance like Redis, metadata is stored by a blockchain, and high security that the data is not damaged and is not tampered and deleted is ensured.
The above, as well as additional objectives, advantages, and features of the present invention will become apparent to those skilled in the art from the following detailed description of a specific embodiment of the present invention when read in conjunction with the accompanying drawings.
Drawings
Some specific embodiments of the invention will be described in detail hereinafter by way of example and not by way of limitation with reference to the accompanying drawings. The same reference numbers will be used throughout the drawings to refer to the same or like parts or portions. It will be appreciated by those skilled in the art that the drawings are not necessarily drawn to scale. The objects and features of the present invention will become more apparent in view of the following description taken in conjunction with the accompanying drawings in which:
FIG. 1 is a conceptual diagram of library-in-library principles according to a preferred embodiment of the present invention;
FIG. 2 is a schematic diagram of a novel database architecture based on a blockchain system in accordance with the preferred embodiment of the present invention.
Fig. 3 is a data structure diagram of metadata according to a preferred embodiment of the present invention.
Fig. 4 is a database process flow according to a preferred embodiment of the present invention.
Fig. 5 is a process flow diagram of a preferred embodiment according to the preferred embodiment of the present invention.
Detailed Description
In order that the invention may be more readily understood, reference will now be made to the following description of the invention taken in conjunction with the accompanying drawings and examples. In the following description, numerous specific details and specific examples are set forth in order to provide a thorough understanding of the present invention and to provide a thorough and complete understanding of the present invention to those skilled in the art. While this invention is susceptible of embodiment in many other forms than those described herein, those skilled in the art will be able to make and use the invention without departing from the spirit of the invention, and therefore the invention is not limited by the specific examples and figures disclosed below.
In the following description, numerous specific details and specific examples are set forth in order to provide a thorough understanding of the present invention and to provide a thorough and complete understanding of the present invention to those skilled in the art. While this invention is susceptible of embodiment in many other forms than those described herein, those skilled in the art will be able to make and use the invention without departing from the spirit of the invention, and therefore the invention is not limited by the specific examples and figures disclosed below.
As shown in fig. 1, the principle of the architecture of the present embodiment: because the blockchain data is still present within the database system, the blockchain system is used within the database system to form a "in-library" architecture: an external blockchain system includes one database system per node, and the database system supports database operations using another (built-in) blockchain system. Because the blockchain system is in the database system, the blockchain system can bear a plurality of operation works corresponding to redundancy and fault tolerance mechanisms, thereby effectively preventing the possibility of cluster damage caused by power failure and avoiding cluster breakdown caused by misoperation.
The present embodiment provides a new database architecture based on a blockchain system, including:
an internal blockchain system and a database system for supporting the operation of an external blockchain system, wherein the database system is contained in the external blockchain system, the external blockchain system comprises a plurality of nodes, each node comprises one database system, and each database system comprises one internal blockchain system. Because one of the external blockchain systems has multiple nodes, each node has an independent database, and each independent database is provided with its own independent built-in blockchain system, one external blockchain system contains at least 16 built-in blockchain nodes, and each built-in blockchain system can independently operate to maintain the functions and performances of the database system.
Preferably, some blockchains may have more than 4 nodes, or fewer than 4 nodes, including external blockchain systems or internal blockchain systems. The more nodes, the less good the performance, but the better the reliability; the fewer nodes, the better the performance, and if there are fewer than 4 nodes, the blockchain system may have difficulty maintaining the bayer general agreement.
The external block chain system is provided with a plurality of nodes, and the more the nodes are, the more the built-in block chain system is, so that the system performance is better.
In a preferred embodiment, some nodes of the plurality of nodes of the external blockchain system may use a traditional database system, no built-in blockchain system is used to store metadata, and some or all nodes use the new database architecture based on the blockchain system to store metadata. Since each node operates independently, it is a preferred embodiment that can be selected as long as the functionality and performance are acceptable.
As a preferred embodiment, each of the database systems uses a different kind of database, including a relational database or other database system.
Because the external block chain system can be an internal block chain system of another database system, a hierarchical complex block chain network can be established
As shown in FIG. 2, all metadata in the new blockchain system-based database architecture is submitted to the built-in blockchain system, the metadata is stored in the built-in blockchain system and the external blockchain system, the data is stored in the database system, a Connection SDK is externally provided, the Connection SDK is provided with an SDK Api, a user uses the SDK Api to perform CRUD operation on the database system based on an application program, wherein the CRUD operation represents creation (creation), reading (Update), updating (Update), and deletion (Delete).
As a preferred embodiment, the database system forms a cluster, a plurality of Master processes, a backup process and a Connection control are set in the cluster, the Master processes are database processes for storing data, the backup process is a process responsible for backing up the Master, when the Master processes fail, the backup process replaces the Master processes to complete work, process information of the Master processes and process information of the backup process are concentrated together and stored on the blockchain system to be managed, and after the Connection space obtains global process information through the blockchain system and establishes Connection, commands of the CRUD operation are distributed to each cluster to be executed, and the more the processes, the stronger the concurrency capability and the disaster tolerance capability of the clusters are.
As a preferred embodiment, the Master process has a plurality of Master sub-processes (3 in this embodiment) running independently, and the number of the Master sub-processes can be increased or decreased according to the requirements of different database systems.
As a preferred embodiment, the backup process has a plurality of backup sub-processes (3 in this embodiment) running independently, and the number of backup sub-processes can be increased or decreased according to the requirements of different database systems.
In a preferred embodiment, the database system in each node may use different databases, and even if the same database is used, different or mixed modes may be used to store data, including structured data as well as unstructured data, in an external blockchain system.
As a preferred embodiment, the processes may have different relationships, for example, some processes may be masters and some processes may be workers, who listen to masters.
As a preferred embodiment, the number of these processes may be increased or decreased. On large machines there may be multiple processes to handle, while on small systems there may be fewer processes.
In this embodiment, as shown in fig. 2, the blockchain system includes 4 nodes (node 1, node 2, node 3, node 4). In a practical system, the number of nodes can be increased; when the system fails, the normal operation of 3 nodes is guaranteed to be continued.
The metadata and data structure diagram is shown in fig. 3.
As a preferred embodiment, the database system uses a mixed mode to store data, including structured data as well as unstructured data. The stored data information includes 4 layers:
layer 1 layer data information: storing basic principles of the database system, thereby maintaining basic information of the database system, including information such as database names, paths of the database on a disk, lists (tablelets) and the like, as a part of metadata;
layer 2 layer data information: the list of layer 1 data information points to (Table) data information of a Table structure of layer 2 data information as part of metadata;
layer 3 data information: the table structure of layer 2 data information points to Block (Index) and Index (Index) data structure information as a part of metadata;
layer 4 layer data information: actual transaction information as data.
Thus, the data structure can start from the list, go to the table and then go to the block, and the block chain system can be queried at high speed; while using the index data to facilitate high-speed queries.
As a preferred embodiment, the layer 4 data information can be partially adjusted according to different database requirements, for example, the change on the storage structure can enable some update of the data structure, but the basic principle is still consistent: list to list, list to block and index, and finally to transaction information.
See fig. 3, wherein:
DB: data information describing the entire database system;
name: describing the name of a database or the name of a certain table, wherein the tables with the same name cannot exist in the same database;
BaseDir: the path of the database on disk;
hash and preHash: marking a current metadata version, if the current metadata version is updated, assigning a current hash to a pre-hash, updating the current hash to a latest version hash, and when the pre-hash is not equal to the hash, indicating that the metadata version is modified, and the metadata version needs to be serialized to a block chain, wherein when serialization is completed, the hash is equal to the pre-hash;
writer: the mark has the written user authority;
reader: marking the user authority which is read;
TableList: describing all table information under the current database;
BlockList: describing all block information under a current table;
TxList: recording all business data information on the current block, wherein the data information is serialized on a magnetic disk;
ActiviTime: recording the last modification time of the current block, if the database is larger than a pool, indicating that not all blocks can be cached in the pool, and serializing the blocks with the farthest activiTime distance to a disk for reservation;
table: describing basic information of a database system table;
block (Block): information describing the sub-table chunks;
index: describing all index data on the table;
tx: the most original transaction data of the user is saved, and the transaction data is a JSON character string type.
As a preferred embodiment, the new database architecture based on the blockchain system uses simple and straightforward object-oriented query statements OQL (Object Query Language) or uses conventional database query language (e.g., SQL) for the CRUD operation of data. If SQL is used, the database workload is large, and the used object-oriented query language is easier; the novel database architecture based on the blockchain system has a database expansibility mechanism, and a traditional database can use different expansibility mechanisms, such as mechanisms of slicing and the like; in this embodiment, the database is divided into two different data types, namely data and metadata, wherein a traditional database can be used as part of the data, and the metadata occupies small storage, so that the database has expansibility; the new database architecture based on the blockchain system has a metadata fault-tolerant mechanism, the traditional database has a self fault-tolerant mechanism, and the built-in blockchain system also has a self fault-tolerant mechanism.
As shown in fig. 4, the preferred embodiment further provides a method for operating a new database architecture based on a blockchain system, which is implemented by adopting a Response mode of Request-Response, and includes:
step 1, a client initiates a request through connection, and the request is handed to a right manager to verify the right;
step 2, submitting the request to other auditing tasks for verification after passing the verification;
step 3, after verification by the other auditing tasks, the information is submitted to a Table, and the Table analyzes whether metadata of an Index needs to be modified;
step 4, calculating the blockId of the metadata to be operated, and sending an operation result to a corresponding BlockManager according to the blockId after performing the CRUD operation on the metadata;
step 5, the information is sent to Pool, a Block entity is obtained according to the Block Id, and the obtaining is achieved through caching or a magnetic disk;
step 6, executing corresponding CRUD operation according to the obtained Block entity and the request of the request;
and 7, writing the result after the operation into a response object and returning to connection.
As a preferred embodiment, the above method may be modified or optimized, including: for example, an optimization algorithm is implemented in step 3; and step 4 adopts a different BlockManager algorithm. For example, table may process multiple tables at once; the Blockmanager can create a plurality of blocks each time, and unused blocks can be stored first and used later when needed. However, none of these optimizations change the overall flow principle, namely table-to-Block management.
As a preferred embodiment, the method further comprises: metadata update, comprising: all metadata are stored in a memory to ensure performance except on a blockchain system, when each process is started, whether the pre Hash of the metadata is consistent with the Hash of the metadata or not is firstly inquired, and if so, the process is executed; if not, reading the updated metadata back from the blockchain system and then executing 'PreHash=Hash'; when the database needs to update metadata, the metadata is updated, the Hash is updated, and the updated metadata is stored in the built-in block chain system; since the blockchain system uses Merkle trees, it is very easy to find the metadata location that needs to be updated.
As shown in the preferred embodiment of fig. 5:
the preferred embodiment proposes a distributed database based on blockchain, wherein all metadata are managed by the blockchain system, a Connection SDK is provided externally, and App uses SDK Api to perform CRUD operation of the table on the database.
Testing and performance:
the corresponding new database system software has been developed, and in the CPU i7 8 core 16 thread, memory 16G ddr4 2200Mhz, disk: 400Gssd 400M/s, network card: the system cache pool size is about 1000M/s, and is compared with the well-known databases Redis and MongoDB in the industry under the environment of 2GB, and the performance of the system cache pool is about the same as that of the Redis, but is much better than that of the MongoDB. The larger the data volume, the better the performance of the patent is than Redis. And MongoDB is very poor in performance. For example, in a 1 million transaction environment, the system of the invention can run 8.6 ten thousand transactions per second, redis can run 8.85 ten thousand transactions, and MongoDB can only run 1 ten thousand transactions. Under the environment of 1 million transactions, the invention can achieve 7.5 ten thousand transactions, the Redis can achieve 6.98 ten thousand transactions, and the MongoDB can only achieve 1.2 ten thousand transactions.
This embodiment uses the Tiande blockchain system to store metadata, whereas Redis and MongoDB do not, the system performance of the present invention greatly surpasses these two well-known database systems in problematic environments. This also represents that our blockchain system has matured to be useful in high throughput database applications.
The present embodiment proposes a high-performance, high-fault-tolerance database system, particularly a database system that can effectively protect metadata and can be expanded, that is, a new database architecture based on a blockchain system, where the new database system stores metadata on the blockchain system and stores data in a conventional database structure, thereby forming a high-performance, unstructured, fused blockchain distributed database. The method is characterized in that the characteristics of MySQL relational lines are reserved, the unstructured storage structure of MongoDB is used for reference, the method has excellent performance like Redis, metadata is stored by a blockchain, and high security that the data is not damaged and is not tampered and deleted is ensured.
While the present invention has been described with reference to the particular illustrative embodiments, it is not to be restricted by the embodiments but only by the appended claims. It will be appreciated by those skilled in the art that changes and modifications may be made to the embodiments of the invention without departing from the scope and spirit thereof.
Claims (10)
1. A blockchain system-based database system that supports operation of a blockchain system, comprising:
a built-in blockchain system and a database system, wherein the blockchain system supported by the built-in blockchain system is an external blockchain system; wherein a plurality of nodes are arranged in the external block chain system, each node comprises the database system, and the database system comprises the built-in block chain system;
the external blockchain system is provided with a plurality of nodes, each node is provided with an independent sub-database, and each independent sub-database is provided with an independent built-in blockchain subsystem, so that the external blockchain system comprises at least 16 built-in blockchain nodes, and each built-in blockchain subsystem can independently operate to maintain the functions and the performances of the database system;
all metadata in the database system based on the blockchain system are managed by the built-in blockchain system, the data are stored in the traditional database system, a Connection SDK is externally provided, the Connection SDK is provided with an SDK Api, a user uses the SDK Api to perform CRUD operation on the database system based on an application program, and the CRUD operation represents creation, reading, updating and deleting;
the metadata includes:
DB: data information describing the entire database system;
name: describing the name of a database or the name of a certain table, wherein the tables with the same name cannot exist in the same database;
BaseDir: the path of the database on disk;
hash and preHash: marking a current metadata version, if the current metadata version is updated, assigning a current hash to a pre-hash, updating the current hash to a latest version hash, and when the pre-hash is not equal to the hash, indicating that the metadata version is modified, and the metadata version needs to be serialized to a block chain, wherein when serialization is completed, the hash is equal to the pre-hash;
writer: the mark has the written user authority;
reader: marking the user authority which is read;
TableList: describing all table information under the current database;
BlockList: describing all block information under a current table;
TxList: recording all business data information on the current block, wherein the data information is serialized on a magnetic disk;
ActiviTime: recording the last modification time of the current block, if the database is larger than a pool, indicating that not all blocks can be cached in the pool, and serializing the blocks with the farthest activiTime distance to a disk for reservation;
table: describing basic information of a database system table;
block: information describing the sub-table chunks;
index: describing all index data on the table;
tx: the most original transaction data of the user is saved, and the transaction data is a JSON character string type.
2. The blockchain system-based database system of claim 1, wherein the external blockchain system has a plurality of nodes, some of which use a legacy database system, none of which use an internal blockchain system to store metadata, and some or all of which use the blockchain system-based database system to store metadata.
3. The blockchain-based system database system of claim 1, wherein each of the database systems uses a different kind of database, including a relational database.
4. A blockchain system-based database system according to claim 3, wherein the database system forms a cluster, a plurality of Master processes, a backup process and a Connection control are arranged in the cluster, the Master process is a database process for storing data, the backup process is a process responsible for backing up the masters, when the Master process fails, the backup process replaces the Master process to complete work, process information of the Master process and the backup process are concentrated together and stored on the blockchain system for management, and the Connection control obtains global process information through the blockchain system and distributes a command of the CRUD operation to each cluster for execution after Connection is established; the Master process is provided with a plurality of Master sub-processes which are independently operated, and the number of the Master sub-processes is increased or reduced according to the requirements of different database systems; the backup process is provided with a plurality of backup sub-processes which are independently operated, and the number of the backup sub-processes is increased or reduced according to the requirements of different database systems; the processes have different relations, including a process which is a Master and a process which is a Worker which listens to the Master; the number of processes increases or decreases depending on the application environment, with multiple processes being handled and run on large machines and correspondingly decreasing on small systems.
5. The blockchain-based database system of claim 4, wherein in one of the external blockchain systems, the database system in each node uses different databases, even if the same database is used, using different or mixed modes to store data, including structured data as well as unstructured data.
6. The blockchain system-based database system of claim 5, wherein the metadata and data information stored includes 4 layers:
layer 1 layer data information: storing basic principles of the database system, thereby maintaining basic information of the database system, including database names, paths of the database on the disk, list information, as part of metadata;
layer 2 layer data information: the list of layer 1 data information points to data information of a Table structure of layer 2 data information as a part of metadata;
layer 3 data information: the table structure of layer 2 data information points to Block and Index data structure information as part of metadata;
layer 4 layer data information: actual transaction information as data.
7. The blockchain system-based database system of claim 1, wherein the blockchain system-based database system performs the CRUD operation of data using a straightforward object-oriented query statement OQL or using a traditional database query language; the database system based on the block chain system is provided with a database expansibility mechanism and a metadata fault tolerance mechanism.
8. A method of operating a blockchain-based database system according to any of claims 1-7, implemented using a Request-Response mode, comprising:
step 1, a client initiates a request through connection, and the request is handed to a right manager to verify the right;
step 2, submitting the request to other auditing tasks for verification after passing the verification;
step 3, after verification by the other auditing tasks, the information is submitted to a Table, and the Table analyzes whether metadata of an Index needs to be modified;
step 4, calculating the blockId of the metadata to be operated, and sending an operation result to a corresponding BlockManager according to the blockId after performing the CRUD operation on the metadata;
step 5, the information is sent to Pool, a Block entity is obtained according to the Block Id, and the obtaining is achieved through caching or a magnetic disk;
step 6, executing corresponding CRUD operation according to the obtained Block entity and the request;
and 7, writing the result after the operation into a response object and returning to connection.
9. The method for operating a BlockManager based database system of claim 8, wherein step 4 employs different BlockManager algorithms, including: table processes multiple tables at a time; the Blockmanager creates a plurality of blocks each time; the unused blocks are stored first and are used after waiting for later use.
10. The method of claim 8, further comprising: metadata update, comprising: all metadata are stored in a memory to ensure performance except on a blockchain system, when each process is started, whether the pre Hash of the metadata is consistent with the Hash of the metadata or not is firstly inquired, and if so, the process is executed; if not, reading the updated metadata back from the blockchain system and then executing 'PreHash=Hash'; each time the database needs to update metadata, the metadata is changed, the Hash is updated, and the updated metadata is stored in the built-in blockchain system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111531166.2A CN114168685B (en) | 2021-12-15 | 2021-12-15 | Novel database architecture based on blockchain system and operation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111531166.2A CN114168685B (en) | 2021-12-15 | 2021-12-15 | Novel database architecture based on blockchain system and operation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114168685A CN114168685A (en) | 2022-03-11 |
| CN114168685B true CN114168685B (en) | 2023-07-18 |
Family
ID=80486588
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111531166.2A Active CN114168685B (en) | 2021-12-15 | 2021-12-15 | Novel database architecture based on blockchain system and operation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114168685B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106897878A (en) * | 2017-02-27 | 2017-06-27 | 钱德君 | A kind of intelligent contract type implementation of quantum catenary system |
| CN112035466A (en) * | 2020-07-29 | 2020-12-04 | 北京智融云河科技有限公司 | Block chain query external index development framework |
| CN112115116A (en) * | 2020-08-05 | 2020-12-22 | 西安电子科技大学 | Block chain-based high-performance tamper-resistant database construction method |
| CN112564985A (en) * | 2020-12-24 | 2021-03-26 | 南京联成科技发展股份有限公司 | Safe operation and maintenance management method based on block chain |
| CN112602076A (en) * | 2018-08-24 | 2021-04-02 | 甲骨文国际公司 | DAG-based transaction processing method and system in distributed ledger |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10868674B2 (en) * | 2016-08-12 | 2020-12-15 | ALTR Solutions, Inc. | Decentralized database optimizations |
| CN107766453A (en) * | 2017-09-26 | 2018-03-06 | 上海策赢网络科技有限公司 | Data base management method, device and storage medium based on block chain |
| CN108023879A (en) * | 2017-11-30 | 2018-05-11 | 济南浪潮高新科技投资发展有限公司 | A kind of NB-IOT device data acquisition system and methods based on block chain |
| CN110019235B (en) * | 2017-12-29 | 2021-08-17 | 百度在线网络技术（北京）有限公司 | Block chain based data storage and recovery method, device, equipment and medium |
| CN111241590A (en) * | 2018-11-29 | 2020-06-05 | 华为技术有限公司 | Database system, node and method |
| US11899817B2 (en) * | 2019-01-31 | 2024-02-13 | Salesforce, Inc. | Systems, methods, and apparatuses for storing PII information via a metadata driven blockchain using distributed and decentralized storage for sensitive user information |
| CN110232080B (en) * | 2019-05-23 | 2021-06-29 | 智慧谷(厦门)物联科技有限公司 | Rapid retrieval method based on block chain |
| WO2021034274A1 (en) * | 2019-08-20 | 2021-02-25 | Singapore University Of Technology And Design | Blockchain for operational data security in industrial control systems |
| SG11202010724PA (en) * | 2019-11-06 | 2020-11-27 | Alipay Hangzhou Inf Tech Co Ltd | Consenus of shared blockchain data storage based on error correction code |
| CN111400761B (en) * | 2020-03-17 | 2022-04-22 | 吉林亿联银行股份有限公司 | Data sharing method and device, storage medium and electronic equipment |
| CN112134834B (en) * | 2020-05-19 | 2021-05-25 | 北京天德科技有限公司 | Data lake system architecture based on block chain |
| CN112714172A (en) * | 2020-12-23 | 2021-04-27 | 平安普惠企业管理有限公司 | Data synchronization method and device based on block chain technology and computer equipment |
-
2021
- 2021-12-15 CN CN202111531166.2A patent/CN114168685B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106897878A (en) * | 2017-02-27 | 2017-06-27 | 钱德君 | A kind of intelligent contract type implementation of quantum catenary system |
| CN112602076A (en) * | 2018-08-24 | 2021-04-02 | 甲骨文国际公司 | DAG-based transaction processing method and system in distributed ledger |
| CN112035466A (en) * | 2020-07-29 | 2020-12-04 | 北京智融云河科技有限公司 | Block chain query external index development framework |
| CN112115116A (en) * | 2020-08-05 | 2020-12-22 | 西安电子科技大学 | Block chain-based high-performance tamper-resistant database construction method |
| CN112564985A (en) * | 2020-12-24 | 2021-03-26 | 南京联成科技发展股份有限公司 | Safe operation and maintenance management method based on block chain |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114168685A (en) | 2022-03-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6448609B2 (en) | Data acquisition method and system | |
| JP5411250B2 (en) | Data placement according to instructions to redundant data storage system | |
| US9146934B2 (en) | Reduced disk space standby | |
| CA2913036C (en) | Index update pipeline | |
| CN101567805B (en) | Method for recovering failed parallel file system | |
| US20090012932A1 (en) | Method and System For Data Storage And Management | |
| US20140032496A1 (en) | Information storage system and data replication method thereof | |
| US10754854B2 (en) | Consistent query of local indexes | |
| JP2016524750A5 (en) | ||
| US8832022B2 (en) | Transaction processing device, transaction processing method and transaction processing program | |
| US20210240585A1 (en) | Database-level automatic storage management | |
| WO2012108015A1 (en) | Data synchronization method, data synchronization program, and data synchronization control device | |
| WO2016192496A1 (en) | Data migration processing method and device | |
| US11860892B2 (en) | Offline index builds for database tables | |
| EP4307137A1 (en) | Transaction processing method, distributed database system, cluster, and medium | |
| JP5721056B2 (en) | Transaction processing apparatus, transaction processing method, and transaction processing program | |
| CN114168685B (en) | Novel database architecture based on blockchain system and operation method | |
| KR102214697B1 (en) | A computer program for providing space managrment for data storage in a database management system | |
| KR20230118269A (en) | Method and apparatus for managing meta information in schema level | |
| CN114661738A (en) | Method, device and equipment for adding index and storage medium | |
| KR20210013747A (en) | A computer program for providing space managrment for data storage in a database management system | |
| CN111966658A (en) | Space-saving security information data management and storage method | |
| CN114168572A (en) | Method and device for managing database | |
| CN116401313A (en) | Shared storage database cluster information synchronization method | |
| CN118093584A (en) | Method and system for quickly storing big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |