CN111858520B - Method and device for separately storing block chain node data - Google Patents
Method and device for separately storing block chain node data Download PDFInfo
- Publication number
- CN111858520B CN111858520B CN202010704665.6A CN202010704665A CN111858520B CN 111858520 B CN111858520 B CN 111858520B CN 202010704665 A CN202010704665 A CN 202010704665A CN 111858520 B CN111858520 B CN 111858520B
- Authority
- CN
- China
- Prior art keywords
- data
- database
- cold
- file
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000005057 refrigeration Methods 0.000 claims abstract description 53
- 238000000926 separation method Methods 0.000 claims abstract description 34
- 238000010257 thawing Methods 0.000 claims abstract description 19
- 230000006837 decompression Effects 0.000 claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 12
- 230000008569 process Effects 0.000 claims description 14
- 238000012217 deletion Methods 0.000 claims 1
- 230000037430 deletion Effects 0.000 claims 1
- 238000004422 calculation algorithm Methods 0.000 description 25
- 238000010586 diagram Methods 0.000 description 9
- 230000006835 compression Effects 0.000 description 7
- 238000007906 compression Methods 0.000 description 7
- 238000013500 data storage Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005242 forging Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the specification discloses a block chain node data separation storage method and device, wherein a cold database and a warm database of block chain nodes are created in a block chain network of the scheme, the cold database is used for storing low-frequency access data, the warm database is used for storing data recovered from the cold database, and the method comprises the following steps: determining the node data meeting a preset data separation storage rule; storing the node data to the cold database for data refrigeration operations; when the use requirement exists, acquiring the data meeting the use condition from the cold database to perform data thawing operation so as to restore the data to the warm database, wherein the warm data restored to the warm database does not need decompression processing. Therefore, the storage space cost of a single node of the block chain is reduced by separating and storing the block chain link point data; meanwhile, the speed of the newly added node of the blockchain for synchronizing the blockchain is improved, and the consumption of network bandwidth is reduced.
Description
Technical Field
The present disclosure relates to the field of computer software technologies, and in particular, to a method, an apparatus, an electronic device, and a storage medium for separately storing block link point data.
Background
The blockchain is a shared database, and data or information stored in the shared database has the characteristics of incapability of forging, whole trace, traceability, openness, transparency, collective maintenance and the like. Because the blockchain is in a chain structure, each blockchain node needs to store all historical data, and the data is always accumulated and grows, so that the data storage space is not greatly increased, and the storage cost is high. The data is separated and stored regularly according to a specific rule, so that more data storage can be provided in a stable data storage space, and the data required by an application program can be quickly recovered; meanwhile, when the block chain is newly added with nodes, the initialization construction of the complete data can be rapidly realized. Therefore, establishing different storage tiers of blockchain node data is critical to improving the database performance of the nodes.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method and an apparatus for separately storing blockchain node data, wherein a warm database and a cold database are created in a blockchain network in advance, and hot data is refrigerated according to a separate storage rule or cold data is thawed according to a usage requirement. In order to solve the above technical problems, the embodiments of the present specification are implemented as follows:
in a first aspect, a method for separately storing blockchain node data is provided, in which a cold database and a warm database of blockchain nodes are created in a blockchain network, wherein the cold database is used for storing low-frequency access data, and the warm database is used for storing data recovered from the cold database, and the method includes:
determining node data meeting preset data separation storage rules;
storing the node data to the cold database for data refrigeration operations;
when a use requirement exists, acquiring the data meeting the use condition from the cold database to perform data thawing operation so as to restore the data to the warm database, wherein the warm data restored to the warm database does not need decompression processing.
In a second aspect, a block chain link point data separation storage device is provided, in which a cold database and a warm database with block chain nodes are created in a block chain network, wherein the cold database is used for storing low frequency access data, and the warm database is used for storing data recovered from the cold database, the device includes:
the data determining module is used for determining node data meeting preset data separation and storage rules;
the data refrigeration module is used for storing the node data to the cold database to perform data refrigeration operation;
and the data thawing module is used for obtaining the data meeting the use conditions from the cold database to carry out data thawing operation when the use requirement exists, so that the data is restored to the warm database, wherein the warm data restored to the warm database does not need decompression processing.
In a third aspect, an electronic device is provided, comprising: a processor; and
a memory arranged to store computer executable instructions which, when executed, cause the processor to perform the method of the first aspect.
In a fourth aspect, a computer readable storage medium is presented, the computer readable storage medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the method of the first aspect.
The specification can achieve at least the following technical effects:
according to the invention, the block chain node data is stored separately, so that the storage space cost of a single block chain node can be reduced; meanwhile, the speed of the newly added node of the blockchain for synchronizing the blockchain is improved, and the consumption of network bandwidth is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present description or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some of the embodiments described in the present description, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a block chain node data separate storage method according to one embodiment of the present disclosure.
FIG. 2 is a second step diagram of a block chain node data separate storage method according to an embodiment of the present disclosure.
FIG. 3 is a third exemplary block chain node data separate storage method according to the present embodiment.
FIG. 4 is a diagram illustrating a fourth exemplary step of a method for storing data of a blockchain node in a separate manner according to the embodiments of the present disclosure.
FIG. 5 is a fifth step diagram of a block chain node data separate storage method according to an embodiment of the present disclosure.
FIG. 6 is a diagram illustrating a sixth exemplary step of a method for storing data of a blockchain node in a separate manner according to the embodiments of the present disclosure.
FIG. 7 is a schematic diagram of a block chain node data separation storage device according to an embodiment of the present disclosure.
Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
In order to make the technical solutions in the present specification better understood by those skilled in the art, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.
Key terms
Blockchain techniques: the system is a brand new distributed infrastructure and calculation paradigm which uses a block chain type data structure to verify and store data, uses a distributed node consensus algorithm to generate and update data, uses a cryptography mode to ensure the safety of data transmission and access, and uses an intelligent contract consisting of automatic script codes to program and operate the data. Essentially, the system is a shared database, and data or information stored in the shared database has the characteristics of incapability of counterfeiting, whole trace, traceability, openness, transparency, collective maintenance and the like. Based on the characteristics, the blockchain technology lays a solid trust foundation, creates a reliable cooperation consensus mechanism and has wide application prospect.
Thermal database: for storing data requiring high frequency access, without decompression, can be accessed directly.
Cold database: in general, in order to increase the storage space rate, data is compressed and then stored, and then decompressed for access. May be a local low-speed storage device, network-attached storage in a local area network, object storage service in public cloud service, etc.
A warm database: for storing data obtained from a cold database through a data thawing operation, warm data generally does not need to be decompressed in a similar manner as the hot database storage, but the data in the warm database is deleted as the heat drops.
And (3) data refrigeration: and transferring and storing the data requiring high frequency access to a cold database.
Refrigerating a file: namely, the data files which meet the separation storage rule and are stored in the cold database after compression processing;
data thawing: a process of obtaining data from the cold database and storing it to the warm database according to usage needs.
Data heat degree: according to the threshold value calculated according to the time and the range frequency, the calculation mode of the heat can be freely realized according to the access requirement of each node.
The blockchain node data splitting storage scheme referred to in this specification is detailed below by way of specific examples.
Example 1
In order to solve the problem of shortage of storage space caused by continuous growth of block chain system data, the invention provides a method for defining the block chain data as hot, warm and hot types, performing compression storage after cold-hot separation, and improving the space utilization rate of the system. The method comprises the following steps: when the block chain system reaches a preset cold data separation rule, if the block chain system reaches a preset height value, the block data of the block height in a set range can be compressed and stored, and index data are generated; when the cold data is required to be accessed, the cold data is acquired from a local cold database or cold databases of other nodes, decompression inquiry is carried out, and the data meeting the inquiry conditions is written into a warm database, so that the inquiry efficiency is improved. The newly added node can respectively acquire compressed cold data in different ranges from a plurality of nodes to perform data synchronization, so that the network bandwidth is reduced, and the synchronization data is improved.
Referring to fig. 1a, a schematic step diagram of a blockchain node data separation and storage method according to an embodiment of the present disclosure is shown, where the method may include the following steps:
step 101: a hot database, a warm database, and a cold database of blockchain nodes are created.
The blockchain data is distributed on the nodes for storage, each blockchain node needs to store all historical data, and the data is accumulated and grows all the time, so that the data storage space is not greatly increased, and the storage cost is high. The data may be stored separately, considering how frequently the blockchain data is used. The separation storage rule is set manually, and data is divided into data which is not accessed for a long time, i.e., cold data, and data which is required to be accessed at a high frequency, i.e., hot data. However, the frequency of data usage is not constant, and cold data that is not accessed for a long period of time may be classified as warm data because the application needs to be restored to service, i.e., the data restored from the cold database. Thus, corresponding to hot data, cold data, and warm data, a hot database, a cold database, and a warm database may be created for the nodes, respectively.
It should be understood that the database herein is not necessarily commercial database software, but is a space in which data can be stored, and the storage form of the database may be a key-value type, a relational type, a file system using an operating system as a storage manner, and the like.
Step 102: and carrying out data refrigeration operation on the node data meeting the preset data separation and storage rule.
Optionally, as shown in fig. 2, the data refrigeration operation includes:
step 121: and constructing a transaction hash record index of the data. Namely, transaction hash records in all blocks in the refrigeration range are used as index tables, and a transaction record hash index table is obtained;
step 122: and compressing the data file to form a refrigeration file, and generating the refrigeration file hash. Namely, all block data in the refrigeration range are stored independently to generate a compressed packet, and a hash algorithm is carried out on the compressed packet to obtain a file hash; it should be appreciated that all nodes require the use of consistent file compression algorithms, file decompression algorithms, and consistent file hashing algorithms. The hash algorithm can calculate hash values for files, data and the like, and common verification hash algorithms in the computer field are MD5, SHA and the like, and the form is a string of 16-system characters with the length of 32 to 256 bytes and the like, for example: the hash value of the character "hi" obtained using the SHA hash algorithm is "55ca6286e3e4f4fba5d0448333fa99fc5a404a73".
Step 123: storing the data separation range, the transaction record hash index, the refrigeration file and the refrigeration file hash into the cold database;
step 124: deleting the data transferred to the cold database from the hot database.
The data refrigeration operation is illustrated as follows: if the block height in the current block chain is 30 ten thousand, 0 to 10 ten thousand blocks and 10 ten thousand to 20 ten thousand blocks of data can be set as cold data, and the separation storage rule is determined according to the block height range. The method comprises the following steps:
separation storage rule: data with the height range of the refrigerating block being 0-100000;
refrigerating a file: "0-100000-blocks. Zip"
Refrigerating file hash: obtained by calculation with a hash algorithm, e.g. shasum (0-100000-blocks. Zip) = "dd09ee53f024a691dabc4777b03d02e9b1e15b46"
Transaction record hash index table: 0-100000-blocks.
Step 103: when a use requirement exists, acquiring the data meeting the use condition from the cold database to perform data thawing operation so as to restore the data to the warm database, wherein the warm data restored to the warm database does not need decompression processing.
Optionally, as shown in fig. 3, the working procedure of the data thawing operation includes:
step 131: judging whether related data exist in the thermal database, if so, reading the data and returning a query result;
step 132: if the hot database has no related data, judging whether the hot database has related data or not, if so, reading the data, improving the data heat, and returning a query result;
it should be appreciated that the data popularity characterizes how frequently the data is used based on at least one of the data access time or number of accesses, and that the particular calculation algorithm of data popularity may vary depending on the access needs of the node itself. In general, for access frequencies, the heat is high if the van frequency is high; the access time, the longer the last access time, the lower.
Step 133: if the warm database has no related data, comparing the hash indexes of the transaction records to determine the storage range of the data in the cold database;
step 134: if the cold database of the current node has the corresponding cold storage file, acquiring the cold storage file from the cold database of the current node, otherwise acquiring the cold storage file from other nodes of the blockchain;
step 135: and storing the cold storage to the warm database of the current node.
Optionally, the cold data stored in the current node is selectively deleted, and the working process is as follows in fig. 5:
step 151: broadcasting inquiry is carried out on the block chain, and whether the data in the corresponding range is stored in a node of the block chain or not is judged;
step 152: if the number of the nodes storing the data in the corresponding range is larger than a preset threshold value, deleting the refrigeration files in the designated range in the cold database of the node; otherwise, the refrigerated file is reserved.
Further, referring to fig. 1b, the blockchain node data separate storage method may further include the following steps based on fig. 1 a:
step 104: the blockchain newly added node acquires data from the cold database of the existing node so that the newly added node has complete data.
Optionally, the working process of the newly added node for obtaining cold data from other nodes is as follows in fig. 4:
step 141: broadcasting the block chain to obtain the hash of the refrigerating files of the cold database of other nodes;
step 142: obtaining the refrigerated file from the nearest node cold database;
step 143: and comparing the refrigerated file with the refrigerated file hash so as to restore the data with consistent comparison into the hot database.
In another alternative, referring to fig. 6a, a schematic diagram of steps of a blockchain node data separation and storage method according to an embodiment of the present disclosure is provided. A hot database, a warm database, and a cold database of blockchain nodes are created in advance. The blockchain data is distributed on the nodes for storage, each blockchain node needs to store all historical data, and the data is accumulated and grows all the time, so that the data storage space is not greatly increased, and the storage cost is high. The data may be stored separately, considering how frequently the blockchain data is used. The separation storage rule is set manually, and data is divided into data which is not accessed for a long time, i.e., cold data, and data which is required to be accessed at a high frequency, i.e., hot data. However, the frequency of data usage is not constant, and cold data that is not accessed for a long period of time may be classified as warm data because the application needs to be restored to service, i.e., the data restored from the cold database. Thus, corresponding to hot data, cold data, and warm data, a hot database, a cold database, and a warm database may be created for the nodes, respectively. It should be understood that the database herein is not necessarily commercial database software, but is a space in which data can be stored, and the storage form of the database may be a key-value type, a relational type, a file system using an operating system as a storage manner, and the like. The method specifically comprises the following steps:
step 111: determining node data meeting preset data separation storage rules;
step 112: and storing the node data to the cold database for data refrigeration operation.
Optionally, as shown in fig. 2, the data refrigeration operation includes:
step 121: and constructing a transaction hash record index of the data. Namely, transaction hash records in all blocks in the refrigeration range are used as index tables, and a transaction record hash index table is obtained;
step 122: and compressing the data file to form a refrigeration file, and generating the refrigeration file hash. Namely, all block data in the refrigeration range are stored independently to generate a compressed packet, and a hash algorithm is carried out on the compressed packet to obtain a file hash; it should be appreciated that all nodes require the use of consistent file compression algorithms, file decompression algorithms, and consistent file hashing algorithms. The hash algorithm can calculate hash values for files, data and the like, and common verification hash algorithms in the computer field are MD5, SHA and the like, and the form is a string of 16-system characters with the length of 32 to 256 bytes and the like, for example: the hash value of the character "hi" obtained using the SHA hash algorithm is "55ca6286e3e4f4fba5d0448333fa99fc5a404a73".
Step 123: storing the data separation range, the transaction record hash index, the refrigeration file and the refrigeration file hash into the cold database;
step 124: deleting the data transferred to the cold database from the hot database.
The data refrigeration operation is illustrated as follows: if the block height in the current block chain is 30 ten thousand, 0 to 10 ten thousand blocks and 10 ten thousand to 20 ten thousand blocks of data can be set as cold data, and the separation storage rule is determined according to the block height range. The method comprises the following steps:
separation storage rule: data with the height range of the refrigerating block being 0-100000;
refrigerating a file: "0-100000-blocks. Zip"
Refrigerating file hash: obtained by calculation with a hash algorithm, e.g. shasum (0-100000-blocks. Zip) = "dd09ee53f024a691dabc4777b03d02e9b1e15b46"
Transaction record hash index table: 0-100000-blocks.
Step 113: when a use requirement exists, acquiring the data meeting the use condition from the cold database to perform data thawing operation so as to restore the data to the warm database, wherein the warm data restored to the warm database does not need decompression processing.
Optionally, as shown in fig. 3, the working procedure of the data thawing operation includes:
step 131: judging whether related data exist in the thermal database, if so, reading the data and returning a query result;
step 132: if the hot database has no related data, judging whether the hot database has related data or not, if so, reading the data, improving the data heat, and returning a query result;
it should be appreciated that the data popularity characterizes how frequently the data is used based on at least one of the data access time or number of accesses, and that the particular calculation algorithm of data popularity may vary depending on the access needs of the node itself. In general, for access frequencies, the heat is high if the van frequency is high; the access time, the longer the last access time, the lower.
Step 133: if the warm database has no related data, comparing the hash indexes of the transaction records to determine the storage range of the data in the cold database;
step 134: if the cold database of the current node has the corresponding cold storage file, acquiring the cold storage file from the cold database of the current node, otherwise acquiring the cold storage file from other nodes of the blockchain;
step 135: and storing the cold storage to the warm database of the current node.
Optionally, the cold data stored in the current node is selectively deleted, and the working process is as follows in fig. 5:
step 151: broadcasting inquiry is carried out on the block chain, and whether the data in the corresponding range is stored in a node of the block chain or not is judged;
step 152: if the number of the nodes storing the data in the corresponding range is larger than a preset threshold, deleting the refrigeration file in the specified range in the cold database of the node, otherwise, reserving the refrigeration file.
Referring to fig. 6b, based on the scheme shown in fig. 6a, the blockchain node data separate storage method further includes:
step 114: the blockchain newly added node acquires data from the cold database of the existing node so that the newly added node has complete data.
Optionally, the working process of the newly added node for obtaining cold data from other nodes is as follows in fig. 4:
step 141: broadcasting the block chain to obtain the hash of the refrigerating files of the cold database of other nodes;
step 142: obtaining the refrigerated file from the nearest node cold database;
step 143: and comparing the refrigerated file with the refrigerated file hash so as to restore the data with consistent comparison into the hot database.
According to the invention, the block chain node data is stored separately, so that the storage space cost of a single block chain node can be reduced; meanwhile, the speed of the newly added node of the blockchain for synchronizing the blockchain is improved, and the consumption of network bandwidth is reduced.
Example two
Fig. 7 is a schematic structural diagram of a block link point data separation storage device 700 according to an embodiment of the present disclosure. Referring to fig. 7, in one embodiment, a block link point data separation storage device 700 includes:
a data determining module 701, configured to determine node data that satisfies a preset data separation storage rule;
the data refrigeration module 702 is configured to perform a data refrigeration operation on the blockchain node data satisfying a preset data separation storage rule, so that the data is stored in a cold database, and the working process is as follows:
constructing a transaction hash record index of the data, namely taking transaction hash records in all blocks in a refrigerating range as an index table to obtain a transaction record hash index table;
compressing the data file to form a refrigeration file, generating a hash of the refrigeration file, namely, independently storing all block data in a refrigeration range, generating a compression packet, and performing a hash algorithm on the compression packet to obtain a file hash; it should be appreciated that all nodes require the use of a consistent file compression algorithm, a file decompression algorithm, and a consistent file hashing algorithm;
storing the data separation range, the transaction record hash index, the refrigeration file and the refrigeration file hash into the cold database;
deleting the data transferred to the cold database from the hot database.
The data thawing module 703 obtains the data satisfying the use condition from the cold database to perform a data thawing operation when the data subjected to data refrigeration is used, so as to restore the data to a warm database, and the working process is as follows:
judging whether related data exists in the thermal database, if so, reading the data and returning a query result;
if the hot database has no related data, judging whether the hot database has related data or not, if so, reading the data, improving the data heat, and returning a query result;
if the warm database has no related data, comparing the hash indexes of the transaction records to determine the storage range of the data in the cold database;
if the cold database of the current node has the corresponding cold storage file, acquiring the cold storage file from the cold database of the current node, otherwise acquiring the cold storage file from other nodes of the blockchain;
and decompressing the refrigerated files obtained from the process by using a consistent decompression algorithm, and finally storing the refrigerated files into the warm database of the current node.
It should be appreciated that the data popularity characterizes how frequently the data is used based on at least one of the data access time or number of accesses, and that the particular calculation algorithm of data popularity may vary depending on the access needs of the node itself. In general, for access frequencies, the heat is high if the van frequency is high; the access time, the longer the last access time, the lower.
Optionally, the blockchain node data separate storage device 700 further includes:
the node data initializing module is used for acquiring data from the cold database of the existing node by the newly added node of the blockchain so that the newly added node has complete data, and the working process is as follows:
broadcasting the block chain to obtain the hash of the refrigerating files of the cold database of other nodes;
obtaining the refrigerated file from the nearest node cold database;
and comparing the refrigerated file with the refrigerated file hash so as to restore the data with consistent comparison into the hot database.
Optionally, the blockchain node data separate storage device 700 further includes:
the cold database management module is used for selectively deleting cold data stored by the current node, and the working process is as follows:
broadcasting inquiry is carried out on the block chain, and whether the data in the corresponding range is stored in a node of the block chain or not is judged;
if the number of the nodes storing the data in the corresponding range is larger than a preset threshold value, deleting the refrigeration files in the designated range in the cold database of the node; otherwise, the refrigeration file is reserved so as to avoid the loss of the block chain cold data.
It should be understood that the blockchain node data splitting storage device of the embodiments of the present disclosure may also perform the method performed by the blockchain node data splitting storage device (or apparatus) in fig. 1 to 6, and implement the functions of the blockchain node data splitting storage device (or apparatus) in the embodiments shown in fig. 1 to 6, which are not described herein.
Example III
Fig. 8 is a schematic structural view of an electronic device according to an embodiment of the present specification. Referring to fig. 8, at the hardware level, the electronic device includes a processor, and optionally an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA (Industry Standard Architecture ) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus, or EISA (Extended Industry Standard Architecture ) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 8, but not only one bus or type of bus.
And the memory is used for storing programs. In particular, the program may include program code including computer-operating instructions. The memory may include memory and non-volatile storage and provide instructions and data to the processor.
The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs, and forms a shared resource access control device on a logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:
after creating the hot database, the warm database, and the cold database of the blockchain node, it includes:
performing data refrigeration operation on the blockchain node data meeting a preset data separation and storage rule so as to store the data into the cold database;
and when the data subjected to data refrigeration is used, acquiring the data meeting the use condition from the cold database, and performing data thawing operation so as to restore the data to the warm database.
The blockchain node data separate storage method disclosed in the embodiments shown in fig. 1 to 6 of the present specification may be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps and logic blocks disclosed in the embodiments of the present specification may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present specification may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.
Of course, in addition to the software implementation, the electronic device of the embodiments of the present disclosure does not exclude other implementations, such as a logic device or a combination of software and hardware, that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or a logic device.
Example IV
The present description also proposes a computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a portable electronic device comprising a plurality of application programs, enable the portable electronic device to perform the method of the embodiments shown in fig. 1-6, and in particular to perform the method of:
after creating the hot database, the warm database, and the cold database of the blockchain node, it includes:
performing data refrigeration operation on the blockchain node data meeting a preset data separation and storage rule so as to store the data into the cold database;
and when the data subjected to data refrigeration is used, acquiring the data meeting the use condition from the cold database, and performing data thawing operation so as to restore the data to the warm database.
In summary, the foregoing description is only a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification should be included in the protection scope of the present specification.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
Claims (10)
1. A blockchain node data separation storage method, wherein a cold database and a warm database of blockchain nodes are created in a blockchain network, wherein the cold database is used for storing low-frequency access data, and the warm database is used for storing data recovered from the cold database, the method comprising:
determining node data meeting preset data separation storage rules;
storing the node data to the cold database for data refrigeration operations;
when a use requirement exists, acquiring the data meeting the use condition from the cold database to perform data thawing operation so as to restore the data to the warm database, wherein the warm data restored to the warm database does not need decompression processing;
a hot database of the blockchain nodes is also created in the blockchain network and is used for storing data needing high-frequency access;
the storing the node data in the cold database for data refrigeration operation specifically comprises:
constructing a transaction hash record index of the data;
compressing the data file to form a refrigeration file, and generating a hash of the refrigeration file;
storing the data separation range, the transaction record hash index, the refrigeration file and the refrigeration file hash into the cold database;
deleting the data transferred to the cold database from the hot database;
the data thawing operation includes:
judging whether related data exist in the thermal database, if so, reading the data and returning a query result;
if the hot database has no related data, judging whether the hot database has related data or not, if so, reading the data, improving the data heat, and returning a query result;
if the warm database has no related data, comparing the hash indexes of the transaction records to determine the storage range of the data in the cold database;
if the cold database of the current node has the corresponding cold storage file, acquiring the cold storage file from the cold database of the current node, otherwise acquiring the cold storage file from other nodes of the blockchain;
decompressing the refrigerated file, and storing the decompressed refrigerated file to the warm database of the current node.
2. The method of claim 1, wherein the data popularity characterizes how frequently the data is used based on at least one of the data access time or number of accesses.
3. The method of claim 1, wherein the data is deleted from the warm database when the heat of use of the warm database data is below a preset deletion threshold.
4. The method of claim 1, further comprising the blockchain newly added node obtaining data from the cold database of existing nodes such that the newly added node has complete data.
5. The method of claim 4, wherein the newly added node obtaining cold data from other nodes comprises:
broadcasting the block chain to obtain hash of the cold database refrigerating file of other nodes;
obtaining the refrigerated file from the nearest node cold database;
and comparing the refrigerated file with the refrigerated file hash so as to restore the data with consistent comparison into the hot database.
6. A blockchain point data separation storage device characterized by creating a cold database with blockchain nodes and a warm database in a blockchain network, wherein the cold database is used for storing low frequency access data and the warm database is used for storing data recovered from the cold database, the device comprising:
the data determining module is used for determining node data meeting preset data separation and storage rules;
the data refrigeration module is used for storing the node data to the cold database to perform data refrigeration operation;
the data thawing module is used for obtaining the data meeting the use conditions from the cold database to conduct data thawing operation when the use requirement exists, so that the data is restored to the warm database, wherein the warm data restored to the warm database does not need decompression processing;
a hot database of the blockchain nodes is also created in the blockchain network and is used for storing data needing high-frequency access; the working process of the data refrigeration module comprises the following steps:
constructing a transaction hash record index of the data;
compressing the data file to form a refrigeration file, and generating a hash of the refrigeration file;
storing the data separation range, the transaction record hash index, the refrigeration file and the refrigeration file hash into the cold database;
deleting the data transferred to the cold database from the hot database;
the data thawing module comprises:
judging whether related data exist in the thermal database, if so, reading the data and returning a query result;
if the hot database has no related data, judging whether the hot database has related data or not, if so, reading the data, improving the data heat, and returning a query result;
if the warm database has no related data, comparing the hash indexes of the transaction records to determine the storage range of the data in the cold database;
if the cold database of the current node has the corresponding cold storage file, acquiring the cold storage file from the cold database of the current node, otherwise acquiring the cold storage file from other nodes of the blockchain;
decompressing the refrigerated file, and storing the decompressed refrigerated file to the warm database of the current node.
7. The apparatus of claim 6, further comprising a node data initialization module for the blockchain newly added node to obtain data from the cold database of an existing node to have the newly added node have complete data.
8. The apparatus of claim 7, wherein the node data initialization module comprises:
broadcasting the block chain to obtain the hash of the refrigerating files of the cold database of other nodes;
obtaining the refrigerated file from the nearest node cold database;
and comparing the refrigerated file with the refrigerated file hash so as to restore the data with consistent comparison into a hot database.
9. An electronic device, comprising:
a processor; and
a memory arranged to store computer executable instructions which, when executed, cause the processor to perform the method of any of claims 1 to 5.
10. A computer readable storage medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010704665.6A CN111858520B (en) | 2020-07-21 | 2020-07-21 | Method and device for separately storing block chain node data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010704665.6A CN111858520B (en) | 2020-07-21 | 2020-07-21 | Method and device for separately storing block chain node data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111858520A CN111858520A (en) | 2020-10-30 |
| CN111858520B true CN111858520B (en) | 2024-03-22 |
Family
ID=73000748
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010704665.6A Active CN111858520B (en) | 2020-07-21 | 2020-07-21 | Method and device for separately storing block chain node data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111858520B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112286465B (en) * | 2020-11-03 | 2023-02-21 | 浪潮云信息技术股份公司 | Rados gateway filing and storing method and system |
| CN112487027B (en) * | 2020-12-02 | 2022-08-23 | 山东浪潮科学研究院有限公司 | Efficient data query implementation method based on block chain electronic transaction |
| CN112558872A (en) * | 2020-12-10 | 2021-03-26 | 东软集团股份有限公司 | Data processing method and device, storage medium and electronic equipment |
| CN112700021A (en) * | 2021-01-14 | 2021-04-23 | 树根互联技术有限公司 | Maintenance method and device for remote equipment, storage medium and electronic equipment |
| CN112988073B (en) * | 2021-03-26 | 2022-07-19 | 中国人民解放军国防科技大学 | Stepped data storage method and system capable of reducing block chain storage overhead |
| KR102688447B1 (en) * | 2021-10-08 | 2024-07-24 | 중앙대학교 산학협력단 | Data management method and device based on data usage in wide area network environment |
| CN114780023A (en) * | 2022-03-29 | 2022-07-22 | 北京柏睿数据技术股份有限公司 | Method and device for quickly accessing data file |
| CN118296083B (en) * | 2024-06-03 | 2024-08-23 | 中国科学院合肥物质科学研究院 | Distributed account book data three-level read-write expansion blockchain storage method |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104932956A (en) * | 2015-06-19 | 2015-09-23 | 华南理工大学 | Big-data-oriented cloud disaster tolerant backup method |
| CN105677509A (en) * | 2015-12-25 | 2016-06-15 | 北京奇虎科技有限公司 | Method and apparatus for recovering data in database |
| WO2018112805A1 (en) * | 2016-12-21 | 2018-06-28 | 深圳前海达闼云端智能科技有限公司 | Blockchain storage method and device, and node device |
| CN108647361A (en) * | 2018-05-21 | 2018-10-12 | 中国工商银行股份有限公司 | A kind of date storage method, apparatus and system based on block chain |
| CN109241022A (en) * | 2018-09-11 | 2019-01-18 | 天津理工大学 | A kind of archive management system and its ant search algorithm based on blue-ray storage |
| CN109274752A (en) * | 2018-10-10 | 2019-01-25 | 腾讯科技（深圳）有限公司 | The access method and device, electronic equipment, storage medium of block chain data |
| CN109324757A (en) * | 2018-08-22 | 2019-02-12 | 深圳前海微众银行股份有限公司 | Block chain data capacity reduction method, device and storage medium |
| CN109409122A (en) * | 2018-09-13 | 2019-03-01 | 远光软件股份有限公司 | File memory method and its electronic equipment, storage medium |
| CN109525671A (en) * | 2018-11-26 | 2019-03-26 | 远光软件股份有限公司 | Date storage method, electronic equipment and storage medium based on block chain |
| CN109711839A (en) * | 2018-12-13 | 2019-05-03 | 平安科技（深圳）有限公司 | Block chain storage method, device, equipment and storage medium based on data compression |
| CN109857737A (en) * | 2019-01-03 | 2019-06-07 | 平安科技（深圳）有限公司 | A kind of cold and hot date storage method and device, electronic equipment |
| CN110264207A (en) * | 2019-08-15 | 2019-09-20 | 广州蚁比特区块链科技有限公司 | Block chain separation processing method and system |
| CN110263035A (en) * | 2019-05-31 | 2019-09-20 | 阿里巴巴集团控股有限公司 | Data storage, querying method and device and electronic equipment based on block chain |
| CN110704688A (en) * | 2018-07-09 | 2020-01-17 | 上海交通大学 | Block chain separation storage system based on associated data |
| CN110865992A (en) * | 2019-10-30 | 2020-03-06 | 腾讯云计算（北京）有限责任公司 | Retrieval library management method, retrieval device and retrieval medium |
| WO2020108441A1 (en) * | 2018-11-27 | 2020-06-04 | 华为技术有限公司 | Data storage method, device, computer apparatus, and readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019023286A1 (en) * | 2017-07-24 | 2019-01-31 | Martino William | Blockchain-based systems, methods, and apparatus for securing access to information stores |
-
2020
- 2020-07-21 CN CN202010704665.6A patent/CN111858520B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104932956A (en) * | 2015-06-19 | 2015-09-23 | 华南理工大学 | Big-data-oriented cloud disaster tolerant backup method |
| CN105677509A (en) * | 2015-12-25 | 2016-06-15 | 北京奇虎科技有限公司 | Method and apparatus for recovering data in database |
| WO2018112805A1 (en) * | 2016-12-21 | 2018-06-28 | 深圳前海达闼云端智能科技有限公司 | Blockchain storage method and device, and node device |
| CN108647361A (en) * | 2018-05-21 | 2018-10-12 | 中国工商银行股份有限公司 | A kind of date storage method, apparatus and system based on block chain |
| CN110704688A (en) * | 2018-07-09 | 2020-01-17 | 上海交通大学 | Block chain separation storage system based on associated data |
| CN109324757A (en) * | 2018-08-22 | 2019-02-12 | 深圳前海微众银行股份有限公司 | Block chain data capacity reduction method, device and storage medium |
| CN109241022A (en) * | 2018-09-11 | 2019-01-18 | 天津理工大学 | A kind of archive management system and its ant search algorithm based on blue-ray storage |
| CN109409122A (en) * | 2018-09-13 | 2019-03-01 | 远光软件股份有限公司 | File memory method and its electronic equipment, storage medium |
| CN109274752A (en) * | 2018-10-10 | 2019-01-25 | 腾讯科技（深圳）有限公司 | The access method and device, electronic equipment, storage medium of block chain data |
| CN109525671A (en) * | 2018-11-26 | 2019-03-26 | 远光软件股份有限公司 | Date storage method, electronic equipment and storage medium based on block chain |
| WO2020108441A1 (en) * | 2018-11-27 | 2020-06-04 | 华为技术有限公司 | Data storage method, device, computer apparatus, and readable storage medium |
| CN109711839A (en) * | 2018-12-13 | 2019-05-03 | 平安科技（深圳）有限公司 | Block chain storage method, device, equipment and storage medium based on data compression |
| CN109857737A (en) * | 2019-01-03 | 2019-06-07 | 平安科技（深圳）有限公司 | A kind of cold and hot date storage method and device, electronic equipment |
| CN110263035A (en) * | 2019-05-31 | 2019-09-20 | 阿里巴巴集团控股有限公司 | Data storage, querying method and device and electronic equipment based on block chain |
| CN110264207A (en) * | 2019-08-15 | 2019-09-20 | 广州蚁比特区块链科技有限公司 | Block chain separation processing method and system |
| CN110865992A (en) * | 2019-10-30 | 2020-03-06 | 腾讯云计算（北京）有限责任公司 | Retrieval library management method, retrieval device and retrieval medium |
Non-Patent Citations (2)
| Title |
|---|
| NUMA架构下数据热度的内存数据库日志恢复技术;吴刚 等;《计算机科学与探索》;第13卷(第06期);941-949 * |
| 物联网下的区块链访问控制综述;史锦山 等;《软件学报》;20190327;第30卷(第06期);1632-1648 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111858520A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111858520B (en) | Method and device for separately storing block chain node data | |
| CN108319654B (en) | Computing system, cold and hot data separation method and device, and computer readable storage medium | |
| CN109947668B (en) | Method and device for storing data | |
| US20170031948A1 (en) | File synchronization method, server, and terminal | |
| US10606806B2 (en) | Method and apparatus for storing time series data | |
| US20240126817A1 (en) | Graph data query | |
| CN111782707B (en) | Data query method and system | |
| CN104281533A (en) | Data storage method and device | |
| CN112015820A (en) | Method, system, electronic device and storage medium for implementing distributed graph database | |
| CN102456076A (en) | System and method for mass fragment data aggregation | |
| US11249987B2 (en) | Data storage in blockchain-type ledger | |
| US11755540B2 (en) | Chunking method and apparatus | |
| CN111930305A (en) | Data storage method and device, storage medium and electronic device | |
| CN112487027B (en) | Efficient data query implementation method based on block chain electronic transaction | |
| CN111625600B (en) | Data storage processing method, system, computer equipment and storage medium | |
| CN112711564B (en) | Merging processing method and related equipment | |
| CN112181302A (en) | Data multilevel storage and access method and system | |
| CN115129791A (en) | Data compression storage method, device and equipment | |
| CN112579650A (en) | Data processing method and system based on Redis cache | |
| CN112817980B (en) | Data index processing method, device, equipment and storage medium | |
| CN116795790A (en) | Method and device for merging small files, electronic equipment and storage medium | |
| CN113536047A (en) | Graph database data deleting method, system, electronic equipment and storage medium | |
| CN116820323A (en) | Data storage method, device, electronic equipment and computer readable storage medium | |
| CN115129781A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN101930451B (en) | For storing the method and apparatus of at least one inquiry data element of efficiently searching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |