CN111984421A

CN111984421A - Data processing method, device and storage medium

Info

Publication number: CN111984421A
Application number: CN202010913919.5A
Authority: CN
Inventors: 冯世伟
Original assignee: OneConnect Financial Technology Co Ltd Shanghai
Current assignee: OneConnect Financial Technology Co Ltd Shanghai
Priority date: 2020-09-03
Filing date: 2020-09-03
Publication date: 2020-11-24
Anticipated expiration: 2040-09-03
Also published as: CN111984421B; WO2022048358A1

Abstract

The application relates to the technical field of block chains, in particular to a data processing method, a data processing device and a storage medium, which are applied to a server, wherein the method comprises the following steps: separating the reading and writing of the database, respectively dividing the database into a reading set database and a writing set database, and introducing a cache concept; furthermore, when each thread runs, different intervals can be accessed correspondingly, so that multi-thread database access is realized, and the database access pressure is reduced; meanwhile, when the MVCC is verified, the cache can be directly accessed, and a database does not need to be frequently accessed, so that the verification efficiency of the MVCC is improved; in addition, the operation only relates to CPU operation, and is beneficial to improving the performance of the block chain.

Description

Data processing method, device and storage medium

Technical Field

The present application relates to the field of block chaining technologies, and in particular, to a data processing method, apparatus, and storage medium.

Background

The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism and an encryption algorithm. In block chain application, when Multi-Version Concurrency Control (MVCC) is used for realizing Concurrency verification, the database is often required to be read or written for many times, so that the access pressure of the database is too large, and the MVCC verification efficiency is low.

Disclosure of Invention

The embodiment of the application provides a data processing method, a data processing device and a storage medium, which are beneficial to improving the verification efficiency of MVCC.

A first aspect of the embodiments of the present application provides a data processing method, which is applied to a server, and includes:

sending a first information instruction to a first thread, wherein the first information instruction is used for instructing the first thread to execute a first queue to generate N first tasks by using a data set corresponding to N blocks before an Mth block if the data set corresponding to the Mth block is read from a read set database, and N is a positive integer less than or equal to M which is a positive integer;

if the current task of a second thread is determined to be P first tasks generated by acquiring data sets corresponding to P blocks before the N blocks through the first queue, executing to send a second information instruction to the second thread, wherein the second information instruction is used for indicating the second thread to carry out MVCC verification on the P blocks; the second thread is instructed to write the data sets corresponding to the P blocks into a write set cache respectively, and P write events are generated; synchronizing the P writing events into a second queue to generate P third tasks, wherein P is a positive integer less than or equal to N;

if the current task of the third thread is determined to be Q third tasks in the second queue obtained through the write set database, executing to send a third information instruction to the third thread, wherein the third information instruction is used for instructing the third thread to execute the Q third tasks and obtaining the current Block ID in the write set cache; when the execution of any one third task in the Q third tasks is completed, updating the current BolckID to be the BlockID corresponding to the block of the any one third task, where Q is a positive integer less than or equal to P, the Q third tasks correspond to Q blocks before the P blocks, and each third task corresponds to one block.

A second aspect of the embodiments of the present application provides a data processing apparatus, which is applied to a server, and the apparatus includes: a first transmission unit, a second transmission unit, and a third transmission unit, wherein,

the first sending unit is configured to send a first information instruction to a first thread, where the first information instruction is used to instruct the first thread to, if a data set corresponding to an M-th block is read from a read set database, execute a first queue to generate N first tasks from the data set corresponding to N blocks before the M-th block, where N is a positive integer less than or equal to M, and M is a positive integer;

the second sending unit is configured to, if it is determined that a current task of a second thread is P first tasks that are generated by acquiring, through the first queue, data sets corresponding to P blocks before the N blocks, execute sending a second information instruction to the second thread, where the second information instruction is used to instruct the second thread to perform MVCC verification on the P blocks; the second thread is instructed to write the data sets corresponding to the P blocks into a write set cache respectively, and P write events are generated; synchronizing the P writing events into a second queue to generate P third tasks, wherein P is a positive integer less than or equal to N;

the third sending unit is configured to, if it is determined that the current task of the third thread is Q third tasks in the second queue obtained through the write set database, execute sending a third information instruction to the third thread, where the third information instruction is used to instruct the third thread to execute the Q third tasks and obtain the current blockadd in the write set cache; when the execution of any one third task in the Q third tasks is completed, updating the current BolckID to be the BlockID corresponding to the block of the any one third task, where Q is a positive integer less than or equal to P, the Q third tasks correspond to Q blocks before the P blocks, and each third task corresponds to one block.

A third aspect of embodiments of the present application provides a server, which includes a processor, a communication interface, a memory, and one or more programs, where the processor, the communication interface, and the memory are connected to each other, where the memory is used to store a computer program, and the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method according to the first aspect of embodiments of the present application.

A fourth aspect of embodiments of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program for electronic data exchange, where the computer program makes a computer perform part or all of the steps as described in the first aspect of embodiments of the present application.

A fifth aspect of embodiments of the present application provides a computer program product, wherein the computer program product comprises a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps as described in the first aspect of embodiments of the present application. The computer program product may be a software installation package.

The embodiment of the application has at least the following beneficial effects: a first information instruction can be sent to the first thread, wherein the first information instruction is used for indicating the first thread to execute the generation of N first tasks by the first queue according to the data sets corresponding to the N blocks before the Mth block if the data sets corresponding to the Mth block are read from the read set database; if the current task of the second thread is determined to be P first tasks generated by acquiring a data set corresponding to P blocks before the N blocks through the first queue, executing to send a second information instruction to the second thread, wherein the second information instruction is used for indicating the second thread to carry out MVCC verification on the P blocks; instructing a second thread to write the data sets corresponding to the P blocks into a write set cache respectively and generating P write events; synchronizing P write-in events into a second queue to generate P third tasks, wherein P is a positive integer less than or equal to N; if the current task of the third thread is determined to be Q third tasks in the second queue obtained through the write set database, sending a third information instruction to the third thread, wherein the third information instruction is used for indicating the third thread to execute the Q third tasks and obtaining the current Block ID in the write set cache; when the execution of any one third task in the Q third tasks is finished, updating the current BolckID to be the Block ID corresponding to the block of any one third task, wherein Q is a positive integer less than or equal to P, the Q third tasks correspond to Q blocks before the P blocks, and each third task corresponds to one block. Therefore, the reading and writing of the database can be separated and respectively divided into a reading set database and a writing set database, and a cache concept is introduced; furthermore, when each thread runs, different intervals can be accessed correspondingly, so that multi-thread database access is realized, and the database access pressure is reduced; meanwhile, when the MVCC is verified, the cache can be directly accessed, and a database does not need to be frequently accessed, so that the verification efficiency of the MVCC is improved; in addition, the operation only relates to CPU operation, and is beneficial to improving the performance of the block chain.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1A is a system architecture diagram illustrating data processing according to an embodiment of the present application;

fig. 1B is a schematic flowchart of a data processing method according to an embodiment of the present application;

fig. 2 is a schematic flow chart of a data processing method according to an embodiment of the present application;

fig. 3 is a schematic flowchart of a data processing method according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of a server according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The terms "first," "second," and the like in the description and claims of the present application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

In order to better understand the embodiments of the present application, methods of applying the embodiments of the present application will be described below.

The servers mentioned in the embodiments of the present application may include, but are not limited to, a background server, a component server, a cloud server, a data processing system server, or a data processing software server, which are merely examples, and are not exhaustive, and include, but are not limited to, the above devices.

Referring to fig. 1A, fig. 1A is a schematic diagram of a system architecture for data processing according to an embodiment of the present disclosure.

The embodiment of the application is applied to a server, a blockchain network can be deployed in the server, and the blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism and an encryption algorithm. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product services layer, and an application services layer.

The embodiments of the present application can be specifically applied to a wimax blockchain platform, where the wimax blockchain platform is a blockchain network composed of frames based on S3C, and S3C is a frame system composed of a blockchain solution module, a blockchain kernel module, a blockchain privacy protection module, and a blockchain network management module.

The system architecture may include: a read set database, a write set database, a first queue, a second queue, a write set cache, and a Multi-Version Concurrency Control (MVCC) module.

The MVCC module is used for version verification of transaction information in blocks, the blocks can be basic unit structures in a block chain network, and each block is composed of a block head and a block body; the block head stores structured data, and the block body is generally of a tree structure and used for recording transaction information of the block; and the MVCC module is mainly used for verifying whether the corresponding data version in the transaction information is correct, and if the used reading set version of the current transaction is not equal to the reading set version in the current database, the transaction is considered to be outdated and marked as an error transaction.

In addition, in the embodiment of the present application, the read-write part in the database can be separated, the database can be accessed through multiple threads (2 or more than 2 threads), the write set data in the database can be stored in the write set cache, and the write set cache can be directly accessed when the MVCC version verification is performed, so that the database write cache does not need to be waited, which is beneficial to improving the write-in performance of the database and avoiding data blocking during writing.

The data processing method can correspond to 3 threads, the read set database is used for completing tasks of a first thread, the MVCC module is used for completing tasks of a second thread, the write set database is used for completing tasks of a third thread, the three threads can be simultaneously performed and can synchronously process data in a plurality of blocks, therefore, related operations of the read set and the write set can be understood as IO operations, after the database is subjected to read-write separation, the MVCC verification can be completed by a CPU, and the operations are synchronously performed by dividing the database into 3 threads, so that the IO operations and the CPU operations are separated, and the performance of a block chain network is improved.

Referring to fig. 1B, fig. 1B is a schematic flowchart of a data processing method provided in an embodiment of the present application, and the method is applied to a server, and the method includes the following steps:

101. and sending a first information instruction to a first thread, wherein the first information instruction is used for instructing the first thread to execute a first queue to generate N first tasks by using a data set corresponding to N blocks before an Mth block if the data set corresponding to the Mth block is read from a read set database, wherein N is a positive integer smaller than or equal to M, and M is a positive integer.

Wherein the reading set database may correspond to the reading set database in the system architecture diagram shown in fig. 1A; the database for storing the corresponding information in the block can be divided into a read set database and a write set database, so that different subsequent threads can access the database at different time intervals, frequent access to the same database is relieved, and access pressure of the database is reduced.

The first information instruction can be set by a user or defaulted by a system, and is not limited herein; the first information instruction is used for instructing the first thread to read the data set corresponding to the block from the read set database.

The mth block may be any block for which the first thread needs to perform MVCC verification on the block within a certain time period, and a data set corresponding to the mth block may be read from a read set database, where the data set may include at least one of the following data: the block id, the data version number, the data key value pair and the reference count corresponding to the block id, etc., which are not limited herein, and the data version number may be the database version number corresponding to the block.

The blocks in the block chain network are arranged in a chain manner, so that each block can be numbered, each BolckID can correspond to one block, the bolckids can be arranged from small to large, and the smaller the BolckID is, the priority is given to the processing by the first thread. For example, for the mth block, the BlockID may be M, and the BlockID is used to identify the block.

The first thread may work while the second thread and the third thread may all process data in parallel, the first thread, the second thread and the third thread may be combined into one data processing flow, each thread may process different blocks, but the processing steps may be the same, for example, for 7 blocks, their corresponding blockids are: 1. 2, 3, 4, 5, 6, 7; during a first time period, if the first thread is processing blocks 6 and 7, then the second thread may process blocks 3, 4, and 5 simultaneously or in parallel; while a third thread may process block 1 and block 2 synchronously or in parallel; and the smaller the blockID corresponding to the block is, the higher the priority is for processing; that is, during a time period immediately preceding the first time period, the first thread is processing block 3, block 4, and block 5, the second thread may be processing block 2 synchronously, and the third thread may be processing block 1 synchronously; and so on.

After the first thread reads the data set corresponding to the block, the block may be placed in a first queue, where the first queue may be used to store the block, and the first queue may provide functions required for executing the task, such as returning a result, retrying the task, and the like; for example, after the first queue receives the block sent by the first thread, a result message may be returned to the subsequent thread, that is, the second thread, to inform the second thread of the completion of the storage of the block, so that the second thread can implement subsequent functions, for example, MVCC verification can be performed on the block; each block may generate a first task when placed in the first queue by the first thread, and each block may correspond to one first task.

When the first thread processes the mth block, the first thread and other threads work synchronously, and the processing speed of each thread may be different, and when the first thread processes the mth block, all the blocks in the first queue are the blocks that have been processed by the first thread, so that the first queue stores N blocks before the mth block, the N blocks are obtained by sending the first queue after the first thread completes processing, and N is an integer less than or equal to M.

102. If the current task of a second thread is determined to be P first tasks generated by acquiring data sets corresponding to P blocks before the N blocks through the first queue, executing to send a second information instruction to the second thread, wherein the second information instruction is used for indicating the second thread to carry out MVCC verification on the P blocks; the second thread is instructed to write the data sets corresponding to the P blocks into a write set cache respectively, and P write events are generated; and synchronizing the P write events into a second queue to generate P third tasks, wherein P is a positive integer less than or equal to N.

The Multi-Version Concurrency Control (MVCC) verification is a Concurrency Control method, and is used for verifying whether the data Version of the transaction data included in the block is correct; that is, whether the read set of the transaction in the block is consistent with the version in the current database book (i.e., no change) is checked, if no change is made, it indicates that the modification of the data in the transaction write set is valid, the transaction is marked as valid, and the write set of the transaction is updated to a specific database (e.g., a write set database).

The second information instruction can be set by the user or defaulted by the system, and is not limited herein; after detecting a preset trigger condition, the server may send the second information instruction to the second thread, where the second information instruction is used to instruct the second thread to implement MVCC verification on the block, and the preset trigger condition may be set by the user or default to the system, and is not limited herein; for example, when the current task of the second thread is the first task generated by acquiring the first queue, or when the first thread receives the first information instruction, the second thread may also be triggered to start running to implement the corresponding function.

The second thread may directly acquire the block from the first queue to perform MVCC verification on the block, and write the block into a write set cache after the verification is completed, where the write set cache may be set by a user or default to a system, and is not limited herein; the write set cache can be used for storing information such as data sets corresponding to the blocks, and in addition, when the block verification is successful or fails, a verification result can be generated and written into the write set database to record the verification result aiming at the block.

The second queue can be set by the user or defaulted by the system, and is not limited herein; the second queue may have the same structure as the first queue, and the second queue may be configured to store blocks verified by the second thread and provide the blocks to the write set cache.

The MVCC module shown in fig. 1A may be configured to implement the MVCC verification on P blocks, where each block may be packaged with at least one transaction, each transaction corresponds to transaction information, and when the MVCC verification is implemented, the minimum unit may be the transaction information.

In a possible example, if the reference count corresponding to the block a is not 0, the following steps may be further included: before writing the block A into the write set cache, if a data set of a block B is associated with the data set of the block A, adding 1 to a reference count in the data set of the block B in the write set cache through the block A, wherein the block B is any one block written into the write set cache before the block A.

When MVCC verification is carried out, if transaction information in a currently processed block is associated with transaction information packed in a write set of a previously processed block, the previously processed block can have an influence on the currently processed block; for example, if a transaction message in the current block is subjected to MVCC verification, it may be necessary to use the transaction messages in other previously processed blocks as reference data to verify the transaction status of the target block (e.g., balance information, consumption information, etc.); thus, when the MVCC verification is performed, other blocks associated with the block may affect the MVCC verification process; therefore, before the MVCC verification, the relevance determination may be performed on the current block and the previous blocks.

The data relevance can be judged by reference counting in the data set, and the reference counting is used for judging whether a block with relevant data corresponding to the block exists in the write set cache or not.

In a specific implementation, if the second thread performs MVCC verification in the current processing block or the reference count corresponding to any block in the block processed by the third thread is not 0, it indicates that there is a correlation between the block and the data sets respectively corresponding to the currently processed blocks, and the data sets in the correlated blocks can be directly called in the MVCC verification to help the currently processed block to implement the MVCC verification.

In addition, the block a is any one of P blocks, and if the data set of the block B is found to be associated with the data set of the block a after the MVCC verification process is performed on the block a and before the block a is written into the write set buffer, the reference count of the block B may be incremented by 1 by the block a, which may indicate that there is data association between the block a and the block B, and at the same time, when the MVCC verification is performed in the next process, the block a may know that there is association between the block B and its own data set through the reference count of the block B; the block B may be any one of the blocks before the block a, and in the embodiment of the present application, only two of the blocks are taken as an example for description.

Optionally, the MVCC verification for the P blocks may include the following steps: the data version number corresponding to each transaction message in each block can be determined based on the data set corresponding to each block, the data version number is compared with a preset data version number, if the data version number is the same as the preset data version number, the MVCC verification for the block is determined to be successful, otherwise, the MVCC verification for the block is determined to be failed; the block may be any one of the P blocks described above.

The preset data version number may be set by a user or a default of a system, and is not limited herein. The preset version number may be set by a background worker, for example, may be 100 or 200, and if the MVCC verification is successful for a transaction, it indicates that the transaction is valid, the block corresponding to the transaction may be stored in the local chain, and the corresponding result data that is successfully verified may be generated by the second queue as a third task for notifying that the write set database is successfully verified, and is stored and updated in the write set database.

In one possible example, when performing MVCC verification on the P blocks, the method may further include the following steps: determining that the data version number of the read set database is a first version number when the block A accesses the corresponding data set from the read set database; determining a block E associated with the block A according to the first version number, and adding 1 to the reference count of the block E; performing the MVCC verification on the block A; when the block A is finished with MVCC verification, the reference count of the block E is decreased by 1.

The association between the blocks can be established based on the database version number, the blocks with data association can be placed in the same database in advance, or the database version number can be written into a plurality of blocks with data association.

The first version number may be a data version number corresponding to the read set database corresponding to the block a, and the associated block E corresponding to the first version number may be determined according to the data version number.

For example, the association between the blocks may be established based on the database version number, for example, when the block 100 with the block id of 100 accesses data from the database, the version number of the database is 60; then, sector 100 may be associated with sector 61 corresponding to database version number 61. In the subsequent step, the reference count of the block 61 may be +1, and the reference count of the block 61 may be-1 only after the block 100 completes the MVCC verification, at this time, the association between the block 100 and the block 61 is also released, that is, there is no association between the block 100 and the block 61.

103. If the current task of the third thread is determined to be Q third tasks in the second queue obtained through the write set database, executing to send a third information instruction to the third thread, wherein the third information instruction is used for instructing the third thread to execute the Q third tasks and obtaining the current Block ID in the write set cache; when the execution of any one third task in the Q third tasks is completed, updating the current BolckID to be the BlockID corresponding to the block of the any one third task, where Q is a positive integer less than or equal to P, the Q third tasks correspond to Q blocks before the P blocks, and each third task corresponds to one block.

And the block corresponding to the current BolckID is the Block ID corresponding to the block currently processed by the current third thread.

Wherein, when the second thread is processing the P blocks, the third thread can synchronously process Q blocks before the blockids corresponding to the P blocks, the Q blocks being a batch of blocks processed by the second thread before the P blocks.

The third information instruction can be set by the user or defaulted by the system, and is not limited herein; when the write set database acquires the Q third tasks in the second queue, the server can be triggered to send the third information instruction to a third thread.

And when the third thread completes one block, the current Block ID in the write set cache can be updated to the Block ID corresponding to the block which is completed currently, so that the current block number of the current processing can be identified, the blocks which are processed currently by other threads can be informed, and the common cooperation of the whole thread can be ensured.

When the third task corresponding to any block is completed, that is, after the data set corresponding to any block is written into the write set database, the current BlockID in the write set cache may be updated.

In one possible example, the data set includes: reference counting, the above method may further comprise the steps of: if the reference count corresponding to the block A is not 0, determining that there is a relationship between a data set of a block and a data set of the block A, wherein the block A is any one of the P blocks; if the reference count corresponding to the block A is 0, determining that there is a relationship between a data set of an absent block and a data set of the block A, and deleting the data set corresponding to the block A from the write set cache after updating the current Block ID to be the Block ID of the block A.

Wherein the data set may comprise at least one of: the BlockID, the version number, the data key value pair and the reference count corresponding to the BlockID, and the like, which are not limited herein; the BlockID may be understood as a block number currently in the thread for data processing, and the version number may be a database version number.

In a specific implementation, after the block a is processed by the second thread, the corresponding reference count may change, so that after entering the third thread, the reference count of the block a may be monitored, and if the reference count is 0, it indicates that after the MVCC verification of the second thread is performed, there may be data association between other blocks and the data set thereof, and the block a may be retained; on the contrary, if the reference count corresponding to the block a is 0, it indicates that there is no association between the block and the data set of the block a, and since, in general, each block may contain thousands of transactions, the memory space required for the transaction may be large, in order to save the memory space of the cache, the data set corresponding to the block a may be deleted from the write set cache after the current block id is updated to the block id of the block a.

Optionally, the method can further comprise the following steps: if a cache recovery mechanism is triggered when the tasks of the first thread, the second thread or the third thread are executed, determining a plurality of blocks existing in the write set cache and a plurality of reference counts corresponding to the plurality of blocks, wherein each block corresponds to one reference count; and if the blockID corresponding to the block C is smaller than the current BolckID and the reference count corresponding to the block C is 0, deleting the block C from the write set cache, wherein the block C is any one of the plurality of blocks.

In order to save the memory space in the cache, the server may further include a cache recovery mechanism, which may be set by default of the system or by the user, and is not limited herein; when any thread triggers the cache reclamation mechanism, all the blocks in the write set cache and the reference count corresponding to each block can be determined, and the data set corresponding to the blocks stored in the write set cache can be processed through the reference counts.

In a specific implementation, if the BlockID corresponding to the block C is smaller than the current BlockID, it indicates that the block C has completed processing of the three threads, and if the reference count of the block C is 0, it indicates that there is no block associated with the data set of the block C, the block C may be deleted from the write set cache, so as to save the memory space of the write set cache, and improve the efficiency of subsequently acquiring data from the write set cache.

In one possible example, the following steps may also be included: if the data set of the block C is associated with the data set corresponding to the block D, after deleting the block C, subtracting 1 from the reference count corresponding to the block D, where the block C and the block D are any two blocks in the write set cache.

The addition and subtraction of the reference count is actually a block-to-block operation, for example, if a block C finds a relationship between a block D and a data set of the block C during MVCC verification, the block C may enable the reference count +1 corresponding to the block D, so that the block C may obtain reference counts corresponding to other blocks during subsequent processes, if the reference counts are not 0, there is a relationship therebetween, if the reference counts are 0, there is no relationship therebetween, and the reference count of the other blocks obtained by each block is relative to the block; for another example, if the reference count of a block is 5, then there may be 5 blocks associated with it.

In one implementation, if the reference count of the block C is 0 but the block D is associated with the data set of the block C, that is, the block C finds that the block D has data associated with it, but the data set corresponding to the block D is already used by the block C when performing MVCC authentication, the reference count corresponding to the block D may be decremented by 1 after deleting the block C.

It can be seen that the data processing method described in this embodiment of the present application is applied to a server, and may send a first information instruction to a first thread, where the first information instruction is used to instruct the first thread to, if a data set corresponding to an mth block is read from a read set database, execute to generate N first tasks through a first queue with the data set corresponding to N blocks before the mth block, where N is a positive integer less than or equal to M, and M is a positive integer; if the current task of the second thread is determined to be P first tasks generated by acquiring a data set corresponding to P blocks before the N blocks through the first queue, executing to send a second information instruction to the second thread, wherein the second information instruction is used for indicating the second thread to carry out MVCC verification on the P blocks; instructing a second thread to write the data sets corresponding to the P blocks into a write set cache respectively and generating P write events; synchronizing P write-in events into a second queue to generate P third tasks, wherein P is a positive integer less than or equal to N; if the current task of the third thread is determined to be Q third tasks in the second queue obtained through the write set database, sending a third information instruction to the third thread, wherein the third information instruction is used for indicating the third thread to execute the Q third tasks and obtaining the current Block ID in the write set cache; when the execution of any one third task in the Q third tasks is finished, updating the current BolckID to be the Block ID corresponding to the block of any one third task, wherein Q is a positive integer less than or equal to P, the Q third tasks correspond to Q blocks before the P blocks, and each third task corresponds to one block. Therefore, the reading and writing of the database can be separated and respectively divided into a reading set database and a writing set database, and a cache concept is introduced; furthermore, when each thread runs, different intervals can be accessed correspondingly, so that multi-thread database access is realized, and the database access pressure is reduced; meanwhile, when the MVCC is verified, the cache can be directly accessed, and a database does not need to be frequently accessed, so that the verification efficiency of the MVCC is improved; in addition, the operation only relates to CPU operation, and is beneficial to improving the performance of the block chain.

In accordance with the above, please refer to fig. 2, fig. 2 is a flowchart illustrating a data processing method disclosed in an embodiment of the present application, which is applied to a server, and as shown in the diagram, the data processing method may include the read set database, the write set cache, the first queue, the second queue, the MVCC module, and the write set database shown in fig. 1A.

Wherein, the read set database is processing block 66 with BolckID 66, and similarly, the write set cache is processing block 57, block 58, block 59, block 60 and block 61 with BolckID 57, 58, 59, 60 and 61 respectively; block 63, block 64, and block 65, having bolckids 63, 64, and 65, respectively, are being processed by the first queue; block 59, block 60, and block 61, with bolckids 59, 61, and 61, respectively, are being processed by the second queue; block 62 with BolckID 62 being processed by MVCC module; the write-set database is processing block 58 with BolckID 58.

Wherein, the first thread can read the data set corresponding to the block 66 from the read set database, and then execute the first queue to generate 3 first tasks with the data sets corresponding to the blocks 63, 64, and 65, respectively; meanwhile, the second thread may perform MVCC verification on tile 62; at this time, the data sets corresponding to the block 57, the block 58, the block 59, the block 60 and the block 61 are being written into the write set buffer, and 5 write events are generated, and the 5 write events are synchronized into the second queue at the same time, and 5 third tasks are generated to inform the second queue of the block being written into at this time and to inform the second queue of the data sets corresponding to the blocks which need to be written into the write set database at this time; meanwhile, in a third thread, a block 59, a block 60 and a block 61 of which the verification of the MVCC module is completed can be placed in the second queue; at this time, if the third task corresponding to the block 58 is completed, the writing of the data set corresponding to the block 58 into the write set database may be performed, and if it is known that the block processed last is the block 57, the BolckID of the block 58 is updated to be the BolckID (57), which is 58. Thus, the reading and writing of the database are separated and respectively divided into a reading set database and a writing set database, and a cache concept is introduced; furthermore, when each thread runs, each thread can correspondingly access different intervals, so that multi-thread database access can be realized, and the reduction of the access pressure of the database is facilitated; meanwhile, when the MVCC is verified, the cache can be directly accessed to obtain the data set corresponding to the block, a database does not need to be frequently accessed, and the verification efficiency of the MVCC is improved; in addition, the operation only relates to CPU operation, and is beneficial to improving the performance of the block chain.

In accordance with the above, please refer to fig. 3, fig. 3 is a flowchart illustrating a data processing method disclosed in an embodiment of the present application, applied to a server, where the data processing method may include the following steps:

301. and sending a first information instruction to a first thread, wherein the first information instruction is used for instructing the first thread to execute a first queue to generate N first tasks by using a data set corresponding to N blocks before an Mth block if the data set corresponding to the Mth block is read from a read set database.

302. If the current task of a second thread is determined to be P first tasks generated by acquiring data sets corresponding to P blocks before the N blocks through the first queue, executing to send a second information instruction to the second thread, wherein the second information instruction is used for indicating the second thread to carry out MVCC verification on the P blocks; the second thread is instructed to write the data sets corresponding to the P blocks into a write set cache respectively, and P write events are generated; and synchronizing the P write events into a second queue to generate P third tasks, wherein P is a positive integer less than or equal to N.

303. If the current task of the third thread is determined to be Q third tasks in the second queue obtained through the write set database, executing to send a third information instruction to the third thread, wherein the third information instruction is used for instructing the third thread to execute the Q third tasks and obtaining the current Block ID in the write set cache; when the execution of any one third task in the Q third tasks is completed, updating the current BolckID to be the BlockID corresponding to the block of the any one third task, where Q is a positive integer less than or equal to P, the Q third tasks correspond to Q blocks before the P blocks, and each third task corresponds to one block.

304. If a cache reclamation mechanism is triggered when the tasks of the first thread, the second thread or the third thread are executed, determining a plurality of blocks existing in the write set cache and a plurality of reference counts corresponding to the plurality of blocks, wherein each block corresponds to one reference count.

305. And if the blockID corresponding to the block C is smaller than the current BolckID and the reference count corresponding to the block C is 0, deleting the block C from the write set cache, wherein the block C is any one of the plurality of blocks.

306. If the data set of the block C is associated with the data set corresponding to the block D, after deleting the block C, subtracting 1 from the reference count corresponding to the block D, where the block C and the block D are any two blocks in the write set cache.

The detailed description of the above steps 301-306 can refer to the corresponding description of the data processing method described in fig. 1B, and will not be described herein again.

It can be seen that the data processing method described in the embodiment of the present application is applied to a server, and can send a first information instruction to a first thread, where the first information instruction is used to instruct the first thread to, if a data set corresponding to an mth block is read from a read set database, execute a first queue to generate N first tasks from the data sets corresponding to N blocks before the mth block, where N is a positive integer less than or equal to M, and M is a positive integer; if the current task of the second thread is determined to be P first tasks generated by acquiring a data set corresponding to P blocks before the N blocks through the first queue, executing to send a second information instruction to the second thread, wherein the second information instruction is used for indicating the second thread to carry out MVCC verification on the P blocks; instructing a second thread to write the data sets corresponding to the P blocks into a write set cache respectively and generating P write events; synchronizing P write-in events into a second queue to generate P third tasks, wherein P is a positive integer less than or equal to N; if the current task of the third thread is determined to be Q third tasks in the second queue obtained through the write set database, sending a third information instruction to the third thread, wherein the third information instruction is used for indicating the third thread to execute the Q third tasks and obtaining the current Block ID in the write set cache; when the execution of any one third task in the Q third tasks is finished, updating the current BolckID to be the Block ID corresponding to the block of any one third task, wherein Q is a positive integer less than or equal to P, the Q third tasks correspond to Q blocks before the P blocks, and each third task corresponds to one block; if a cache recovery mechanism is triggered when a task of a first thread, a second thread or a third thread is executed, determining a plurality of blocks existing in a write set cache and a plurality of reference counts corresponding to the plurality of blocks, wherein each block corresponds to one reference count; if the blockID corresponding to the block C is smaller than the current BolckID and the reference count corresponding to the block C is 0, deleting the block C from the write set cache, wherein the block C is any one of the blocks; if the data set of block C is associated with the data set corresponding to block D, the execution decrements the reference count corresponding to block D by 1 after deleting block C, and block C and block D are any two blocks in the write set cache. Therefore, the reading and writing of the database can be separated and respectively divided into a reading set database and a writing set database, and a cache concept is introduced; furthermore, when each thread runs, different intervals can be accessed correspondingly, so that multi-thread database access is realized, and the database access pressure is reduced; meanwhile, when the MVCC is verified, the cache can be directly accessed, and a database does not need to be frequently accessed, so that the verification efficiency of the MVCC is improved; in addition, the operation only relates to CPU operation, which is beneficial to improving the performance of the block chain; furthermore, a cache recovery mechanism can be set, and the block in the write set cache can be processed no matter which thread triggers the mechanism, so that the memory space written in the cache can be saved.

In accordance with the above, please refer to fig. 4, fig. 4 is a schematic structural diagram of a server according to an embodiment of the present application, and as shown in fig. 4, the server includes a processor, a communication interface, a memory and one or more programs, where the processor, the communication interface and the memory are connected to each other, the memory is used for storing a computer program, the computer program includes program instructions, the processor is configured to call the program instructions, and the one or more program programs include instructions for performing the following steps:

It can be seen that, in the server described in this embodiment of the present application, a first information instruction may be sent to a first thread, where the first information instruction is used to instruct the first thread to, if a data set corresponding to an mth block is read from a read set database, execute to generate, by a first queue, N first tasks with the data set corresponding to N blocks before the mth block, where N is a positive integer less than or equal to M, and M is a positive integer; if the current task of the second thread is determined to be P first tasks generated by acquiring a data set corresponding to P blocks before the N blocks through the first queue, executing to send a second information instruction to the second thread, wherein the second information instruction is used for indicating the second thread to carry out MVCC verification on the P blocks; instructing a second thread to write the data sets corresponding to the P blocks into a write set cache respectively and generating P write events; synchronizing P write-in events into a second queue to generate P third tasks, wherein P is a positive integer less than or equal to N; if the current task of the third thread is determined to be Q third tasks in the second queue obtained through the write set database, sending a third information instruction to the third thread, wherein the third information instruction is used for indicating the third thread to execute the Q third tasks and obtaining the current Block ID in the write set cache; when the execution of any one third task in the Q third tasks is finished, updating the current BolckID to be the Block ID corresponding to the block of any one third task, wherein Q is a positive integer less than or equal to P, the Q third tasks correspond to Q blocks before the P blocks, and each third task corresponds to one block. Therefore, the reading and writing of the database can be separated and respectively divided into a reading set database and a writing set database, and a cache concept is introduced; furthermore, when each thread runs, different intervals can be accessed correspondingly, so that multi-thread database access is realized, and the database access pressure is reduced; meanwhile, when the MVCC is verified, the cache can be directly accessed, and a database does not need to be frequently accessed, so that the verification efficiency of the MVCC is improved; in addition, the operation only relates to CPU operation, and is beneficial to improving the performance of the block chain.

In one possible example, the data set includes: reference counting, the program for executing the instructions of:

if the reference count corresponding to the block A is not 0, determining that there is a relationship between a data set of a block and a data set of the block A, wherein the block A is any one of the P blocks;

if the reference count corresponding to the block A is 0, determining that there is a relationship between a data set of an absent block and a data set of the block A, and deleting the data set corresponding to the block A from the write set cache after updating the current Block ID to be the Block ID of the block A.

In one possible example, if the reference count corresponding to block a is not 0, the program is configured to execute the following steps:

before writing the block A into the write set cache, if a data set of a block B is associated with the data set of the block A, adding 1 to a reference count in the data set of the block B in the write set cache through the block A, wherein the block B is any one block written into the write set cache before the block A.

In one possible example, the program is for instructions to perform the steps of:

if a cache recovery mechanism is triggered when the tasks of the first thread, the second thread or the third thread are executed, determining a plurality of blocks existing in the write set cache and a plurality of reference counts corresponding to the plurality of blocks, wherein each block corresponds to one reference count;

and if the blockID corresponding to the block C is smaller than the current BolckID and the reference count corresponding to the block C is 0, deleting the block C from the write set cache, wherein the block C is any one of the plurality of blocks.

In one possible example, the program is further for instructions to perform the steps of:

if the data set of the block C is associated with the data set corresponding to the block D, after deleting the block C, subtracting 1 from the reference count corresponding to the block D, where the block C and the block D are any two blocks in the write set cache.

In one possible example, the data set further comprises: a data version number, the program being for instructions to perform the following steps in the MVCC validation of the P blocks:

determining that the data version number of the read set database is a first version number when the block A accesses the corresponding data set from the read set database;

determining a block E associated with the block A according to the first version number, and adding 1 to the reference count of the block E;

performing the MVCC verification on the block A;

when the block A is finished performing MVCC verification, determining to execute to reduce the reference count of the block E by 1.

The above description has introduced the solution of the embodiment of the present application mainly from the perspective of the method-side implementation process. It is understood that the server includes hardware structures and/or software modules for performing the respective functions in order to implement the above-described functions. Those of skill in the art will readily appreciate that the present application is capable of hardware or a combination of hardware and computer software implementing the various illustrative elements and algorithm steps described in connection with the embodiments provided herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiment of the present application, the server may be divided into the functional units according to the above method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. It should be noted that the division of the unit in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation.

In accordance with the above, please refer to fig. 5, fig. 5 is a schematic structural diagram of a data processing apparatus disclosed in the embodiment of the present application, applied to a server, the apparatus including: a first sending unit 501, a second sending unit 502 and a third sending unit 503, wherein,

the first sending unit 501 is configured to send a first information instruction to a first thread, where the first information instruction is used to instruct the first thread to, if a data set corresponding to an M-th block is read from a read set database, execute a first queue to generate N first tasks from the data set corresponding to N blocks before the M-th block, where N is a positive integer less than or equal to M, and M is a positive integer;

the second sending unit 502 is configured to, if it is determined that the current task of the second thread is P first tasks that are generated by acquiring, through the first queue, data sets corresponding to P blocks before the N blocks, execute sending a second information instruction to the second thread, where the second information instruction is used to instruct the second thread to perform MVCC verification on the P blocks; the second thread is instructed to write the data sets corresponding to the P blocks into a write set cache respectively, and P write events are generated; synchronizing the P writing events into a second queue to generate P third tasks, wherein P is a positive integer less than or equal to N;

the third sending unit 503 is configured to, if it is determined that the current task of the third thread is Q third tasks obtained from the write set database in the second queue, execute sending a third information instruction to the third thread, where the third information instruction is used to instruct the third thread to execute the Q third tasks and obtain the current blockacid in the write set cache; when the execution of any one third task in the Q third tasks is completed, updating the current BolckID to be the BlockID corresponding to the block of the any one third task, where Q is a positive integer less than or equal to P, the Q third tasks correspond to Q blocks before the P blocks, and each third task corresponds to one block.

It can be seen that the data processing apparatus described in this embodiment of the present application, which is applied to a server, may send a first information instruction to a first thread, where the first information instruction is used to instruct the first thread to, if a data set corresponding to an mth block is read from a read set database, execute to generate, by a first queue, N first tasks from data sets corresponding to N blocks before the mth block, where N is a positive integer less than or equal to M, and M is a positive integer; if the current task of the second thread is determined to be P first tasks generated by acquiring a data set corresponding to P blocks before the N blocks through the first queue, executing to send a second information instruction to the second thread, wherein the second information instruction is used for indicating the second thread to carry out MVCC verification on the P blocks; instructing a second thread to write the data sets corresponding to the P blocks into a write set cache respectively and generating P write events; synchronizing P write-in events into a second queue to generate P third tasks, wherein P is a positive integer less than or equal to N; if the current task of the third thread is determined to be Q third tasks in the second queue obtained through the write set database, sending a third information instruction to the third thread, wherein the third information instruction is used for indicating the third thread to execute the Q third tasks and obtaining the current Block ID in the write set cache; when the execution of any one third task in the Q third tasks is finished, updating the current BolckID to be the Block ID corresponding to the block of any one third task, wherein Q is a positive integer less than or equal to P, the Q third tasks correspond to Q blocks before the P blocks, and each third task corresponds to one block. Therefore, the reading and writing of the database can be separated and respectively divided into a reading set database and a writing set database, and a cache concept is introduced; furthermore, when each thread runs, different intervals can be accessed correspondingly, so that multi-thread database access is realized, and the database access pressure is reduced; meanwhile, when the MVCC is verified, the cache can be directly accessed, and a database does not need to be frequently accessed, so that the verification efficiency of the MVCC is improved; in addition, the operation only relates to CPU operation, and is beneficial to improving the performance of the block chain.

In one possible example, the data set further comprises: a data version number, in the MVCC verification for the P blocks, the second transmitting unit 502 is specifically configured to:

performing the MVCC verification on the block A;

Embodiments of the present application also provide a computer-readable storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any one of the data processing methods as described in the above method embodiments.

Embodiments of the present application also provide a computer program product comprising a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the data processing methods as set forth in the above method embodiments.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implementing, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of some interfaces, devices or units, and may be an electric or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may be implemented in the form of a software program module.

The integrated units, if implemented in the form of software program modules and sold or used as stand-alone products, may be stored in a computer readable memory. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in the form of a software product stored in a memory, and including several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned memory comprises: various media capable of storing program codes, such as a usb disk, a read-only memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and the like.

Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable memory, which may include: flash disk, ROM, RAM, magnetic or optical disk, and the like.

The foregoing detailed description of the embodiments of the present application has been presented to illustrate the principles and implementations of the present application, and the above description of the embodiments is only provided to help understand the method and the core concept of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A data processing method is applied to a server and comprises the following steps:

2. The method of claim 1, wherein the data set comprises: reference counting, the method further comprising:

3. The method according to claim 1 or 2, wherein if the reference count corresponding to the block a is not 0, the method further comprises:

4. The method of claim 1, further comprising:

5. The method of claim 4, further comprising:

6. The method of claim 1 or 2, wherein the data set further comprises: the MVCC verification of the P blocks comprises the following steps:

performing the MVCC verification on the block A;

7. A data processing apparatus, applied to a server, the apparatus comprising: a first transmission unit, a second transmission unit, and a third transmission unit, wherein,

8. The apparatus of claim 7, wherein the data set further comprises: a data version number, and in the MVCC verification for the P blocks, the second transmitting unit is specifically configured to:

performing the MVCC verification on the block A;

9. A server, comprising a processor, a communication interface, a memory, and one or more programs, the processor, the communication interface, and the memory being interconnected, wherein the memory is configured to store a computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method of any of claims 1-6.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1-6.