CN111245897A

CN111245897A - Data processing method, device, system, storage medium and processor

Info

Publication number: CN111245897A
Application number: CN201911399324.6A
Authority: CN
Inventors: 尚德重; 焦臻桢
Original assignee: Shenzhen Material Margin Technology Co ltd
Current assignee: Shenzhen Material Margin Technology Co ltd
Priority date: 2019-12-30
Filing date: 2019-12-30
Publication date: 2020-06-05
Anticipated expiration: 2039-12-30
Also published as: CN111245897B

Abstract

The application discloses a data processing method, a device, a system, a storage medium and a processor. The method comprises the following steps: sending the write request message to a message queue through a service node; judging whether the storage node stores the content of the request for writing, determining a write request confirmation message based on the judgment result, and sending the write request confirmation message to a message queue; under the condition that the write request message and the write request confirmation message meet the preset node condition, the write request message and the write request confirmation message are packaged into a block message through a packaging node, and the block message is sent to a message queue; and storing the block message to a plurality of storage nodes under the condition that the block message meets the preset node condition. By the method and the device, the problem that in the related art, when data are read and written in the data storage system, the safety of data reading and writing and the data reading and writing efficiency are difficult to be considered is solved.

Description

Data processing method, device, system, storage medium and processor

Technical Field

The present application relates to the field of data storage technologies, and in particular, to a data processing method, apparatus, system, storage medium, and processor.

Background

Database storage technology has wide application in various fields of computers.

When data storage is performed in the related art, data is usually stored in a server owned by a certain party, and the storage method has the following defects: the owner of the server has the tampering right and the deletion right of all stored data, and can delete the data in any mode at any time, so that when the data needs to be used among multiple parties, the data is difficult to ensure to be credible; since data is stored in a server on one side, there is a possibility that data is lost due to unexpected causes such as damage to the server; since the data is stored on a server of a certain party, the owner of the server owns the data, the privacy of the owner of the original data is at risk of being violated, which is a great risk especially for cloud computing.

In order to overcome the above problems, the related art may adopt a block chain to store data, but the storage method has the following problems: the block chain cannot store large data and multimedia data due to the chain structure of the block chain, otherwise the block size of the block chain is difficult to control, and the whole chain expandability of the block chain is influenced; due to the chain structure of the block chain, the data access efficiency of the block chain is poor, and particularly, due to the existence of the packing process, the block chain data storage is difficult to be used for high-speed writing and fast reading; in the practical application process, due to the fact that the data volume is too large, the data is stored under the line generally, only the check of the data is stored in a block chain, the block in the block chain has uncertainty, the block chain cannot guarantee that the data cannot be lost, and the risk of losing the data under the line exists.

Aiming at the problem that the safety of data reading and writing and the efficiency of data reading and writing are difficult to be considered when data reading and writing are carried out in a data storage system in the related technology, an effective solution is not provided at present.

Disclosure of Invention

The application provides a data processing method, a device, a system, a storage medium and a processor, which are used for solving the problem that the data reading and writing safety and the data reading and writing efficiency are difficult to be considered when data reading and writing are carried out in a data storage system in the related technology.

According to one aspect of the present application, a data processing method is provided. The method comprises the following steps: sending a write request message to a message queue through a service node, wherein the write request message represents that a target write node requests to write a target key value and a target key value item into a storage node; judging whether the storage node stores the content of the request for writing, determining a write request confirmation message based on the judgment result, and sending the write request confirmation message to a message queue; under the condition that the write request message and the write request confirmation message meet the preset node condition, the write request message and the write request confirmation message are packaged into a block message through a packaging node, and the block message is sent to a message queue; and under the condition that the block message meets the preset node condition, storing the block message to a plurality of storage nodes, wherein the monitoring node monitors and judges whether the message in the message queue meets the preset node condition.

Optionally, the method further comprises: sending a read request message to a plurality of storage nodes through a service node, wherein the read request message is used for requesting to read a target key value item corresponding to a written target key value; acquiring a plurality of target key value items obtained by a plurality of storage nodes based on target key value query; and comparing the target key value items, and determining a credible target key value item according to a comparison result.

Optionally, the step of encapsulating, by the storage node, the target key value item obtained by the query as a read request response message, comparing the plurality of target key value items, and determining a trusted target key value item includes: and under the condition that the service node verifies that the signature of the read request response message is correct, determining a determined credible target key value item from the target key value items in the plurality of read request response messages according to a simple majority principle.

Optionally, determining whether the storage node has stored the content requested to be written to, and determining the write request acknowledgement message based on the determination result includes: under the condition that the public key of the target write-in node contained in the write request message is a legal node public key and the signature in the write request message is verified to be correct, judging whether a target key value requesting for writing exists in the storage node; under the condition that the target key value exists, determining a first write request confirmation message, wherein the write state of the first write request confirmation message is as follows: the target key value is occupied; determining a second write request confirmation message under the condition that the target key value does not exist, wherein the write state of the second write request confirmation message is as follows: the writing was successful.

Optionally, the monitoring and determining, by the supervisory node, whether the message in the message queue meets the preset node condition includes: judging whether a target write node in the write request message is a registered node or not, and disconnecting the node from the message queue under the condition that the target write node is not the registered node; judging whether a storage node in the write request confirmation message is a registered node or not, and disconnecting the node from the message queue under the condition that the storage node is not the registered node; judging whether a packing node in the block message is a registered node or not, and disconnecting the node from the message queue under the condition that the packing node in the block message is not the registered node; and/or judging whether the frequency of the write request message sent by the target write-in node exceeds a first preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency exceeds the first preset frequency; judging whether the frequency of the write request confirmation message sent by the storage node exceeds a second preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency of the write request confirmation message exceeds the second preset frequency; and judging whether the frequency of the block message sent by the packaging node exceeds a third preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency exceeds the third preset frequency.

Optionally, the method further comprises: after the service node sends the write request message to the message queue, determining to enter a state that the write request is submitted; and/or after the service node monitors a write request confirmation message in the message queue, determining to enter a state that the write request is confirmed; and/or after the service node monitors the block message containing the write request confirmation message in the message queue, determining that the entering write request is in a finally confirmed state.

Optionally, before the writing request message and the writing request acknowledgement message are packaged into the chunk message by the packaging node, the method further comprises: sending the registration information of the packaging node to a message queue; updating cluster configuration information through a supervision node under the condition that the registration information is verified to be valid, and sending the cluster configuration information to a message queue, wherein the cluster configuration information at least comprises a packaging sequence and a packaging time interval; and under the condition that the public key contained in the cluster configuration information belongs to the supervision node and the signature verification of the cluster configuration information passes, the step of packaging the write request message and the write request confirmation message into the block message by the packaging node is executed.

Optionally, before the writing request message and the writing request acknowledgement message are packaged into the chunk message by the packaging node, the method further comprises: judging whether a target write node corresponding to the write request message and/or a storage node corresponding to the write request confirmation message are valid nodes or not; verifying whether the signature in the write request message is correct and/or verifying whether the signature in the write request confirmation message is correct under the condition that the target write node and/or the storage node are valid nodes; the write request message is added to the message pool of the packing node in case the signature in the write request message is correct and/or the write request acknowledge message is added to the message pool of the packing node in case the signature in the write request acknowledge message is correct.

Optionally, the packaging, by the packaging node, the write request message and the write request acknowledge message into the chunk message comprises: determining a time slot corresponding to each packing node in a preset time period based on the packing sequence and the packing time interval; and each packing node selects a preset number of messages from the message pool in the corresponding time slot to pack to obtain a plurality of block messages.

Optionally, the method further comprises: determining that the packaging of the packaging node fails under the condition that the packaging of the packaging node is not completed in the corresponding time slot; and under the condition that the packaging failure times of the packaging nodes exceed the preset times, the packaging nodes are removed through the supervision nodes, and the cluster configuration information is updated.

Optionally, the method further comprises: after monitoring the target block message in the message queue, the target packaging node verifies the validity of the target block message, and analyzes the target block message under the valid condition to obtain a target write request message and a target write request confirmation message; detecting whether a target write request message and a target write request confirmation message exist in a message pool of a target packaging node; if so, the target write request message and the target write request acknowledge message are removed from the message pool of the target packaging node.

According to another aspect of the present application, a data processing system is provided. The system comprises: the service layer comprises a plurality of service nodes and is used for initiating a write request and a read request, wherein the write request is used for requesting to write a target transaction into the storage node, and the read request is used for reading the target transaction from the storage node; the storage layer comprises a plurality of packing nodes and is used for packing a write request message and a write request confirmation message into a block message, wherein the write request message represents that the write node requests to write a target transaction into the storage node, and the write request confirmation message represents whether the target transaction is stored in the storage node or not; a storage node for storing the block message; and the supervision node is used for supervising the admission of the node in the storage layer and the exit of the node, and supervising the condition of writing data into the storage node and the condition of reading data from the storage node.

Optionally, the storage layer further comprises: the message queue is used for caching association messages generated in a data writing process, wherein the association messages at least comprise: a write request message, a write request acknowledge message, and a block message; and the communication network is used for providing communication for the service node and the storage node when reading data.

According to another aspect of the present application, a data processing apparatus is provided. The device includes: the write request message represents a target write node to request to write a target key value and a target key value item into a storage node; the judging unit is used for judging whether the storage node stores the content of the write request, determining a write request confirmation message based on the judging result and sending the write request confirmation message to the message queue; the device comprises a packaging unit, a message queue and a message sending unit, wherein the packaging unit is used for packaging a write request message and a write request confirmation message into a block message through a packaging node under the condition that the write request message and the write request confirmation message meet a preset node condition, and sending the block message to the message queue; and the storage unit is used for storing the block message to a plurality of storage nodes under the condition that the block message meets the preset node condition, wherein the monitoring node monitors and judges whether the message in the message queue meets the preset node condition.

In order to achieve the above object, according to another aspect of the present application, there is provided a storage medium including a stored program, wherein the program executes any one of the data processing methods described above.

In order to achieve the above object, according to another aspect of the present application, there is provided a processor for executing a program, wherein the program executes to perform any one of the data processing methods described above.

Through the application, the following steps are adopted: sending a write request message to a message queue through a service node, wherein the write request message represents that a target write node requests to write a target key value and a target key value item into a storage node; judging whether the storage node stores the content of the request for writing, determining a write request confirmation message based on the judgment result, and sending the write request confirmation message to a message queue; under the condition that the write request message and the write request confirmation message meet the preset node condition, the write request message and the write request confirmation message are packaged into a block message through a packaging node, and the block message is sent to a message queue; under the condition that the block message meets the preset node condition, the block message is stored to a plurality of storage nodes, wherein the monitoring node monitors and judges whether the message in the message queue meets the preset node condition, so that the problem that the safety of data reading and writing and the efficiency of data reading and writing are difficult to be considered when the data reading and writing are carried out in a data storage system in the related technology is solved. By storing the block message in the storage node and adopting the supervision node for supervision in the storage, the effects of improving the safety of data reading and writing and the efficiency of reading and writing are further achieved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:

FIG. 1 is a flow chart of a data processing method provided according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a data processing system provided in accordance with an embodiment of the present application; and

fig. 3 is a schematic diagram of a data processing apparatus provided according to an embodiment of the present application.

Detailed Description

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

According to an embodiment of the present application, a data processing method is provided.

Fig. 1 is a flowchart of a data processing method according to an embodiment of the present application. As shown in fig. 1, the method comprises the steps of:

step S101, a write request message is sent to a message queue through a service node, wherein the write request message represents a target write node to request to write a target key value and a target key value item into a storage node.

It should be noted that the write request message represents that the target write node requests to write the target key value and the target key value item into the storage node, and the write request message includes the target key value (K value) to be written, the target key value item (V value), the id of the write node, the public key of the write node, and the signature of the write node on the message.

Step S102, judging whether the storage node stores the content requested to be written, determining a write request confirmation message based on the judgment result, and sending the write request confirmation message to a message queue.

Specifically, in the case where the storage node has stored the content requested to be written, the illustrated storage node has been occupied by the same data to obtain the corresponding request confirmation message, and in the case where the storage node has not stored the content requested to be written, the illustrated storage node may store the data to obtain the corresponding request confirmation message.

Optionally, in the data processing method provided in the embodiment of the present application, determining whether the storage node has stored content requested to be written, and determining the write request acknowledgement message based on the determination result includes: under the condition that the public key of the target write-in node contained in the write request message is a legal node public key and the signature in the write request message is verified to be correct, judging whether a target key value requesting for writing exists in the storage node; under the condition that the target key value exists, determining a first write request confirmation message, wherein the write state of the first write request confirmation message is as follows: the target key value is occupied; determining a second write request confirmation message under the condition that the target key value does not exist, wherein the write state of the second write request confirmation message is as follows: the writing was successful.

Specifically, after receiving the write request message, the storage node compares the public key thereof with a locally stored legal node public key to judge whether the node is a valid node, and if not, the message is not processed; if the node is a valid node, the public key is used for verifying whether the signature in the write request message is correct, and if the signature is incorrect, the message is not processed.

If the signature is correct, judging whether a K value in the write request message exists locally, if so, sending a write request confirmation message to the message queue, wherein the write request confirmation message comprises the write request message and the public key of the storage node, and the signature and the write state of the storage node to the write request confirmation message, namely that the K is occupied.

And if the K value in the write request message does not exist in the local storage, sending a write request confirmation message to the message queue, wherein the write request confirmation message comprises the write request message, the public key of the storage node, the signature of the storage node on the write request confirmation message and the write state, namely 'write success'.

And step S103, packing the write request message and the write request confirmation message into a block message through a packing node under the condition that the write request message and the write request confirmation message meet the preset node condition, and sending the block message to a message queue.

Specifically, a packing node acquires a write request message and a write request confirmation message by monitoring a message queue, after determining that the write request message and the write request confirmation message meet a preset node condition set by a supervisory node, the write request message and the write request confirmation message are placed into a message pool of the packing node, the packing node packs the messages according to a sequence specified by the supervisory node within a specified time, specifically, the messages are selected from the message pool and added into a block, the block is created, and the created block is published.

Step S104, storing the block message to a plurality of storage nodes under the condition that the block message meets the preset node condition, wherein the monitoring node monitors and judges whether the message in the message queue meets the preset node condition.

Specifically, the storage node acquires the block message by monitoring the message queue, signs the message verification message after determining that the block message meets the preset node condition set by the supervisory node, and locally stores the block in the block message under the condition that the message signature verification is passed.

According to the data processing method provided by the embodiment of the application, the writing request message is sent to the message queue through the service node, wherein the writing request message represents that the target writing node requests to write the target key value and the target key value item into the storage node; judging whether the storage node stores the content of the request for writing, determining a write request confirmation message based on the judgment result, and sending the write request confirmation message to a message queue; under the condition that the write request message and the write request confirmation message meet the preset node condition, the write request message and the write request confirmation message are packaged into a block message through a packaging node, and the block message is sent to a message queue; under the condition that the block message meets the preset node condition, the block message is stored to a plurality of storage nodes, wherein the monitoring node monitors and judges whether the message in the message queue meets the preset node condition, so that the problem that the safety of data reading and writing and the efficiency of data reading and writing are difficult to be considered when the data reading and writing are carried out in a data storage system in the related technology is solved. By storing the block message in the storage node and adopting the supervision node for supervision in the storage, the effects of improving the safety of data reading and writing and the efficiency of reading and writing are further achieved.

After writing data, data in the storage node may be read, and optionally, in the data processing method provided in the embodiment of the present application, the method further includes: sending a read request message to a plurality of storage nodes through a service node, wherein the read request message is used for requesting to read a target key value item corresponding to a written target key value; acquiring a plurality of target key value items obtained by a plurality of storage nodes based on target key value query; and comparing the target key value items, and determining a credible target key value item according to a comparison result.

It should be noted that, the service node directly sends the read request message to each storage node through the peer-to-peer network, and each storage node performs data query based on the read request message, where the read request message includes the value of K to be read, the id of the write node, the public key of the write node, and the signature of the write node on the message.

Optionally, in order to read trusted data, in the data processing method provided in this embodiment of the present application, the storage node encapsulates a target key value item obtained by query as a read request response message, compares a plurality of target key value items, and determines the trusted target key value item includes: and under the condition that the service node verifies that the signature of the read request response message is correct, determining a determined credible target key value item from the target key value items in the plurality of read request response messages according to a simple majority principle.

It should be noted that, after receiving the read request message, each storage node first verifies the message signature, then queries the value of V corresponding to K in the local storage according to the value of K in the message, and composes the queried value of V into a read request response message, and after signing the read request response message, replies to the service node.

After receiving the read request response message replied by the storage node, the service node sequentially verifies the message signature for the read request response message, compares the V values in all the read request response messages after the verification of the signature is passed, and obtains the credible V value through a simple majority principle.

Through the embodiment, the data is stored in the plurality of storage nodes in a multi-copy storage mode, and the obtained plurality of data are compared, so that the problem that the data is not accurately read due to the fact that the data is abnormal and only stored in one storage node is solved.

In order to maintain the validity and stability of the nodes in the node cluster, optionally, in the data processing method provided in the embodiment of the present application, the monitoring and determining, by the monitoring node, whether the message in the message queue meets the preset node condition includes: judging whether a target write node in the write request message is a registered node or not, and disconnecting the node from the message queue under the condition that the target write node is not the registered node; judging whether a storage node in the write request confirmation message is a registered node or not, and disconnecting the node from the message queue under the condition that the storage node is not the registered node; judging whether a packing node in the block message is a registered node or not, and disconnecting the node from the message queue under the condition that the packing node in the block message is not the registered node; and/or judging whether the frequency of the write request message sent by the target write-in node exceeds a first preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency exceeds the first preset frequency; judging whether the frequency of the write request confirmation message sent by the storage node exceeds a second preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency of the write request confirmation message exceeds the second preset frequency; and judging whether the frequency of the block message sent by the packaging node exceeds a third preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency exceeds the third preset frequency.

It should be noted that the supervision node acquires the write request message, the write request acknowledgement message, and the block message by monitoring the message queue, and monitors the message, specifically, the supervision node forcibly disconnects the corresponding node from the message queue if the public key in the message belongs to the registered node or not, and if the initiation frequency exceeds the limit of the supervision node.

Optionally, in the data processing method provided in this embodiment of the present application, the method further includes: after the service node sends the write request message to the message queue, determining to enter a state that the write request is submitted; and/or after the service node monitors a write request confirmation message in the message queue, determining to enter a state that the write request is confirmed; and/or after the service node monitors the block message containing the write request confirmation message in the message queue, determining that the entering write request is in a finally confirmed state.

It should be noted that the service node is an initiator of the write flow, and the service node may continuously update the change of the state of the write request according to the change of the message in the message queue.

Optionally, in the data processing method provided in this embodiment of the present application, before the writing request message and the writing request acknowledgement message are packaged into the chunk message by the packaging node, the method further includes: sending the registration information of the packaging node to a message queue; updating cluster configuration information through a supervision node under the condition that the registration information is verified to be valid, and sending the cluster configuration information to a message queue, wherein the cluster configuration information at least comprises a packaging sequence and a packaging time interval; and under the condition that the public key contained in the cluster configuration information belongs to the supervision node and the signature verification of the cluster configuration information passes, the step of packaging the write request message and the write request confirmation message into the block message by the packaging node is executed.

Specifically, the packaging node sends a registration message to the message queue, where the registration message includes a public key and an id of the packaging node itself and available resources, such as a hard disk, a cpu, and a memory.

The supervision node acquires the registration information of the packing nodes through the subscription message queue, verifies the validity of the message, redistributes the packing sequence of the packing nodes according to the existing packing node state and resource condition in the cluster to form new cluster configuration information, and sends the new cluster configuration information to the message queue, wherein the cluster configuration information comprises the current block height, the packing node information in the cluster and the like besides the node packing sequence and the packing time interval.

Before the packing, it is necessary to determine validity of a node corresponding to the message and correctness of a signature of the message, and optionally, in the data processing method provided in the embodiment of the present application, before the writing request message and the writing request acknowledgement message are packed into a block message by the packing node, the method further includes: judging whether a target write node corresponding to the write request message and/or a storage node corresponding to the write request confirmation message are valid nodes or not; verifying whether the signature in the write request message is correct and/or verifying whether the signature in the write request confirmation message is correct under the condition that the target write node and/or the storage node are valid nodes; the write request message is added to the message pool of the packing node in case the signature in the write request message is correct and/or the write request acknowledge message is added to the message pool of the packing node in case the signature in the write request acknowledge message is correct.

Specifically, the packing node verifies the public key of the message first, then verifies the signature, if the verification is successful, the packing node starts to monitor the message queue to obtain the write request message, the write request confirmation message and the block message, adds the message into the message pool of the packing node, and waits for packing.

The method comprises the steps of comparing a public key of a target write-in node contained in a write request message and a public key of a storage node contained in a write request confirmation node with a legal node public key stored in a packaging node, judging whether the target write-in node and the storage node are valid nodes or not based on a comparison result, verifying whether a signature in the write request message is correct or not based on the public key of the target write-in node and verifying whether the signature in the write request confirmation message is correct or not based on the public key of the storage node under the condition that the target write-in node and the storage node are valid nodes.

Optionally, in the data processing method provided in the embodiment of the present application, the packing, by the packing node, the write request message and the write request acknowledge message into a chunk message includes: determining a time slot corresponding to each packing node in a preset time period based on the packing sequence and the packing time interval; and each packing node selects a preset number of messages from the message pool in the corresponding time slot to pack to obtain a plurality of block messages.

It should be noted that the supervisory node divides the preset time into equal time slots, and allocates these time slots to each of the packing nodes for packing blocks according to the resource status of the packing nodes, such as the status of hard disk, cpu, and memory.

In addition, each block also comprises a hash of the block generated in the last time slot, the hash of the block, a timestamp generated by the block, a public key of the packing node and a signature of the hash of the block by using a corresponding private key of the public key.

In order to improve the packing efficiency, optionally, in the data processing method provided in the embodiment of the present application, the method further includes: determining that the packaging of the packaging node fails under the condition that the packaging of the packaging node is not completed in the corresponding time slot; and under the condition that the packaging failure times of the packaging nodes exceed the preset times, the packaging nodes are removed through the supervision nodes, and the cluster configuration information is updated.

It should be noted that each packing node must complete the packing of the blocks within the time slot specified by the supervisory node, and construct the packed blocks into block messages to be sent to the message queue. If one packing node can not complete packing in the specified time slot, the next packing node starts to perform block packing, the supervision node can record the node which can not complete the packing task as a packing failure, if one packing node fails to pack for specified times in a period of continuous time, the node is considered as an unqualified packing node, and the supervision node can kick out the unqualified packing node to re-configure the packing sequence and the packing time slot in the cluster configuration information.

In order to avoid that multiple packing nodes pack the same message, optionally, in the data processing method provided in the embodiment of the present application, the method further includes: after monitoring the target block message in the message queue, the target packaging node verifies the validity of the target block message, and analyzes the target block message under the valid condition to obtain a target write request message and a target write request confirmation message; detecting whether a target write request message and a target write request confirmation message exist in a message pool of a target packaging node; if so, the target write request message and the target write request acknowledge message are removed from the message pool of the target packaging node.

It should be noted that other packing nodes obtain the packed blocks by listening to the message queue. After receiving the block, firstly, the validity of the block is verified through information such as block height, block hash and the like, signature and the like, and if the block is valid, the message write request message and the write request confirmation message contained in the block are removed from a local 'write request message pool'. Through the embodiment, the resource occupation of repeated packing of a plurality of packing nodes is avoided.

FIG. 2 is a schematic diagram of a data processing system according to an embodiment of the present application. As shown in fig. 2, the system includes:

the service layer comprises a plurality of service nodes and is used for initiating a write request and a read request, wherein the write request is used for requesting to write a target transaction into the storage node, and the read request is used for reading the target transaction from the storage node.

Specifically, the service node initiates the write request by sending a write request message, where the write request message includes written key-value pair information, a signature of the key-value pair information, a public key of the write node, and a signature of the write node. The service node initiates a read request by sending a read request message, wherein the read request corresponds to a read request message and comprises a value of K to be read, an id of a write node, a public key of the write node and a signature of the write node on the message.

In addition, it should be noted that the service layer may interact with the storage node on one hand, and may also interact with an external access layer on the other hand, specifically, interact with multiple access devices in the access layer through the service node.

The storage layer comprises a plurality of packing nodes and is used for packing write request messages and write request confirmation messages into block messages, wherein the write request messages represent that the write nodes request to write target transactions into the storage nodes, and the write request confirmation messages represent whether the target transactions are stored in the storage nodes or not.

Specifically, in the packing process, the packing node selects the transaction messages to be packed according to the principle of entity correlation priority and packs the selected transaction messages into block messages. It should be noted that the number of transactions in a block does not exceed the number of lines, for example, 1000, and each block contains a hash of the previous block, a creation time of the block, the stamped transaction information, a public key of the block creation node (packing node), and a signature of the block creation node (packing node).

And the storage node is used for storing the block message.

In particular, the bottom layer of storage nodes may be supported by a rockDB, each storage node providing the functionality of a variety of range queries.

When storing data, the storage node can acquire the block packed by the packing node, verify the validity of the block and store the valid block, and can interact with the service node of the service layer.

And the supervision node is used for supervising the admission of the node in the storage layer and the exit of the node, and supervising the condition of writing data into the storage node and the condition of reading data from the storage node.

Specifically, in the node admission aspect, the supervision node is responsible for the trust of a legal node and the elimination of an illegal node, and in the node exit aspect, the supervision node is responsible for the normal exit processing of the node and the abnormal exit processing of the node.

In the aspect of data writing, the supervision node is responsible for supervision of abnormal writing behaviors (specifically comprising abnormal writing request and abnormal writing confirmation), supervision of normal writing behaviors and block validity check, specifically, when the block validity check is carried out, a packing node for creating a block is evaluated after an error block is found, and after the block is evaluated as a failure node or a malicious node, a block sequence is removed from the node and the packing sequence is adjusted.

In terms of data reading, the supervisory node is responsible for supervision of abnormal read behavior (specifically including read request exception, read response exception), and normal read behavior.

Optionally, in the data processing system provided in the embodiment of the present application, the storage layer further includes: the message queue is used for caching association messages generated in a data writing process, wherein the association messages at least comprise: a write request message, a write request acknowledge message, and a block message; and the communication network is used for providing communication for the service node and the storage node when reading data.

It should be noted that, a communication mechanism is established between nodes through a communication network and a message queue independent of a service party, specifically, the message queue is used as a communication medium when data is written into the storage node, so that the writing security is improved, and the communication network is used as a communication medium when data is read from the storage node, for example, the communication network may be a P2P network, so that the reading efficiency is improved.

When data are written into the storage nodes, each node sends the associated information generated in the data writing process to the message queue, and each node also acquires the transaction information by subscribing the message queue.

For example, after creating a block, the packaging node serializes the created block, publishes the block after serialization to a message queue, the storage node subscribes to the message queue to obtain the block after serialization, performs deserialization after obtaining the block, then verifies the validity of the block, and stores the block when the verification passes.

In the data processing system provided by the embodiment of the application, through a service layer, the service layer includes a plurality of service nodes and is used for initiating a write request and a read request, wherein the write request is used for requesting to write a target transaction into a storage node, and the read request is used for reading the target transaction from the storage node; the storage layer comprises a plurality of packing nodes and is used for packing a write request message and a write request confirmation message into a block message, wherein the write request message represents that the write node requests to write a target transaction into the storage node, and the write request confirmation message represents whether the target transaction is stored in the storage node or not; a storage node for storing the block message; the supervision node is used for supervising the access of the node in the storage layer and the exit of the node, and supervising the condition of writing data into the storage node and the condition of reading data from the storage node, so that the problem that the safety of data reading and writing and the efficiency of data reading and writing are difficult to be considered when data reading and writing are carried out in a data storage system in the related technology is solved, and the block message is stored in the storage node and supervised by adopting the supervision node in the storage process, so that the effects of improving the safety of data reading and writing and the efficiency of reading and writing are achieved.

It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.

The embodiment of the present application further provides a data processing apparatus, and it should be noted that the data processing apparatus according to the embodiment of the present application may be used to execute the method for data processing provided in the embodiment of the present application. The following describes a data processing apparatus according to an embodiment of the present application.

Fig. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 3, the apparatus includes: a first sending unit 10, a judging unit 20, a packing unit 30 and a storing unit 40.

Specifically, the first sending unit 10 is configured to send, by the service node, a write request message to the message queue, where the write request message indicates that the target write node requests to write the target key value and the target key value item to the storage node.

A judging unit 20, configured to judge whether the storage node has stored the content requested to be written, determine a write request acknowledgement message based on the judgment result, and send the write request acknowledgement message to the message queue.

And a packing unit 30, configured to pack, by a packing node, the write request message and the write request acknowledge message into a block message and send the block message to a message queue, when the write request message and the write request acknowledge message satisfy a preset node condition.

And the storage unit 40 is used for storing the block message to a plurality of storage nodes under the condition that the block message meets the preset node condition, wherein the monitoring node monitors and judges whether the message in the message queue meets the preset node condition.

The data processing apparatus provided in the embodiment of the present application is configured to send, by a first sending unit 10, a write request message to a message queue through a service node, where the write request message represents that a target write node requests to write a target key value and a target key value item into a storage node; a judging unit 20, configured to judge whether the storage node has stored the content requested to be written, determine a write request acknowledgement message based on a judgment result, and send the write request acknowledgement message to the message queue; a packing unit 30, configured to pack, by a packing node, the write request message and the write request acknowledge message into a block message and send the block message to a message queue when the write request message and the write request acknowledge message satisfy a preset node condition; the storage unit 40 is configured to store the block message to a plurality of storage nodes when the block message meets a preset node condition, where a monitoring node monitors and determines whether the message in the message queue meets the preset node condition, so as to solve the problem in the related art that it is difficult to consider both data read-write security and data read-write efficiency when data is read and written in a data storage system.

Optionally, in the data processing apparatus provided in the embodiment of the present application, the apparatus further includes: the second sending unit is used for sending a read request message to the plurality of storage nodes through the service node, wherein the read request message is used for requesting to read a target key value item corresponding to a written target key value; the acquisition unit is used for acquiring a plurality of target key value items obtained by a plurality of storage nodes based on target key value query; and the comparison unit is used for comparing the target key value items and determining the credible target key value items according to the comparison result.

Optionally, in the data processing apparatus provided in this embodiment of the present application, the storage node encapsulates a target key value item obtained by query as a read request response message, and the comparing unit includes: and the first determining module is used for determining a credible target key value item from the target key value items in the multiple read request response messages according to a simple majority principle under the condition that the service node verifies that the signature of the read request response message is correct.

Optionally, in the data processing apparatus provided in the embodiment of the present application, the determining unit 20 includes: the first judgment module is used for judging whether a target key value requesting for writing exists in the storage node or not under the condition that the public key of the target writing node contained in the writing request message is a legal node public key and the signature in the writing request message is verified to be correct; a second determining module, configured to determine, in the presence of a target key value, a first write request acknowledge message, where a write status of the first write request acknowledge message is: the target key value is occupied; a third determining module, configured to determine, in the absence of the target key value, a second write request acknowledgement message, where a write status of the second write request acknowledgement message is: the writing was successful.

Optionally, in the data processing apparatus provided in this embodiment of the present application, the apparatus further includes a monitoring unit, configured to monitor and determine whether a message in the message queue satisfies a preset node condition through a monitoring node, where the monitoring unit includes: the second judging module is used for judging whether a target writing node in the writing request message is a registered node or not, and disconnecting the node from the message queue under the condition that the target writing node is not the registered node; the third judging module is used for judging whether the storage node in the write request confirmation message is a registered node or not, and disconnecting the node from the message queue under the condition that the storage node is not the registered node; the fourth judging module is used for judging whether the packaging node in the block message is a registered node or not, and disconnecting the node from the message queue under the condition that the packaging node is not the registered node; and/or the fifth judging module is used for judging whether the frequency of the target writing node for sending the writing request message exceeds a first preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency exceeds the first preset frequency; the sixth judging module is used for judging whether the frequency of the write request confirmation message sent by the storage node exceeds a second preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency of the write request confirmation message sent by the storage node exceeds the second preset frequency; and the seventh judging module is used for judging whether the frequency of the block message sent by the packaging node exceeds a third preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency of the block message sent by the packaging node exceeds the third preset frequency.

Optionally, in the data processing apparatus provided in the embodiment of the present application, the apparatus further includes: the first determining unit is used for determining to enter a state that the write request is submitted after the service node sends the write request message to the message queue; and/or the second determining unit is used for determining to enter a state that the write request is confirmed after the service node monitors the write request confirmation message in the message queue; and/or the third determining unit is used for determining that the write request is entered into a finally confirmed state after the service node monitors the block message containing the write request confirmation message in the message queue.

Optionally, in the data processing apparatus provided in the embodiment of the present application, the apparatus further includes: a third sending unit, configured to send the registration information of the packing node to the message queue before the write request message and the write request acknowledgement message are packed into the chunk message by the packing node; the updating unit is used for updating the cluster configuration information through the supervision node under the condition of verifying that the registration information is valid, and sending the cluster configuration information to the message queue, wherein the cluster configuration information at least comprises a packaging sequence and a packaging time interval; and the execution unit is used for executing the step of packaging the write request message and the write request confirmation message into the block message through the packaging node under the condition that the public key contained in the cluster configuration information belongs to the supervision node and the signature verification of the cluster configuration information passes.

Optionally, in the data processing apparatus provided in the embodiment of the present application, the apparatus further includes: the valid node judging unit is used for judging whether a target write-in node corresponding to the write request message and/or a storage node corresponding to the write request confirmation message are valid nodes or not; the first verification unit is used for verifying whether the signature in the write request message is correct and/or verifying whether the signature in the write request confirmation message is correct under the condition that the target write node and/or the storage node are valid nodes; and the adding unit is used for adding the write request message into the message pool of the packaging node under the condition that the signature in the write request message is correct, and/or adding the write request confirmation message into the message pool of the packaging node under the condition that the signature in the write request confirmation message is correct.

Optionally, in the data processing apparatus provided in the embodiment of the present application, the packing unit includes: the fourth determining module is used for determining a time slot corresponding to each packing node in a preset time period based on the packing sequence and the packing time interval; and the selecting module is used for selecting a preset number of messages from the message pool in the corresponding time slot through each packing node to pack the messages to obtain a plurality of block messages.

Optionally, in the data processing apparatus provided in the embodiment of the present application, the apparatus further includes: a fifth determining module, configured to determine that a packing node fails to pack when the packing node does not complete packing in a corresponding time slot; and the updating module is used for eliminating the packaging node through the supervision node and updating the cluster configuration information under the condition that the packaging failure times of the packaging node exceed the preset times.

Optionally, in the data processing apparatus provided in the embodiment of the present application, the apparatus further includes: the second verification unit is used for verifying the validity of the target block message through the target packaging node after monitoring the target block message in the message queue, and analyzing the target block message under the valid condition to obtain a target write request message and a target write request confirmation message; the detection unit is used for detecting whether a target writing request message and a target writing request confirmation message exist in a message pool of the target packaging node; and the removing unit is used for removing the target writing request message and the target writing request confirmation message from the message pool of the target packaging node under the condition that the target writing request message and the target writing request confirmation message exist.

The data processing device comprises a processor and a memory, wherein the first sending unit 10, the judging unit 20, the packing unit 30, the storage unit 40 and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.

The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to be one or more than one, and the problem that the data reading and writing safety and the data reading and writing efficiency are difficult to be considered when data reading and writing are carried out in a data storage system in the related technology is solved by adjusting the kernel parameters.

The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.

An embodiment of the present invention provides a storage medium on which a program is stored, the program implementing a data processing method when executed by a processor.

The embodiment of the invention provides a processor, which is used for running a program, wherein the program runs to execute a data processing method.

The embodiment of the invention provides equipment, which comprises a processor, a memory and a program which is stored on the memory and can run on the processor, wherein the processor executes the program and realizes the following steps: sending a write request message to a message queue through a service node, wherein the write request message represents that a target write node requests to write a target key value and a target key value item into a storage node; judging whether the storage node stores the content of the request for writing, determining a write request confirmation message based on the judgment result, and sending the write request confirmation message to a message queue; under the condition that the write request message and the write request confirmation message meet the preset node condition, the write request message and the write request confirmation message are packaged into a block message through a packaging node, and the block message is sent to a message queue; and under the condition that the block message meets the preset node condition, storing the block message to a plurality of storage nodes, wherein the monitoring node monitors and judges whether the message in the message queue meets the preset node condition.

The method further comprises the following steps: sending a read request message to a plurality of storage nodes through a service node, wherein the read request message is used for requesting to read a target key value item corresponding to a written target key value; acquiring a plurality of target key value items obtained by a plurality of storage nodes based on target key value query; and comparing the target key value items, and determining a credible target key value item according to a comparison result.

The storage node encapsulates the target key value items obtained by query into a read request response message, compares the target key value items, and determines the credible target key value items, wherein the step of determining the credible target key value items comprises the following steps: and under the condition that the service node verifies that the signature of the read request response message is correct, determining a determined credible target key value item from the target key value items in the plurality of read request response messages according to a simple majority principle.

Judging whether the storage node stores the content requested to be written, and determining the write request confirmation message based on the judgment result comprises the following steps: under the condition that the public key of the target write-in node contained in the write request message is a legal node public key and the signature in the write request message is verified to be correct, judging whether a target key value requesting for writing exists in the storage node; under the condition that the target key value exists, determining a first write request confirmation message, wherein the write state of the first write request confirmation message is as follows: the target key value is occupied; determining a second write request confirmation message under the condition that the target key value does not exist, wherein the write state of the second write request confirmation message is as follows: the writing was successful.

The monitoring and judging whether the messages in the message queue meet the preset node condition by the monitoring node comprises the following steps: judging whether a target write node in the write request message is a registered node or not, and disconnecting the node from the message queue under the condition that the target write node is not the registered node; judging whether a storage node in the write request confirmation message is a registered node or not, and disconnecting the node from the message queue under the condition that the storage node is not the registered node; judging whether a packing node in the block message is a registered node or not, and disconnecting the node from the message queue under the condition that the packing node in the block message is not the registered node; and/or judging whether the frequency of the write request message sent by the target write-in node exceeds a first preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency exceeds the first preset frequency; judging whether the frequency of the write request confirmation message sent by the storage node exceeds a second preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency of the write request confirmation message exceeds the second preset frequency; and judging whether the frequency of the block message sent by the packaging node exceeds a third preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency exceeds the third preset frequency.

The method further comprises the following steps: after the service node sends the write request message to the message queue, determining to enter a state that the write request is submitted; and/or after the service node monitors a write request confirmation message in the message queue, determining to enter a state that the write request is confirmed; and/or after the service node monitors the block message containing the write request confirmation message in the message queue, determining that the entering write request is in a finally confirmed state.

Before packaging the write request message and the write request acknowledge message into a chunk message by the packaging node, the method further comprises: sending the registration information of the packaging node to a message queue; updating cluster configuration information through a supervision node under the condition that the registration information is verified to be valid, and sending the cluster configuration information to a message queue, wherein the cluster configuration information at least comprises a packaging sequence and a packaging time interval; and under the condition that the public key contained in the cluster configuration information belongs to the supervision node and the signature verification of the cluster configuration information passes, the step of packaging the write request message and the write request confirmation message into the block message by the packaging node is executed.

Before packaging the write request message and the write request acknowledge message into a chunk message by the packaging node, the method further comprises: judging whether a target write node corresponding to the write request message and/or a storage node corresponding to the write request confirmation message are valid nodes or not; verifying whether the signature in the write request message is correct and/or verifying whether the signature in the write request confirmation message is correct under the condition that the target write node and/or the storage node are valid nodes; the write request message is added to the message pool of the packing node in case the signature in the write request message is correct and/or the write request acknowledge message is added to the message pool of the packing node in case the signature in the write request acknowledge message is correct.

Packaging, by a packaging node, a write request message and a write request acknowledge message into a chunk message comprises: determining a time slot corresponding to each packing node in a preset time period based on the packing sequence and the packing time interval; and each packing node selects a preset number of messages from the message pool in the corresponding time slot to pack to obtain a plurality of block messages.

The method further comprises the following steps: determining that the packaging of the packaging node fails under the condition that the packaging of the packaging node is not completed in the corresponding time slot; and under the condition that the packaging failure times of the packaging nodes exceed the preset times, the packaging nodes are removed through the supervision nodes, and the cluster configuration information is updated.

The method further comprises the following steps: after monitoring the target block message in the message queue, the target packaging node verifies the validity of the target block message, and analyzes the target block message under the valid condition to obtain a target write request message and a target write request confirmation message; detecting whether a target write request message and a target write request confirmation message exist in a message pool of a target packaging node; if so, the target write request message and the target write request acknowledge message are removed from the message pool of the target packaging node. The device herein may be a server, a PC, a PAD, a mobile phone, etc.

The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device: sending a write request message to a message queue through a service node, wherein the write request message represents that a target write node requests to write a target key value and a target key value item into a storage node; judging whether the storage node stores the content of the request for writing, determining a write request confirmation message based on the judgment result, and sending the write request confirmation message to a message queue; under the condition that the write request message and the write request confirmation message meet the preset node condition, the write request message and the write request confirmation message are packaged into a block message through a packaging node, and the block message is sent to a message queue; and under the condition that the block message meets the preset node condition, storing the block message to a plurality of storage nodes, wherein the monitoring node monitors and judges whether the message in the message queue meets the preset node condition.

The method further comprises the following steps: after monitoring the target block message in the message queue, the target packaging node verifies the validity of the target block message, and analyzes the target block message under the valid condition to obtain a target write request message and a target write request confirmation message; detecting whether a target write request message and a target write request confirmation message exist in a message pool of a target packaging node; if so, the target write request message and the target write request acknowledge message are removed from the message pool of the target packaging node.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data processing method, comprising:

sending a write request message to a message queue through a service node, wherein the write request message represents that a target write node requests to write a target key value and a target key value item into a storage node;

judging whether the storage node stores the content of the request for writing, determining a writing request confirmation message based on the judgment result, and sending the writing request confirmation message to the message queue;

under the condition that the write request message and the write request confirmation message meet preset node conditions, packaging the write request message and the write request confirmation message into a block message through a packaging node, and sending the block message to the message queue;

and under the condition that the block message meets the preset node condition, storing the block message to a plurality of storage nodes, wherein a monitoring node monitors and judges whether the message in the message queue meets the preset node condition.

2. The method of claim 1, further comprising:

sending a read request message to the plurality of storage nodes through the service node, wherein the read request message is used for requesting to read the target key value item corresponding to the written target key value;

acquiring a plurality of target key value items obtained by a plurality of storage nodes based on the target key value query;

and comparing the target key value items, and determining a credible target key value item according to a comparison result.

3. The method of claim 2, wherein the step of encapsulating, by the storage node, the target key value item obtained by the query as a read request response message, comparing the target key value items, and determining a trusted target key value item according to the comparison result comprises:

and under the condition that the service node verifies that the signature of the read request response message is correct, determining a determined credible target key value item from the target key value items in the read request response messages according to a simple majority principle.

4. The method of claim 1, wherein determining whether the storage node has stored content requested to be written, and wherein determining a write request acknowledge message based on the determination comprises:

under the condition that the public key of the target write-in node contained in the write request message is a legal node public key and the signature in the write request message is verified to be correct, judging whether the target key value requesting for writing exists in the storage node;

determining a first write request acknowledge message under the condition that the target key value exists, wherein the write state of the first write request acknowledge message is as follows: the target key value is occupied;

determining a second write request confirmation message under the condition that the target key value does not exist, wherein the write state of the second write request confirmation message is as follows: the writing was successful.

5. The method of claim 1, wherein listening and determining, by the supervising node, whether the messages in the message queue satisfy the preset node condition comprises:

judging whether the target writing node in the writing request message is a registered node or not, and disconnecting the node from the message queue under the condition that the target writing node is not the registered node;

judging whether the storage node in the write request confirmation message is a registered node or not, and disconnecting the node from the message queue under the condition that the storage node is not the registered node;

judging whether the packaging node in the block message is a registered node or not, and disconnecting the node from the message queue under the condition that the packaging node in the block message is not the registered node;

and/or judging whether the frequency of the write request message sent by the target write-in node exceeds a first preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency exceeds the first preset frequency;

judging whether the frequency of the write request confirmation message sent by the storage node exceeds a second preset frequency or not, and disconnecting the node from the message queue under the condition that the frequency of the write request confirmation message exceeds the second preset frequency;

and judging whether the frequency of the block message sent by the packing node exceeds a third preset frequency or not, and disconnecting the node from the message queue under the condition of exceeding the third preset frequency.

6. The method of claim 1, further comprising:

after the service node sends the write request message to the message queue, determining to enter a state that the write request is submitted;

and/or after the service node monitors the write request confirmation message in the message queue, determining to enter a state that the write request is confirmed;

and/or after the service node monitors the block message containing the write request confirmation message in the message queue, determining that the write request is entered into a finally confirmed state.

7. The method of claim 1, wherein prior to packaging the write request message and the write request acknowledge message into a chunk message by a packaging node, the method further comprises:

sending the registration information of the packaging node to the message queue;

updating cluster configuration information through the supervisory node under the condition that the registration information is verified to be valid, and sending the cluster configuration information to the message queue, wherein the cluster configuration information at least comprises a packing sequence and a packing time interval;

and if the public key contained in the cluster configuration information belongs to the supervision node and the signature verification of the cluster configuration information passes, performing a step of packaging the write request message and the write request confirmation message into a block message through a packaging node.

8. The method of claim 7, wherein prior to packaging the write request message and the write request acknowledge message into a chunk message by a packaging node, the method further comprises:

judging whether the target write-in node corresponding to the write request message and/or the storage node corresponding to the write request confirmation node are valid nodes or not;

verifying whether the signature in the write request message is correct and/or verifying whether the signature in the write request acknowledge message is correct under the condition that the target write node and/or the storage node is a valid node;

adding the write request message to a message pool of the packing node if the signature in the write request message is correct, and/or adding the write request acknowledge message to the message pool of the packing node if the signature in the write request acknowledge message is correct.

9. The method of claim 8, wherein packaging the write request message and the write request acknowledge message into a chunk message by a packaging node comprises:

determining a time slot corresponding to each packing node in a preset time period based on the packing sequence and the packing time interval;

and each packing node selects a preset number of messages from the message pool in a corresponding time slot to pack to obtain a plurality of block messages.

10. The method of claim 9, further comprising:

determining that the packing node fails to pack under the condition that the packing node does not complete the packing in the corresponding time slot;

and under the condition that the packaging failure times of the packaging nodes exceed the preset times, the packaging nodes are removed through the supervision nodes, and the cluster configuration information is updated.

11. The method of claim 8, further comprising:

after monitoring the target block message in the message queue, the target packaging node verifies the validity of the target block message, and analyzes the target block message under the valid condition to obtain a target write request message and a target write request confirmation message;

detecting whether the target writing request message and the target writing request confirmation message exist in the message pool of the target packaging node;

if so, removing the target write request message and the target write request acknowledge message from the message pool of the target packing node.

12. A data processing system, comprising:

the service layer comprises a plurality of service nodes and is used for initiating a write request and a read request, wherein the write request is used for requesting a target transaction to be written into a storage node, and the read request is used for reading the target transaction from the storage node;

the storage layer comprises a plurality of packing nodes and is used for packing a write request message and a write request confirmation message into a block message, wherein the write request message represents that the write node requests to write the target transaction into the storage node, and the write request confirmation message represents whether the target transaction is stored in the storage node or not;

a storage node for storing the block message;

and the supervision node is used for supervising the admission of the node and the exit of the node, and supervising the condition of writing data into the storage node and the condition of reading data from the storage node.

13. The system of claim 12, wherein the storage layer further comprises:

the message queue is used for buffering association messages generated in a data writing process, wherein the association messages at least comprise: the write request message, the write request acknowledge message, and the chunk message;

a communication network for providing communication for the service node and the storage node when reading data.

14. A data processing apparatus, comprising:

the write request message represents a target write node to request to write a target key value and a target key value item into a storage node;

the judging unit is used for judging whether the storage node stores the content of the request for writing, determining a writing request confirmation message based on the judgment result and sending the writing request confirmation message to the message queue;

a packing unit, configured to pack, by a packing node, the write request message and the write request acknowledgement message into a block message and send the block message to the message queue, when the write request message and the write request acknowledgement message satisfy a preset node condition;

and the storage unit is used for storing the block message to a plurality of storage nodes under the condition that the block message meets the preset node condition, wherein a monitoring node monitors and judges whether the message in the message queue meets the preset node condition.

15. A storage medium characterized in that the storage medium includes a stored program, wherein the program executes the data processing method of any one of claims 1 to 11.

16. A processor, characterized in that the processor is adapted to run a program, wherein the program when running performs the data processing method of any of claims 1 to 11.