CN110602251B

CN110602251B - Data processing method, device, apparatus and medium based on inter-node data sharing

Info

Publication number: CN110602251B
Application number: CN201910940530.7A
Authority: CN
Inventors: 章天豪
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2019-09-30
Filing date: 2019-09-30
Publication date: 2021-12-17
Anticipated expiration: 2039-09-30
Also published as: CN110602251A

Abstract

The disclosure provides a data processing method, device, apparatus, and medium based on inter-node data sharing. The data processing method based on the data sharing among the nodes comprises the following steps: processing data to be identified by utilizing a hash function to obtain a hash value of the data to be identified; determining whether the data to be identified is the same as source data corresponding to the stored hash value by comparing the hash value of the data to be identified with the hash value stored in a chained data structure of the current node, wherein the hash value stored in the chained data structure is obtained by processing the source data by using the hash function; and determining to update the chained data structure of the current node and sending chained data structure update messages to other nodes under the condition that the data to be identified is different from the source data corresponding to the stored hash value, wherein the chained data structure update messages are used for updating the chained data structures of the other nodes.

Description

Data processing method, device, apparatus and medium based on inter-node data sharing

Technical Field

The present disclosure relates to the field of data processing, and in particular, to a data processing method, device, apparatus, and medium based on inter-node data sharing.

Background

The internet is widely used in real life, and users can chat, play games, look up data and the like on the internet, and can also carry out advertising, shopping and the like on the internet. However, bad information such as bad pictures, false information, etc. is also spread by means of the internet. In general, a person skilled in the art performs post-verification of information propagated through the internet, and if it is determined that the information is bad, a masking measure is taken for the information. However, for bad information that has been masked, the propagator can employ a means, such as tampering, to spoof the audit mechanism to propagate the information again without affecting the content of the information body, which reduces the effectiveness of the information masking. In addition, the propagator of the bad information can also propagate the bad information on a plurality of internet system platforms, and each system platform needs to respectively audit the bad information propagated on the platform, which increases the audit burden. In order to maintain a good network environment, it is necessary to provide a data processing method for examining bad information propagated on the internet to take effective shielding measures and achieve a synchronous shielding effect of each system platform.

Disclosure of Invention

The disclosure provides a data processing method, device, apparatus, and medium based on inter-node data sharing.

According to an aspect of the present disclosure, a data processing method based on inter-node data sharing is provided, including: processing data to be identified by utilizing a hash function to obtain a hash value of the data to be identified; determining whether the data to be identified is the same as source data corresponding to the stored hash value by comparing the hash value of the data to be identified with the hash value stored in a chained data structure of the current node, wherein the hash value stored in the chained data structure is obtained by processing the source data by using the hash function; and determining to update the chained data structure of the current node and sending chained data structure update messages to other nodes under the condition that the data to be identified is different from the source data corresponding to the stored hash value, wherein the chained data structure update messages are used for updating the chained data structures of the other nodes.

According to some embodiments of the present disclosure, the data processing method further includes, in a case that the data to be identified is the same as the source data corresponding to the stored hash value, masking the data to be identified.

According to some embodiments of the disclosure, the data processing method further comprises updating the chained data structure of the current node: and creating a data block based on the hash value of the data to be identified, and adding the created data block into the chained data structure of the current node to update the chained data structure.

According to some embodiments of the disclosure, sending the chained data structure update message to the other node comprises: acquiring the identification of other nodes, and sending a chained data structure updating message to the other nodes based on the identification, wherein the chained data structure updating message comprises the created data block, the created data block is used for being verified by the other nodes, and the chained data structure of the other nodes is updated under the condition that the data block passes the verification.

According to some embodiments of the present disclosure, determining whether the data to be identified is the same as the source data corresponding to the stored hash value includes: under the condition that the hash value of the data to be identified is the same as the hash value stored in the chained data structure, determining that the data to be identified is the same as the source data corresponding to the stored hash value; and determining that the data to be identified is different from the source data corresponding to the stored hash value under the condition that the hash value of the data to be identified is different from the hash value stored in the chained data structure.

According to some embodiments of the disclosure, the chained data structure is comprised of at least one data block, each of the at least one data block comprising: a chunk body to store data, wherein the data includes a hash value; and a block header for storing characteristic information of the data block, wherein the characteristic information includes a characteristic value, a version number, a time stamp, and a difficulty value of the data.

According to some embodiments of the present disclosure, processing data to be recognized by using a hash function, and obtaining a hash value of the data to be recognized includes: processing a data stream of data to be identified by using a hash function to obtain a hash value of the data to be identified; and the hash value stored in the chained data structure is obtained by processing the data stream of the source data by using the hash function.

According to some embodiments of the present disclosure, processing data to be recognized by using a hash function, and obtaining a hash value of the data to be recognized further includes: intercepting at least one part of data flow in the data flow, and processing the at least one part of data flow by utilizing a hash function to obtain a hash value.

According to some embodiments of the present disclosure, processing data to be recognized by using a hash function, and obtaining a hash value of the data to be recognized further includes: converting the data stream into floating point numbers; squaring the floating point number to obtain a square result; and processing at least one part of the square result by utilizing the hash function to obtain a hash value.

According to some embodiments of the present disclosure, the source data includes at least one of a picture, a link, and a video.

According to some embodiments of the disclosure, the data processing method further comprises: receiving a chained data structure update message from other nodes, wherein the chained data structure update message includes a data block, and the method further includes: and checking the data block, and updating the chained data structure of the current node under the condition of passing the check.

According to another aspect of the present disclosure, there is also provided a data processing apparatus based on inter-node data sharing, including: the processing unit is configured to process data to be identified by utilizing a hash function to obtain a hash value of the data to be identified; the determining unit is configured to determine whether the data to be identified is the same as source data corresponding to the stored hash value by comparing the hash value of the data to be identified with the hash value stored in a chained data structure of the current node, wherein the hash value stored in the chained data structure is obtained by processing the source data by using the hash function; and the transmission unit is configured to determine to update the chained data structure of the current node and send a chained data structure update message to other nodes under the condition that the data to be identified is different from the source data corresponding to the stored hash value, wherein the chained data structure update message is used for updating the chained data structures of the other nodes.

According to some embodiments of the present disclosure, the data processing apparatus further includes a masking unit configured to mask the data to be identified if the data to be identified is the same as the source data corresponding to the stored hash value.

According to some embodiments of the present disclosure, the data processing apparatus further comprises an updating unit configured to update the chained data structure of the current node, including: and creating a data block based on the hash value of the data to be identified, adding the created data block to the chained data structure of the current node to update the chained data structure, wherein the transmission unit is further configured to acquire an identifier of another node, and sending a chained data structure update message to the other node based on the identifier, wherein the chained data structure update message includes the created data block, and the created data block is used for being checked by the other node, and if the check is passed, the chained data structure of the other node is updated.

According to some embodiments of the present disclosure, the determining unit is configured to determine that the data to be identified is the same as the source data corresponding to the stored hash value, if the hash value of the data to be identified is the same as the hash value stored in the chained data structure; and determining that the data to be identified is different from the source data corresponding to the stored hash value under the condition that the hash value of the data to be identified is different from the hash value stored in the chained data structure.

According to some embodiments of the disclosure, the chained data structure is comprised of at least one data block, each of the at least one data block comprising: a chunk body to store data, wherein the data includes a hash value; and characteristic information of the chained data structure, wherein the characteristic information comprises a characteristic value, a version number, a time stamp and a difficulty value of the data.

According to some embodiments of the present disclosure, the processing unit is configured to process a data stream of data to be identified by using a hash function, to obtain a hash value of the data to be identified; and the hash value stored in the chained data structure is obtained by processing the data stream of the source data by using the hash function.

According to some embodiments of the present disclosure, the processing unit is further configured to intercept at least a portion of the data stream, and process the at least a portion of the data stream by using a hash function to obtain a hash value.

According to some embodiments of the disclosure, the processing unit is further configured to convert the data stream into floating point numbers; squaring the floating point number to obtain a square result; and processing at least one part of the square result by utilizing the hash function to obtain a hash value.

According to some embodiments of the disclosure, the transmission unit is further configured to receive a chained data structure update message from other nodes, wherein the chained data structure update message includes a data block therein, and the processing unit is further configured to: and checking the data block, and updating the chained data structure of the current node under the condition of passing the check.

According to another aspect of the present disclosure, there is also provided a data processing apparatus based on inter-node data sharing, including: one or more processors; and one or more memories, wherein the memories have stored therein computer readable code, which when executed by the one or more processors, performs the inter-node data sharing based data processing method as described above.

According to yet another aspect of the present disclosure, there is also provided a computer-readable storage medium having stored thereon instructions, which, when executed by a processor, cause the processor to perform the inter-node data sharing-based data processing method as described above.

According to the data processing method based on the data sharing among the nodes, the data to be shielded can be determined by processing the data to be identified, so that effective shielding measures are realized, and synchronous shielding effects among the nodes are realized.

Drawings

In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 shows a flow diagram of a data processing method according to an embodiment of the present disclosure;

FIG. 2 shows a schematic diagram of a data sharing system;

FIG. 3A illustrates a schematic diagram of a chained data structure according to an embodiment of the disclosure;

FIG. 3B shows a schematic flow diagram of data processing using a method according to the present disclosure;

FIG. 4 shows a schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 5 shows a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 6 shows a schematic diagram of an architecture of an exemplary computing device, according to an embodiment of the present disclosure;

FIG. 7 shows a schematic diagram of a storage medium according to an embodiment of the disclosure.

Detailed Description

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. It is to be understood that the described embodiments are merely exemplary of some, and not all, of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without any inventive step, are intended to be within the scope of the present disclosure.

The use of "first," "second," and similar terms in this disclosure is not intended to indicate any order, quantity, or importance, but rather is used to distinguish one element from another. Likewise, the word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

Flow charts are used in this disclosure to illustrate steps of methods according to embodiments of the disclosure. It should be understood that the preceding and following steps are not necessarily performed in the exact order in which they are performed. Rather, various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to the processes, or a certain step or steps may be removed from the processes.

As described above, bad information such as bad pictures, videos, false information, and links to the bad information can be quickly spread via the internet. The existing review mechanism is post review, that is, the essential content of the information which is already published on the internet is reviewed, and if the information is determined to be bad information, shielding measures are taken to prevent the further propagation of the information. For example, an Instant Messaging (IM) system platform may perform such post-mortem review of information propagated on its platform, e.g., if a bad picture is found, the system platform will take masking measures against it (such as prohibiting the display and propagation of the bad picture).

However, for the bad picture that has been shielded by a certain system platform, for example, the propagator may also generate a new picture by taking a way such as altering or deleting on the picture without affecting the main content of the picture, so as to cheat the review mechanism of the system platform, thereby achieving the purpose of propagating the bad picture again. At this time, the system platform needs to perform the post-audit on the new picture again to determine whether the re-propagated picture is bad information. In addition, the propagator of the bad information can synchronously propagate the information on a plurality of system platforms, and each system platform needs to respectively audit the bad information propagated on the platform, so that the audit load is increased, and the shielding synchronism among the system platforms is reduced.

The utility model provides a data processing method based on data sharing between nodes, is used for treating the data of treating the discernment, in order to confirm whether should treat the discernment data be the bad information that needs to shield it, thereby realizes effectively shielding measure, and can realize the synchronous shielding effect between each node, be favorable to reducing the manpower and examine and verify the cost, build good internet environment.

Fig. 1 shows a flow chart of a data processing method according to an embodiment of the present disclosure. As shown in fig. 1, first, in step S101, data to be recognized is processed by a hash function, so as to obtain a hash value of the data to be recognized. According to the embodiment of the disclosure, the data to be identified is data which needs to be audited to determine whether the data is bad information. For example, the data format of the data to be identified may include at least one of a picture and a link, or the data to be identified may be data in any format, such as a video, which is transmitted over the internet.

The Hash function, which may also be referred to as a Hash function, is mainly used as an encryption algorithm in the field of information security. The data is hashed by a hash function, and the obtained value may be referred to as a hash value. In other words, the hash function may represent a mapping relationship between data and a hash value. For example, the data to be identified may be denoted as str, and the hash process may be expressed as the following formula (1):

Key1＝f(str) (1)

wherein f represents a hash function, and Key1 represents a hash value obtained by processing the to-be-recognized data str by using the hash function. In practical application, the hash function can be flexibly designed according to requirements to achieve a required security level, and generally, the more complicated the calculation method is, the greater the difficulty of being cracked is, and the higher the security level is.

Next, as shown in fig. 1, in step S102, by comparing the hash value of the data to be identified with the hash value stored in the chained data structure of the current node, it is determined whether the data to be identified is the same as the source data corresponding to the stored hash value. According to the embodiment of the present disclosure, the hash value stored in the chained data structure is obtained by processing the source data using the hash function. For example, the source data may be data determined to be bad information. For example, hash values derived based on the source data may be stored in a chained data structure. In the above data masking application, the current node may correspond to one or more system platforms, such as an instant messaging system platform, and may also be a terminal device. Regarding the chained data structure, a detailed description will be made below.

According to the embodiment of the disclosure, the source data may be historical data which is determined to be bad information, or may be sensitive information which needs to be shielded. For example, the system platform may construct a bad information database to facilitate data comparison, and if data identical to the data stored in the database is found, the data may be directly determined as bad information. According to an embodiment of the present disclosure, the source data may be data stored in the bad information database. For example, the data format of the source data may include at least one of a picture and a link, or the source data may be data in any format, such as a video, etc., that is spread over the internet.

For example, the source data may be represented as source, and the processing of the source data by the hash function f may be represented as:

Key2＝f(source) (2)

the Key2 represents a hash value obtained by hashing the source data source by using a hash function. The source data can be converted into a hash value corresponding to the source data by the hash function f. In the case where the number of the source data is more than one, the plurality of source data may be converted into hash values corresponding thereto one by one using the hash function f.

In step S102, it is determined whether the data to be identified is the same as the source data by comparing the hash value of the data to be identified with the hash value stored in the chained data structure of the current node. Specifically, when the hash value of the data to be identified is the same as the hash value stored in the chained data structure, it is determined that the data to be identified is the same as the source data corresponding to the stored hash value. As described above, the data to be identified is data that needs to be audited to determine whether it is bad information, and the source data is history data that has been determined to be bad information, whereby it can be assumed that the data to be identified that is determined to be the same as the source data also belongs to bad information. The data processing method according to the embodiment of the present disclosure may further include: and shielding the data to be identified under the condition that the data to be identified is the same as the source data corresponding to the stored hash value. The masking may be processing of the data to be identified by the system platform, such as refusing to display, propagating the data to be identified, or may also include other necessary processing, without limitation.

Next, as shown in fig. 1, in step S103, when the data to be identified is different from the source data corresponding to the stored hash value, determining to update the chained data structure of the current node, and sending a chained data structure update message to another node. According to the embodiment of the present disclosure, the chained data structure update message is used to update the chained data structure of the other node. According to the embodiment of the disclosure, under the condition that the hash value of the data to be identified is different from the hash value stored in the chained data structure, it is determined that the data to be identified is different from the source data corresponding to the stored hash value. In other words, in the data processing method according to the present disclosure, the data to be identified and the source data are not directly compared, but hash values of the data to be identified and the source data are compared, and the hash values of the source data are stored in a chained data structure.

And under the condition that the data to be identified is different from the source data, determining to update the chained data structure of the current node. For example, the data to be identified may be newly generated bad information other than the source data. As an example, the displayed data to be identified may be audited by a special auditor to determine whether a masking operation is required. If it is determined to update the chained data structure of the current node, the chained data structure of the current node may be updated, and a chained data structure update message may also be sent to update the chained data structures of other nodes.

The other node may be different from the current node, e.g. may be another system platform, or another terminal device. As an example, the current node may be represented as node 1, and the other nodes may be represented as node 2, node 3, …, node N. Data sharing can be realized among the nodes, such as sharing the chained data structure, and a data sharing system is formed.

FIG. 2 shows a schematic diagram of a data sharing system according to an embodiment of the present disclosure. Specifically, the data sharing system 100 refers to a system for performing data sharing between nodes. The data sharing system 100 may be comprised of node 1, node 2, node 3, and node 4. Further, it is noted that the system 100 may also include more nodes in addition to node 1, node 2, node 3, and node 4. In the data sharing system 100 as shown in fig. 2, each node may receive input information, which may be a hash value of source data, for example, and implement data sharing based on the received input information.

According to the embodiment of the present disclosure, each node shown in fig. 2 may store a respective chained data structure therein as the shared data, and the chained data structures in the respective nodes may implement synchronous updating. In order to ensure information intercommunication within the data sharing system 100, for example, for implementing the synchronization update, an information connection may exist between each node, and information transmission may be performed between the nodes through the information connection. For example, when any node in the data sharing system 100 receives input information, other nodes in the data sharing system 100 may obtain the input information according to a consensus algorithm, and store the input information as data in shared data, so that the data stored in all nodes in the data sharing system 100 are consistent, for example, so that the chained data structures stored in the respective nodes are consistent. As an example, if the current node 1 updates the chained data structures stored therein, the chained data structures in other nodes in the data sharing system 100 may also be updated, i.e., data synchronization updates are implemented. The process of synchronizing updates of data in the data sharing system 100 will be described in detail below.

For each node in the data sharing system 100 in fig. 2, there may be a node identification corresponding to the node, and each node in the data sharing system 100 may store the node identifications of the other nodes in the data sharing system 100 so as to transmit the update data to the other nodes in the data sharing system 100 according to the node identifications of the other nodes. The node identifier may be an ip (internet protocol) address, or any other information that can be used to identify the node. As an example, a node identification list as shown in table 1 below may be stored in each node. The node identification list includes node names and node identifications (IP addresses) one-to-one corresponding to the node names.

TABLE 1

Node name	Node identification
		Node 1	117.114.151.174
Node 2	117.116.189.145
		…	…
Node N	119.123.789.258

As described above, each node may store a respective chained data structure therein as the shared data, and the chained data structures in the respective nodes may implement synchronous updating.

According to an embodiment of the present disclosure, the chained data structure is composed of at least one data block, which may also be represented as a block. Each of the at least one data chunk may include a chunk body to store data, wherein the data includes a hash value. The data block may further include a block header for storing characteristic information of the data block, wherein the characteristic information includes a characteristic value, a version number, a timestamp, and a difficulty value of the data.

FIG. 3A illustrates a schematic diagram of a chained data structure according to an embodiment of the disclosure. As shown in fig. 3A, the chained data structure 200 may include 3 blocks, wherein the 1 st block located at the head of the chained data structure may be represented as a starting block. The starting block may include a block header for storing characteristic information of the starting block and a block body for storing data. Specifically, the data stored in the chunk body may include a hash value of the source data obtained based on a hash function. The characteristic information may include a characteristic value, a version number, a time stamp, and a difficulty value of input information, for example, data stored by the tile body. Next, as shown in fig. 3A, the next block of the starting block is represented as block 1, and the block 1 takes the starting block as a parent block. Similarly, the block 1 may include a block header in which an input information characteristic value of the block 1, a block header characteristic value of a parent block (i.e., a starting block), a version number, a time stamp, and a difficulty value are stored, and a block body in which data such as a hash value is stored. The next block 2 of block 1 has the block 1 as the parent block. Similarly, the block 2 may include a block header in which an input information characteristic value of the block 2, a block header characteristic value, a version number, a time stamp, and a difficulty value of a parent block (i.e., block 1) are stored, and a block body in which data such as a hash value is stored. By analogy, the data stored in each block in the chained data structure 200 is associated with the block data stored in the parent block, so that the security and consistency of the input information in each block in the chained data structure are ensured.

As an example, the chained data structure may be implemented based on a blockchain technique, which may also be referred to as a blockchain in this example, which may be composed of one or more data blocks (alternatively referred to as chunks). The block chain technology is a fusion technology in multiple fields of point-to-point communication, digital encryption, multi-party collaborative consensus algorithm, distributed accounts book and the like, and has the characteristics of being not falsifiable, being traceable to data on a chain and the like. The chain data structure based on the block chain technology can ensure that data on a chain is credible and can be circulated, is beneficial to improving the operation efficiency and reducing the service cost.

According to the data processing method based on the data sharing among the nodes, the data to be shielded can be determined by processing the data to be identified, so that effective shielding measures are realized, and synchronous shielding effects among the nodes are realized. In addition, the data stored in the chained data structure is the hash value of the source data, and the data volume of the hash value of the source data is smaller than that of the source data, so that the storage space required by the shielding operation can be reduced.

According to an embodiment of the present disclosure, the data processing method may further include: updating the chained data structure of the current node: and creating a data block based on the hash value of the data to be identified, and adding the created data block into the chained data structure of the current node to update the chained data structure. In a case where it is determined that the masking operation needs to be performed on the data to be identified, the data to be identified may be subjected to the masking operation, and the data to be identified may also be represented as new source data (new bad information), and a hash value of the new source data (represented as a new hash value) is stored in the chained data structure to update the chained data structure in the current node.

In a case where it is determined that the data to be identified is not identical to the source data corresponding to the stored hash value, as described above, the data to be identified is data that needs to be audited to determine whether it is bad information, and the source data is history data that has been determined to be bad information, whereby normal propagation and display operations can be performed for the data to be identified that is determined to be not identical to the source data.

Hereinafter, the steps of creating a data block based on the newly added hash value of the newly added source data and adding the created data block to the chained data structure of the current node will be described in detail. The current node may be represented as node 1, which may check a newly added hash value of newly added source data, store the newly added hash value in a memory after the check is completed, and update a hash tree for recording the newly added hash value, and then update a timestamp to a time when the newly added hash value is received, and try different random numbers to perform a feature value calculation, so that the calculated feature value may satisfy the following formula:

SHA256(SHA256(version+prev_hash+merkle_root+ntime+nbits+x))＜TARGET (3)

the SHA256 is a characteristic value algorithm used for calculating a characteristic value, version (version number) is version information of a related block protocol in a chain data structure, prev _ hash is a block head characteristic value of a parent block of a current block, merkle _ root is a characteristic value of the newly added hash value, ntime is update time of a timestamp, nbits is a difficulty value which is a fixed value within a period of time and is determined again after exceeding a fixed time period, x is a random number, TARGET is a characteristic value threshold, and the characteristic value threshold can be determined according to the difficulty value nbits. When the random number satisfying the formula (3) is obtained, the newly added hash value may be stored correspondingly to generate a block header and a block body, so as to obtain a newly added block, and thus, an updated chained data structure including the newly added block may be obtained.

According to an embodiment of the present disclosure, sending the chained data structure update message to the other node includes: acquiring the identification of other nodes, and sending a chained data structure updating message to the other nodes based on the identification, wherein the chained data structure updating message comprises the created data block, the created data block is used for being verified by the other nodes, and the chained data structure of the other nodes is updated under the condition that the data block passes the verification.

Hereinafter, a step of updating the chained data structure of the other node will be described in detail. After the current node 1 updates the chained data structure stored therein, it may send the generated newly added block to other nodes (such as node 1, node 2, and node 3) in the data sharing system 100 where it is located according to node identifiers of the other nodes in the data sharing system 100, respectively, check the newly added block by the other nodes, and add the newly added block to the chained data structure in each node after the check is completed, thereby implementing synchronous update of the chained data structure in each node.

Based on the system for sharing data among the nodes, the chained data structures stored in the nodes can be updated synchronously, and in the process of data processing, the comparison benchmark for determining whether the data to be identified belongs to bad information (namely, whether the data is the same as the source data) of each node can be kept consistent. Moreover, when one of the nodes finds new bad information (new source data) which needs to be masked, a new data block can be created based on the new bad information to update the chained data structure stored in the node, and the chained data structures in other nodes in the system are updated based on the above data sharing process to ensure consistency of hash values stored in the chained data structures of the nodes. Therefore, the asynchronous information shielding of each node can be effectively avoided, and the auditing cost is saved.

According to the embodiment of the present disclosure, processing data to be recognized by using a hash function, and obtaining a hash value of the data to be recognized may include: processing a data stream of data to be identified by using a hash function to obtain a hash value of the data to be identified; and the hash value stored in the chained data structure is obtained by processing the data stream of the source data by using the hash function. For example, the data stream is represented by binary values.

According to the embodiment of the present disclosure, processing data to be recognized by using a hash function, and obtaining a hash value of the data to be recognized may further include: intercepting at least one part of data flow in the data flow, and processing the at least one part of data flow by utilizing a hash function to obtain a hash value.

In the course of data processing by a processor such as a computer, the processing object may be a binary data stream. In other words, processing the data to be identified using the hash function may be understood as processing the data stream of the data to be identified using the hash function, and similarly, processing the source data using the hash function may be understood as processing the data stream of the source data using the hash function.

At least a portion of the data stream may be intercepted before being processed by a hash function to obtain a hash value. For example, the data flow may be intercepted according to a predetermined rule. As an example, a data stream may be truncated into 5 segments of sub-data streams, for example, 0% -20% of the data stream may be regarded as a 1 st sub-data stream, 20% -40% of the data stream may be regarded as a2 nd sub-data stream, and so on. Then, for the 5 segments of the intercepted sub-data stream, the 1 st, 3 rd and 5 th segments of the sub-data stream may be processed by using a hash function to obtain an intercepted hash value, where the number of bits of the intercepted hash value is reduced compared to the hash value directly obtained from the data stream. The data streams of the source data and the data to be identified can be processed respectively by using the above manner of intercepting the data streams, and respective hash values are obtained. Therefore, the data quantity of the hash value which needs to be stored in the chain data structure can be reduced by intercepting the data flow, and the intercepted hash value with reduced digit number is easier to compare with the hash value of the data to be identified, thereby effectively improving the examination efficiency of the bad information.

During the processing of a data stream using a hash function, similar or duplicate blocks, referred to as redundant chains, may be generated. For example, for a certain bad picture, the propagator can change part of the data in the picture to obtain a modified picture with about the same content as the bad picture. Processing the bad picture by using a hash function will obtain a hash value 1, and may construct a data block 1 based on the hash value 1, and then, processing the modified picture by using the hash function to obtain a hash value 2, and may construct a data block 2 based on the hash value 2. Since the bad picture has substantially the same content as the modified picture, the resulting data block 1 is similar to the data block 2. The data blocks constructed based on the data block 1 and the data block 2, respectively, may be referred to as the redundant chain. These approximate redundancy chains will increase the data storage capacity of the node.

According to the embodiment of the present disclosure, processing data to be recognized by using a hash function, and obtaining a hash value of the data to be recognized may further include: converting the data stream into floating point numbers; squaring the floating point number to obtain a square result; and processing at least one part of the square result by utilizing the hash function to obtain a hash value.

In order to reduce or avoid the redundant chain, a data stream of the source data or the data to be identified may be converted into a floating point number, and the floating point number is squared to obtain a squared result. Then, at least a part of the square result is processed by the hash function to obtain a hash value. For partially differentiated data streams, at least a portion of the square result, e.g., the square result after discarding the values of the last two bits, may be processed to obtain a hash value after being squared. The squaring process may be implemented using square addressing. By using the above processing, the hash values after the approximate data stream encryption can be kept consistent, and taking the bad picture and the modified picture as an example, the hash value 1 and the hash value 2 obtained by calculation can be the same. For the same hash value, only one data block is required to be constructed, so that the redundant chain is avoided, and the data storage capacity of the node is reduced.

According to another embodiment of the present disclosure, the data stream may also be hashed twice, so as to further shorten the number of bits of the generated hash value. For example, for a certain picture, a first hash value may be obtained by performing hash processing on the certain picture first, and then, another hash value may be obtained by performing hash processing on the first hash value to obtain a second hash value, so as to shorten the number of bits of the hash value for storage, which is beneficial to reducing the amount of stored data of each node.

Fig. 3B shows a schematic flow chart of data processing using a method according to an embodiment of the present disclosure, and an embodiment of the data processing method will be described in detail below in conjunction with fig. 3B.

First, in step S201, a chained data structure may be constructed based on historical mask data. For example, a historical bad information database may be constructed in a big data processing manner, wherein the stored data may be used as source data. And then, carrying out hash processing on the source data in the database to obtain a hash value, and constructing the data block of the chain data structure based on the obtained hash value. Thus, an initial chained data structure for vetting bad information may be established based on the historical bad information database. In the subsequent auditing process, the chained data structure can be supplemented.

Next, in step S202, for a picture for which data identification is required, a hash value of the picture may be obtained by using a hash function and compared with the hash values stored in the chained data structure, and a node performing this step S202 may be referred to as a current node. In case the hash value of the picture is the same as the stored hash value in the chained data structure, a masking policy, such as a rejection of the display, may be taken for the recipient of the picture in step S203.

If the hash value of the picture is not the same as the hash value stored in the chained data structure, step S204 is performed to determine whether to update the chained data structure, for example, a manual auditing policy is adopted to determine whether the picture needs to be used as new source data. On the one hand, if it is determined that the chained data structure is updated, in step S205, the chained data structures of the current node and other nodes are updated, and a masking policy is taken for the receiver of the picture. The specific process of updating the chained data structure is described in detail above and is not described herein again. On the other hand, if it is determined that the chained data structure is not updated, a display policy is taken for the recipient of the picture in step S207.

The data processing method comprises the steps of comparing a hash value of data to be identified with a hash value stored in a chained data structure of a current node to determine whether the data to be identified is the same as source data corresponding to the stored hash value, determining to update the chained data structure of the current node and chained data structures of the current node and other nodes under the condition that the data to be identified is different from the source data corresponding to the stored hash value, and shielding the data to be identified under the condition that the data to be identified is the same as the source data corresponding to the stored hash value. According to the data processing method disclosed by the invention, effective data shielding can be realized, the synchronous shielding effect among all nodes can be realized, the manpower auditing cost can be reduced, and a good internet environment can be created.

According to another aspect of the present disclosure, there is also provided a data processing apparatus based on inter-node data sharing. Fig. 4 shows a schematic block diagram of a data processing device according to an embodiment of the present disclosure.

As shown in fig. 4, the data processing device 300 may include a processing unit 301, a determining unit 302, and a transmitting unit 303.

The processing unit 301 may be configured to process data to be identified by using a hash function, so as to obtain a hash value of the data to be identified. The determining unit 302 may be configured to determine whether the source data corresponding to the hash value stored in the chained data structure of the current node is the same as the hash value of the data to be identified by comparing the hash value of the data to be identified with the hash value stored in the chained data structure of the current node. According to the embodiment of the present disclosure, the hash value stored in the chained data structure is obtained by processing the source data using the hash function. The transmission unit 303 may be configured to determine to update the chained data structure of the current node and send a chained data structure update message to other nodes when the data to be identified is different from the source data corresponding to the stored hash value. According to the embodiment of the present disclosure, the chained data structure update message is used to update the chained data structure of the other node.

As shown in fig. 4, the data processing apparatus 300 according to the embodiment of the present disclosure may further include a shielding unit 304. The masking unit 304 may be configured to mask the data to be identified if the data to be identified is the same as the source data corresponding to the stored hash value.

As shown in fig. 4, the data processing apparatus 300 according to the embodiment of the present disclosure may further include an updating unit 305. The updating unit 305 may be configured to update the chained data structure of the current node, including: and creating a data block based on the hash value of the data to be identified, and adding the created data block into the chained data structure of the current node to update the chained data structure. The transmission unit 303 may be further configured to acquire an identifier of another node, and send a chained data structure update message to the other node based on the identifier, where the chained data structure update message includes the created data block, and the created data block is used for being checked by the other node, and if the created data block passes the check, the chained data structure of the other node is updated.

According to some embodiments of the present disclosure, the determining unit 302 may be configured to determine that the data to be identified is the same as the source data corresponding to the stored hash value, in a case that the hash value of the data to be identified is the same as the hash value stored in the chained data structure; and determining that the data to be identified is different from the source data corresponding to the stored hash value under the condition that the hash value of the data to be identified is different from the hash value stored in the chained data structure.

According to some embodiments of the present disclosure, the processing unit 301 may be configured to process a data stream of data to be identified by using a hash function, so as to obtain a hash value of the data to be identified. According to the embodiment of the present disclosure, the hash value stored in the chained data structure is obtained by processing the data stream of the source data by using the hash function.

According to some embodiments of the present disclosure, the processing unit 301 may be further configured to intercept at least a part of the data stream, and process the at least a part of the data stream by using a hash function to obtain a hash value.

According to some embodiments of the present disclosure, the processing unit 301 may be further configured to convert the data stream into floating point numbers; squaring the floating point number to obtain a square result; and processing at least one part of the square result by utilizing the hash function to obtain a hash value.

According to some embodiments of the present disclosure, the transmitting unit 303 may be further configured to receive a chained data structure update message from other nodes, wherein the chained data structure update message includes a data block therein. The processing unit 301 may be further configured to: and checking the data block, and updating the chained data structure of the current node under the condition of passing the check.

According to another aspect of the present disclosure, a data processing apparatus based on inter-node data sharing is also provided. Fig. 5 shows a schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure.

As shown in fig. 5, the data processing apparatus 400 may include one or more processors 401 and one or more memories 402. The memory 402 has stored therein computer readable code, which when executed by the one or more processors 401, performs the data processing method based on inter-node data sharing as described above, according to an embodiment of the present disclosure.

The method or apparatus according to embodiments of the present disclosure may also be implemented by means of an architecture of a computing device. Fig. 6 shows a schematic diagram of an architecture of an exemplary computing device, according to an embodiment of the present disclosure. As shown in fig. 6, the computing device 500 may include a bus 501, one or more CPUs 502, a Read Only Memory (ROM)503, a Random Access Memory (RAM)504, a communication port 505 connected to a network, an input/output component 506, a hard disk 507, and the like. A storage device in the computing device 500, such as the ROM 503 or the hard disk 507, may store various data or files used for processing and/or communication of the data processing method based on inter-node data sharing provided by the present disclosure and program instructions executed by the CPU. Computing device 500 may also include a user interface 508. Of course, the architecture shown in FIG. 6 is merely exemplary, and one or more components of the computing device shown in FIG. 6 may be omitted when implementing different devices, as desired.

According to yet another aspect of the present disclosure, there is also provided a computer-readable storage medium. Fig. 7 shows a schematic diagram of a storage medium 600 according to an embodiment of the disclosure.

As shown in FIG. 7, the computer storage medium 602 has computer readable instructions 601 stored thereon. The computer readable instructions 601, when executed by a processor, may perform the data processing method based on inter-node data sharing according to the embodiments of the present disclosure described with reference to the above drawings. The computer-readable storage medium includes, but is not limited to, volatile memory and/or non-volatile memory, for example. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, Read Only Memory (ROM), hard disk, flash memory, etc.

Those skilled in the art will appreciate that the disclosure of the present disclosure is susceptible to numerous variations and modifications. For example, the various devices or components described above may be implemented in hardware, or may be implemented in software, firmware, or a combination of some or all of the three.

Further, while the present disclosure makes various references to certain elements of a system according to embodiments of the present disclosure, any number of different elements may be used and run on a client and/or server. The units are illustrative only, and different aspects of the systems and methods may use different units.

It will be understood by those skilled in the art that all or part of the steps of the above methods may be implemented by instructing the relevant hardware through a program, and the program may be stored in a computer readable storage medium, such as a read only memory, a magnetic or optical disk, and the like. Alternatively, all or part of the steps of the above embodiments may be implemented using one or more integrated circuits. Accordingly, each module/unit in the above embodiments may be implemented in the form of hardware, and may also be implemented in the form of a software functional module. The present disclosure is not limited to any specific form of combination of hardware and software.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The foregoing is illustrative of the present disclosure and is not to be construed as limiting thereof. Although a few exemplary embodiments of this disclosure have been described, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of this disclosure. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the claims. It is to be understood that the foregoing is illustrative of the present disclosure and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed embodiments, as well as other embodiments, are intended to be included within the scope of the appended claims. The present disclosure is defined by the claims and their equivalents.

Claims

1. A data processing method based on data sharing among nodes comprises the following steps:

processing data to be identified by utilizing a hash function to obtain a hash value of the data to be identified;

determining whether the data to be identified is the same as source data corresponding to the stored hash value by comparing the hash value of the data to be identified with the hash value stored in a chained data structure of the current node, wherein the hash value stored in the chained data structure is obtained by processing the source data by using the hash function; and

determining to update the chained data structure of the current node and sending chained data structure update information to other nodes under the condition that the data to be identified is different from the source data corresponding to the stored hash value,

wherein the chained data structure update message is used to update the chained data structure of the other node,

wherein updating the chained data structure of the current node comprises:

creating a data block based on the hash value of the data to be identified and adding the created data block to the chained data structure of the current node to update the chained data structure, and

wherein sending the chained data structure update message to the other node comprises:

obtaining an identity of the other node, sending a chained data structure update message to the other node based on the identity, wherein,

the created data block is included in the chained data structure update message,

and the created data block is used for being checked by the other nodes, and the chained data structures of the other nodes are updated under the condition that the data block passes the check.

2. The method of claim 1, further comprising: and shielding the data to be identified under the condition that the data to be identified is the same as the source data corresponding to the stored hash value.

3. The method of claim 1, wherein determining whether the data to be identified is the same as the source data corresponding to the stored hash value comprises:

under the condition that the hash value of the data to be identified is the same as the hash value stored in the chained data structure, determining that the data to be identified is the same as the source data corresponding to the stored hash value; and

and under the condition that the hash value of the data to be identified is different from the hash value stored in the chained data structure, determining that the data to be identified is different from the source data corresponding to the stored hash value.

4. The method of claim 1, wherein the chained data structure is comprised of at least one data block, each of the at least one data block comprising:

a chunk body to store data, wherein the data includes a hash value; and

a block header for storing characteristic information of the data block, wherein the characteristic information includes a characteristic value, a version number, a timestamp, and a difficulty value of the data.

5. The method of claim 1, wherein processing the data to be identified by using a hash function to obtain the hash value of the data to be identified comprises:

processing a data stream of data to be identified by using a hash function to obtain a hash value of the data to be identified; and

and the hash value stored in the chained data structure is obtained by processing the data stream of the source data by using the hash function.

6. The method of claim 5, wherein processing the data to be identified by using a hash function to obtain the hash value of the data to be identified further comprises:

intercepting at least one part of data flow in the data flow, and processing the at least one part of data flow by utilizing a hash function to obtain a hash value.

7. The method of claim 5, wherein processing the data to be identified by using a hash function to obtain the hash value of the data to be identified further comprises:

converting the data stream into floating point numbers;

squaring the floating point number to obtain a square result;

and processing at least one part of the square result by utilizing the hash function to obtain a hash value.

8. The method of claim 1, the source data comprising at least one of a picture, a link, and a video.

9. The method of claim 1, further comprising: receiving a chained data structure update message from other nodes, wherein the chained data structure update message includes a data block, and the method further includes: and checking the data block, and updating the chained data structure of the current node under the condition of passing the check.

10. A data processing apparatus based on inter-node data sharing, comprising:

the processing unit is configured to process data to be identified by utilizing a hash function to obtain a hash value of the data to be identified;

the determining unit is configured to determine whether the data to be identified is the same as source data corresponding to the stored hash value by comparing the hash value of the data to be identified with the hash value stored in a chained data structure of the current node, wherein the hash value stored in the chained data structure is obtained by processing the source data by using the hash function; and

a transmission unit configured to determine to update the chained data structure of the current node and send chained data structure update messages to other nodes if the data to be identified is not the same as the source data corresponding to the stored hash value,

wherein the transmission unit is further configured to:

creating a data block based on the hash value of the data to be identified and adding the created data block to the chained data structure of the current node to update the chained data structure, an

11. The apparatus according to claim 10, further comprising a masking unit configured to mask the data to be identified if the data to be identified is the same as the source data corresponding to the stored hash value.

12. A data processing apparatus based on inter-node data sharing, comprising:

one or more processors; and

one or more memories, wherein the memory has stored therein computer readable code, which when executed by the one or more processors, performs the method of data processing based on inter-node data sharing of any one of claims 1-9.

13. A computer-readable storage medium having stored thereon instructions, which, when executed by a processor, cause the processor to execute the inter-node data sharing-based data processing method according to any one of claims 1 to 9.