CN114866334A - Data fusion processing method and device - Google Patents
Data fusion processing method and device Download PDFInfo
- Publication number
- CN114866334A CN114866334A CN202210646120.3A CN202210646120A CN114866334A CN 114866334 A CN114866334 A CN 114866334A CN 202210646120 A CN202210646120 A CN 202210646120A CN 114866334 A CN114866334 A CN 114866334A
- Authority
- CN
- China
- Prior art keywords
- data
- partner
- fusion
- data fusion
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000007499 fusion processing Methods 0.000 title claims abstract description 55
- 230000004927 fusion Effects 0.000 claims abstract description 111
- 238000004364 calculation method Methods 0.000 claims abstract description 61
- 239000012634 fragment Substances 0.000 claims abstract description 58
- 238000012545 processing Methods 0.000 claims abstract description 41
- 238000013467 fragmentation Methods 0.000 claims abstract description 22
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 22
- 238000004590 computer program Methods 0.000 claims description 15
- 230000002159 abnormal effect Effects 0.000 claims description 13
- 238000012544 monitoring process Methods 0.000 claims description 13
- 238000005520 cutting process Methods 0.000 claims description 10
- 238000003860 storage Methods 0.000 claims description 8
- 238000010586 diagram Methods 0.000 description 14
- 238000007726 management method Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 5
- 238000012546 transfer Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000593 degrading effect Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
Abstract
The invention provides a data fusion processing method and device, relates to the technical field of data processing, and can be used in the financial field or other technical fields. The method comprises the following steps: receiving a data fusion request sent by a data partner; determining a target node for executing calculation of a corresponding data file according to the data partner identifier; respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the residual data fragments to the target nodes of the residual data partners; and summarizing the calculation results executed by all target nodes to obtain a data fusion result, and sending the data fusion result to the data partner. The device performs the above method. The data fusion processing method and device provided by the embodiment of the invention are convenient for managing and controlling the distributed and deployed data files and can improve the efficiency of fusion calculation.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data fusion processing method and device.
Background
In order to enhance data privacy protection when multi-party data cooperates, a technical means of data distributed deployment is mostly used.
The distributed deployment has the problem of difficult management and control when fusion calculation is needed.
Disclosure of Invention
For solving the problems in the prior art, embodiments of the present invention provide a data fusion processing method and apparatus, which can at least partially solve the problems in the prior art.
In one aspect, the present invention provides a data fusion processing method, including:
receiving a data fusion request sent by a data partner, wherein the data fusion request carries a data partner identifier and a data file;
determining a target node for executing calculation of a corresponding data file according to the data partner identifier;
respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the residual data fragments to the target nodes of the residual data partners;
and summarizing the calculation results executed by all target nodes to obtain a data fusion result, and sending the data fusion result to the data partner.
The data file carried by the data fusion request is an encrypted data file; correspondingly, before the step of performing slicing processing on each data file, the method further includes:
and decrypting the encrypted data files, and respectively carrying out fragment processing on each decrypted data file.
Wherein, the determining a target node for executing the calculation of the corresponding data file according to the data partner identifier comprises:
determining a target node for executing calculation of a corresponding data file according to the data partner identifier by using a preset rule; the preset rule comprises traversing each node in the data partner, and determining the target node according to the calculation performance index of each node.
Wherein the distributing the remaining data fragments to the target nodes of the remaining data partners comprises:
and uniformly distributing the residual data fragments to target nodes of the residual data partners.
The data fusion processing method further comprises the following steps:
carrying out liquidity monitoring on a target data file with a preset data type;
and if the target data file is determined to have outflow, cutting off the network connection.
The data fusion processing method further comprises the following steps:
and generating a prompt message for prompting that the target data file outflows while cutting off the network connection.
The data fusion processing method further comprises the following steps:
monitoring the availability of the target node;
and if the target node is determined to have the abnormal availability, isolating the target node with the abnormal availability.
In one aspect, the present invention provides a data fusion processing apparatus, including:
the data fusion device comprises a receiving unit, a processing unit and a processing unit, wherein the receiving unit is used for receiving a data fusion request sent by a data partner, and the data fusion request carries a data partner identifier and a data file;
the determining unit is used for determining a target node for executing calculation of the corresponding data file according to the data partner identification;
the fragmentation unit is used for respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the rest data fragments to the target nodes of the rest data partners;
and the fusion unit is used for summarizing calculation results executed by all target nodes to obtain a data fusion result and sending the data fusion result to the data partner.
In another aspect, an embodiment of the present invention provides an electronic device, including: a processor, a memory, and a bus, wherein,
the processor and the memory are communicated with each other through the bus;
the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform a method comprising:
receiving a data fusion request sent by a data partner, wherein the data fusion request carries a data partner identifier and a data file;
determining a target node for executing calculation of a corresponding data file according to the data partner identifier;
respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the residual data fragments to the target nodes of the residual data partners;
and summarizing the calculation results executed by all target nodes to obtain a data fusion result, and sending the data fusion result to the data partner.
An embodiment of the present invention provides a non-transitory computer-readable storage medium, including:
the non-transitory computer readable storage medium stores computer instructions that cause the computer to perform a method comprising:
receiving a data fusion request sent by a data partner, wherein the data fusion request carries a data partner identifier and a data file;
determining a target node for executing calculation of a corresponding data file according to the data partner identifier;
respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the residual data fragments to the target nodes of the residual data partners;
and summarizing the calculation results executed by all target nodes to obtain a data fusion result, and sending the data fusion result to the data partner.
The data fusion processing method and device provided by the embodiment of the invention receive a data fusion request sent by a data partner, wherein the data fusion request carries a data partner identifier and a data file; determining a target node for executing calculation of a corresponding data file according to the data partner identifier; respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the residual data fragments to the target nodes of the residual data partners; the calculation results executed by all the target nodes are collected to obtain a data fusion result, and the data fusion result is sent to the data partner, so that the distributed and deployed data files are conveniently controlled, and the efficiency of fusion calculation can be improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
fig. 1 is a schematic flow chart of a data fusion processing method according to an embodiment of the present invention.
Fig. 2 is a schematic structural diagram of modularization of the data fusion processing method provided by the embodiment of the present invention.
Fig. 3 is a schematic flowchart of a data fusion processing method according to another embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a data fusion processing apparatus according to an embodiment of the present invention.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.
Fig. 1 is a schematic flow diagram of a data fusion processing method according to an embodiment of the present invention, and as shown in fig. 1, the data fusion processing method according to the embodiment of the present invention includes:
step S1: and receiving a data fusion request sent by a data partner, wherein the data fusion request carries a data partner identifier and a data file.
Step S2: and determining a target node for executing calculation of the corresponding data file according to the data partner identification.
Step S3: and respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the rest data fragments to the target nodes of the rest data partners.
Step S4: and summarizing the calculation results executed by all target nodes to obtain a data fusion result, and sending the data fusion result to the data partner.
In the step S1, the device receives a data fusion request sent by a data partner, where the data fusion request carries a data partner identifier and a data file. The apparatus may be a computer device that performs the method, and may include, for example, a server that centralizes the deployment. It should be noted that the embodiments of the present invention relate to the acquisition and analysis of user data being authorized by the user.
As shown in fig. 2, the data partners may include a party a and a party B. The A side has a corresponding data transfer unit 102 and a data fusion unit 103 (located on the left side of the data fusion computing platform); the B side has a corresponding data transfer unit 102 and a data fusion unit 103 (located on the right side of the data fusion computing platform).
The description of each module is as follows:
the data transmission service unit 101 belongs to the partner components, is respectively deployed at each data partner, and is responsible for encrypting the own data, sending the encrypted own data to the data fusion computing platform, initiating a data fusion request to the management and control unit 105, and receiving the returned data fusion computing result.
The data transfer unit 102 belongs to a data fusion computing platform component, is deployed at a centralized deployment party (which may be an a party, a B party or an independent party, the same below), is responsible for decrypting transmitted data, may not perform data storage, divides a data plaintext into a plurality of pieces, and reserves at least one piece of data to a target node in the data fusion unit of the local party. The data fusion unit may include a plurality of nodes, one node holding at least one piece of data being a target node.
And the residual data fragments are sent to the target nodes in the data fusion unit of the residual data partner. The unit server belongs to each data partner and is managed in a centralized mode, the management right of the unit server belongs to each data partner, a complex password needs to be set, strict log recording and audit need to be carried out, network data message flow monitoring needs to be started, network connection can be immediately and automatically cut off through a specific data identifier if outflow of sensitive data is found, and related data partners are notified in a mail or short message mode.
The data fusion unit 103 is a computing component belonging to a data fusion computing platform, is deployed in a centralized deployment party, and is used for jointly processing computing services of each data partner.
The calculation rule unit 104, which belongs to an algorithm component of a data fusion calculation platform, is deployed in a centralized deployment party and is mainly used for calculation rule management, and calculation rules of tasks can be determined by negotiation with each data partner in advance.
The management and control unit 105, which belongs to a management and control component of a data fusion computing platform, is deployed in a centralized deployment party, and is used for monitoring the availability of a target node in the data fusion unit 103, isolating an abnormal node, fusing and degrading the platform, limiting the current of a task, scheduling the task (the task can be evenly distributed to an optimal node in the data fusion unit 103 meeting conditions according to the identification and weight indexes such as the number of executed tasks, the number of queued tasks, a CPU (Central processing Unit), a memory and the like), and returning a task result.
The data fusion request sent by the data partner may be received synchronously, or may be received asynchronously, as shown in fig. 3.
In the above step S2, the apparatus determines the target node for performing the calculation of the corresponding data file according to the data partner identity. As shown in fig. 2, the corresponding data fusion unit 103 may be determined according to the data partner identifier, and nodes in the data fusion unit 103 are traversed, and one node is selected as a target node for calculating the corresponding data file.
The determining a target node for executing the calculation of the corresponding data file according to the data partner identifier includes:
determining a target node for executing calculation of a corresponding data file according to the data partner identifier by using a preset rule; the preset rule comprises traversing each node in the data partner, and determining the target node according to the calculation performance index of each node. The preset rules can be configured autonomously according to actual conditions.
The calculation performance index can be selected independently according to actual conditions, the indexes corresponding to the CPU and the memory can be selected as the calculation performance index, and the node corresponding to the calculation performance index with the optimal value is selected as the target node.
In step S3, the apparatus performs fragmentation processing on each data file, retains at least one data fragment to the target node, and distributes the remaining data fragments to the target node of the remaining data partner. Taking data partner a, data partner B, and data partner C as an example, the following is illustrated:
for data partner a: and (3) carrying out fragmentation processing on the data file corresponding to the A to obtain 10 data fragments, keeping 1 of the data fragments in a target node in the data fusion unit 103 of the A, and distributing the remaining 9 data fragments to a target node of the B and a target node of the C. In order to ensure distributed computing, it is necessary that both the target node of B and the target node of C can be allocated to at least one data slice, and the target node of B and the target node of C are allocated 9 data slices in total. The rules of the data fragment can be flexibly configured through the management and control unit 105.
The distributing the remaining data fragments to the target nodes of the remaining data partners comprises:
and uniformly distributing the residual data fragments to target nodes of the residual data partners. Referring to the above example, 4 data fragments out of 9 data fragments may be distributed to the target node of B, and the remaining 5 data fragments may be distributed to the target node of C.
The data file carried by the data fusion request is an encrypted data file; correspondingly, before the step of performing fragment processing on each data file, the method further includes:
and decrypting the encrypted data files, and respectively carrying out fragment processing on each decrypted data file. Referring to fig. 3, that is, the data file may be encrypted in step 100 to obtain an encrypted data file, and the encryption method is not particularly limited in the embodiment of the present invention. Referring to fig. 3, in step 104, the encrypted data file may be decrypted to obtain plaintext data, and then the plaintext data may be sliced.
In step S4, the device summarizes the calculation results performed by all the target nodes to obtain a data fusion result, and sends the data fusion result to the data partner. Referring to fig. 2, the calculation results of the target nodes in the data fusion units 103 may be summarized to the management and control unit 105, and the data fusion results may be sent to the data partners through the management and control unit 105.
The data fusion processing method further comprises the following steps:
carrying out liquidity monitoring on a target data file with a preset data type; the preset data type can be set autonomously according to actual conditions and is used for marking the sensitive data.
And if the target data file is determined to have outflow, cutting off the network connection. And generating a prompt message for prompting that the target data file flows outwards while cutting off the network connection, and reminding related workers to take corresponding measures in time through the prompt message.
The data fusion processing method further comprises the following steps:
monitoring the availability of the target node; the availability monitoring may be performed on the target node in the data fusion unit 103 in each data partner.
And if the target node is determined to have the abnormal availability, isolating the target node with the abnormal availability. Further, a prompt message for prompting that the target node has an abnormal availability can be generated, and related workers can be reminded to take corresponding measures in time through the prompt message.
And the standby node can be started to replace the target node with the abnormal availability as a new target node.
The data fusion processing method provided by the embodiment of the invention receives a data fusion request sent by a data partner, wherein the data fusion request carries a data partner identifier and a data file; determining a target node for executing calculation of a corresponding data file according to the data partner identifier; respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the residual data fragments to the target nodes of the residual data partners; the calculation results executed by all the target nodes are collected to obtain a data fusion result, and the data fusion result is sent to the data partner, so that the distributed and deployed data files are conveniently controlled, and the efficiency of fusion calculation can be improved.
Further, the data file carried by the data fusion request is an encrypted data file; correspondingly, before the step of performing slicing processing on each data file, the method further includes:
and decrypting the encrypted data files, and respectively carrying out fragment processing on each decrypted data file. Reference is made to the above description and no further description is made.
The data fusion processing method provided by the embodiment of the invention can further ensure the information security of the data file.
Further, the determining a target node for performing computation of a corresponding data file according to the data partner identifier includes:
determining a target node for executing calculation of a corresponding data file according to the data partner identifier by using a preset rule; the preset rule comprises traversing each node in the data partner, and determining the target node according to the calculation performance index of each node. Reference is made to the above description and no further description is made.
The data fusion processing method provided by the embodiment of the invention can further improve the efficiency of fusion calculation.
Further, the distributing the remaining data fragments to the target nodes of the remaining data partners includes:
and uniformly distributing the residual data fragments to target nodes of the residual data partners. Reference is made to the above description and no further description is made.
The data fusion processing method provided by the embodiment of the invention can further improve the efficiency of fusion calculation.
Further, the data fusion processing method further includes:
carrying out liquidity monitoring on a target data file with a preset data type; reference is made to the above description and no further description is made.
And if the target data file is determined to have outflow, cutting off the network connection. Reference is made to the above description and no further description is made.
The data fusion processing method provided by the embodiment of the invention can further strengthen the control of the outflow of the data file.
Further, the data fusion processing method further includes:
and generating a prompt message for prompting that the target data file outflows while cutting off the network connection. Reference is made to the above description and no further description is made.
The data fusion processing method provided by the embodiment of the invention is beneficial to timely handling the condition of outflow of the data files.
Further, the data fusion processing method further includes:
monitoring the availability of the target node; reference is made to the above description and no further description is made.
And if the target node is determined to have the abnormal availability, isolating the target node with the abnormal availability. Reference is made to the above description and no further description is made.
The data fusion processing method provided by the embodiment of the invention can further enhance the exception control of the target node.
It should be noted that the data fusion processing method provided in the embodiment of the present invention may be used in the financial field, and may also be used in any technical field other than the financial field.
Fig. 4 is a schematic structural diagram of a data fusion processing apparatus according to an embodiment of the present invention, and as shown in fig. 4, the data fusion processing apparatus according to the embodiment of the present invention includes a receiving unit 401, a determining unit 402, a slicing unit 403, and a fusing unit 404, where:
the receiving unit 401 is configured to receive a data fusion request sent by a data partner, where the data fusion request carries a data partner identifier and a data file; the determining unit 402 is configured to determine, according to the data partner identifier, a target node for performing computation on a corresponding data file; the fragmentation unit 403 is configured to perform fragmentation processing on each data file, reserve at least one data fragment to the target node, and distribute the remaining data fragments to the target node of the remaining data partner; the fusion unit 404 is configured to summarize calculation results executed by all target nodes to obtain a data fusion result, and send the data fusion result to the data partner.
Specifically, a receiving unit 401 in the device is configured to receive a data fusion request sent by a data partner, where the data fusion request carries a data partner identifier and a data file; the determining unit 402 is configured to determine, according to the data partner identifier, a target node for performing computation on a corresponding data file; the fragmentation unit 403 is configured to perform fragmentation processing on each data file, reserve at least one data fragment to the target node, and distribute the remaining data fragments to the target nodes of the remaining data partners; the fusion unit 404 is configured to summarize calculation results executed by all target nodes to obtain a data fusion result, and send the data fusion result to the data partner.
The data fusion processing device provided by the embodiment of the invention receives a data fusion request sent by a data partner, wherein the data fusion request carries a data partner identifier and a data file; determining a target node for executing calculation of a corresponding data file according to the data partner identifier; respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the residual data fragments to the target nodes of the residual data partners; the calculation results executed by all the target nodes are collected to obtain a data fusion result, and the data fusion result is sent to the data partner, so that the distributed and deployed data files are conveniently controlled, and the efficiency of fusion calculation can be improved.
Further, the data file carried by the data fusion request is an encrypted data file; correspondingly, before the step of performing fragment processing on each data file, the data fusion processing apparatus is further configured to:
and decrypting the encrypted data files, and respectively carrying out fragment processing on each decrypted data file.
The data fusion processing device provided by the embodiment of the invention can further ensure the information security of the data file.
Further, the determining unit 402 is specifically configured to:
determining a target node for executing calculation of a corresponding data file according to the data partner identifier by using a preset rule; the preset rule comprises traversing each node in the data partner, and determining the target node according to the calculation performance index of each node.
The data fusion processing device provided by the embodiment of the invention can further improve the efficiency of fusion calculation.
Further, the determining unit 402 is further specifically configured to:
and uniformly distributing the residual data fragments to target nodes of the residual data partners.
The data fusion processing device provided by the embodiment of the invention can further improve the efficiency of fusion calculation.
Further, the data fusion processing device is further configured to:
carrying out liquidity monitoring on a target data file with a preset data type;
and if the target data file is determined to have outflow, cutting off the network connection.
The data fusion processing device provided by the embodiment of the invention can further strengthen the control of the outflow of the data file.
Further, the data fusion processing device is further configured to:
and generating a prompt message for prompting that the target data file outflows while cutting off the network connection.
The data fusion processing device provided by the embodiment of the invention is beneficial to timely dealing with the situation that the data file outflows.
Further, the data fusion processing device is further configured to:
monitoring the availability of the target node;
and if the target node is determined to have the abnormal availability, isolating the target node with the abnormal availability.
The data fusion processing device provided by the embodiment of the invention can further enhance the exception control of the target node.
The embodiment of the data fusion processing apparatus provided in the embodiment of the present invention may be specifically configured to execute the processing flows of the above method embodiments, and the functions of the embodiment are not described herein again, and refer to the detailed description of the above method embodiments.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 5, the electronic device includes: a processor (processor)501, a memory (memory)502, and a bus 503;
the processor 501 and the memory 502 complete communication with each other through a bus 503;
the processor 501 is configured to call program instructions in the memory 502 to perform the methods provided by the above-mentioned method embodiments, for example, including:
receiving a data fusion request sent by a data partner, wherein the data fusion request carries a data partner identifier and a data file;
determining a target node for executing calculation of a corresponding data file according to the data partner identifier;
respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the residual data fragments to the target nodes of the residual data partners;
and summarizing the calculation results executed by all target nodes to obtain a data fusion result, and sending the data fusion result to the data partner.
The present embodiment discloses a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method provided by the above method embodiments, for example, including:
receiving a data fusion request sent by a data partner, wherein the data fusion request carries a data partner identifier and a data file;
determining a target node for executing calculation of a corresponding data file according to the data partner identifier;
respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the residual data fragments to the target nodes of the residual data partners;
and summarizing the calculation results executed by all target nodes to obtain a data fusion result, and sending the data fusion result to the data partner.
The present embodiment provides a computer-readable storage medium, which stores a computer program, where the computer program causes the computer to execute the method provided by the above method embodiments, for example, the method includes:
receiving a data fusion request sent by a data partner, wherein the data fusion request carries a data partner identifier and a data file;
determining a target node for executing calculation of a corresponding data file according to the data partner identifier;
respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the rest data fragments to the target node of the rest data partner;
and summarizing the calculation results executed by all target nodes to obtain a data fusion result, and sending the data fusion result to the data partner.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In the description herein, reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," "an example," "a particular example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A data fusion processing method is characterized by comprising the following steps:
receiving a data fusion request sent by a data partner, wherein the data fusion request carries a data partner identifier and a data file;
determining a target node for executing calculation of a corresponding data file according to the data partner identifier;
respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the residual data fragments to the target nodes of the residual data partners;
and summarizing the calculation results executed by all target nodes to obtain a data fusion result, and sending the data fusion result to the data partner.
2. The data fusion processing method according to claim 1, wherein the data file carried by the data fusion request is an encrypted data file; correspondingly, before the step of performing slicing processing on each data file, the method further includes:
and decrypting the encrypted data files, and respectively carrying out fragment processing on each decrypted data file.
3. The data fusion processing method of claim 1, wherein determining a target node for performing computation of a corresponding data file according to the data partner identity comprises:
determining a target node for executing calculation of a corresponding data file according to the data partner identifier by using a preset rule; the preset rule comprises traversing each node in the data partner, and determining the target node according to the calculation performance index of each node.
4. The data fusion processing method of claim 3, wherein the distributing of the remaining data fragments to the target nodes of the remaining data partners comprises:
and uniformly distributing the residual data fragments to target nodes of the residual data partners.
5. The data fusion processing method according to any one of claims 1 to 4, characterized in that the data fusion processing method further comprises:
carrying out liquidity monitoring on a target data file with a preset data type;
and if the target data file is determined to have outflow, cutting off the network connection.
6. The data fusion processing method according to claim 5, further comprising:
and generating a prompt message for prompting that the target data file outflows while cutting off the network connection.
7. The data fusion processing method according to any one of claims 1 to 4, characterized in that the data fusion processing method further comprises:
monitoring the availability of the target node;
and if the target node is determined to have the abnormal availability, isolating the target node with the abnormal availability.
8. A data fusion processing apparatus, comprising:
the data fusion device comprises a receiving unit, a processing unit and a processing unit, wherein the receiving unit is used for receiving a data fusion request sent by a data partner, and the data fusion request carries a data partner identifier and a data file;
the determining unit is used for determining a target node for executing calculation of the corresponding data file according to the data partner identification;
the fragmentation unit is used for respectively carrying out fragmentation processing on each data file, reserving at least one data fragment to the target node, and distributing the rest data fragments to the target nodes of the rest data partners;
and the fusion unit is used for summarizing calculation results executed by all target nodes to obtain a data fusion result and sending the data fusion result to the data partner.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210646120.3A CN114866334B (en) | 2022-06-09 | 2022-06-09 | Data fusion processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210646120.3A CN114866334B (en) | 2022-06-09 | 2022-06-09 | Data fusion processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114866334A true CN114866334A (en) | 2022-08-05 |
| CN114866334B CN114866334B (en) | 2023-11-24 |
Family
ID=82623856
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210646120.3A Active CN114866334B (en) | 2022-06-09 | 2022-06-09 | Data fusion processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114866334B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108337122A (en) * | 2018-02-22 | 2018-07-27 | 深圳市脉山龙信息技术股份有限公司 | The operation management system calculated based on distributed stream |
| CN111290841A (en) * | 2018-12-10 | 2020-06-16 | 北京沃东天骏信息技术有限公司 | Task scheduling method and device, computing equipment and storage medium |
| CN111381972A (en) * | 2018-12-27 | 2020-07-07 | 北京奇虎科技有限公司 | Distributed task scheduling method, device and system |
| CN113010382A (en) * | 2021-03-16 | 2021-06-22 | 杭州数澜科技有限公司 | Buried point data calculation method and device, storage medium and electronic equipment |
| CN113722755A (en) * | 2021-08-25 | 2021-11-30 | 上海阵方科技有限公司 | Data processing system, method, device and equipment for realizing privacy protection |
-
2022
- 2022-06-09 CN CN202210646120.3A patent/CN114866334B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108337122A (en) * | 2018-02-22 | 2018-07-27 | 深圳市脉山龙信息技术股份有限公司 | The operation management system calculated based on distributed stream |
| CN111290841A (en) * | 2018-12-10 | 2020-06-16 | 北京沃东天骏信息技术有限公司 | Task scheduling method and device, computing equipment and storage medium |
| CN111381972A (en) * | 2018-12-27 | 2020-07-07 | 北京奇虎科技有限公司 | Distributed task scheduling method, device and system |
| CN113010382A (en) * | 2021-03-16 | 2021-06-22 | 杭州数澜科技有限公司 | Buried point data calculation method and device, storage medium and electronic equipment |
| CN113722755A (en) * | 2021-08-25 | 2021-11-30 | 上海阵方科技有限公司 | Data processing system, method, device and equipment for realizing privacy protection |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114866334B (en) | 2023-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200104223A1 (en) | Distributed data monitoring device | |
| CN112835977B (en) | Database management method and system based on block chain | |
| CN110289965B (en) | Application program service management method and device | |
| CN102281141B (en) | Document permission management method, apparatus and system | |
| CN113900598B (en) | Data storage method, device, equipment and storage medium based on block chain | |
| CN107111510B (en) | Method and device for operating VNF packet | |
| CN111510474A (en) | Data transmission method based on message middleware and related equipment | |
| CN113032357A (en) | File storage method and device and server | |
| CN109510840A (en) | Sharing method, device, computer equipment and the storage medium of unstructured data | |
| CN107423942B (en) | Service transfer method and device | |
| CN111753324A (en) | Private data processing method, private data computing method and applicable equipment | |
| CN113901505A (en) | Data sharing method and device, electronic equipment and storage medium | |
| CN116418876A (en) | Migration method and system of computing power network service and cloud management platform | |
| CN114780982A (en) | Flow business circulation method, device and system | |
| CN111460038A (en) | Quasi-real-time data synchronization method and device | |
| CN117149884B (en) | Data processing transaction method | |
| CN107465717B (en) | Password on-demand service method, device and equipment | |
| US9639414B1 (en) | Remote real-time storage system monitoring and management | |
| US20130297821A1 (en) | Client based congestion management | |
| CN114866334B (en) | Data fusion processing method and device | |
| WO2016122686A1 (en) | Authentication for multiple user data storage and separation | |
| CN116647567A (en) | Privacy protection set intersection method and device | |
| CN108540374B (en) | Information processing method, device, equipment and storage medium based on instant messaging | |
| CN112699391B (en) | Target data sending method and privacy computing platform | |
| CN115176452B (en) | Method and system for data management in a communication network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |