WO2022183644A1 - 数据处理方法、装置、电子设备及存储介质 - Google Patents

数据处理方法、装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2022183644A1
WO2022183644A1 PCT/CN2021/103669 CN2021103669W WO2022183644A1 WO 2022183644 A1 WO2022183644 A1 WO 2022183644A1 CN 2021103669 W CN2021103669 W CN 2021103669W WO 2022183644 A1 WO2022183644 A1 WO 2022183644A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
processed
party
computing
processing tool
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/103669
Other languages
English (en)
French (fr)
Inventor
季石磊
黄海平
王建奎
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Baidu Online Network Technology Beijing Co Ltd
Original Assignee
Baidu Online Network Technology Beijing Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Baidu Online Network Technology Beijing Co Ltd filed Critical Baidu Online Network Technology Beijing Co Ltd
Publication of WO2022183644A1 publication Critical patent/WO2022183644A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/907—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/908—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/211—Schema design and management
    • G06F16/212—Schema design and management with details for data modelling support
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database

Definitions

  • the present disclosure relates to the technical field of data processing, such as federated learning technology, and in particular, to a data processing method, apparatus, electronic device, and storage medium.
  • Data is an important means of production. In order to mine data value, it is necessary to conduct joint analysis or joint modeling of data from different institutions to generate new data value and create new business forms. Considering regulatory requirements such as data privacy and data security, cross-agency data cooperation will face many new challenges.
  • FIG. 1 is a schematic structural diagram of a cross-institutional data cooperation manner in the related art.
  • method 1 vertical federation XXY type
  • one party provides data X1, X2, and the other party provides data X3, X4 and Y.
  • Mode 2 Vertical federation XY mode
  • one party provides data X
  • the other party provides data Y.
  • Method 3 Controlled isolation, one party provides data, the other party does not provide data, and only analyzes the data.
  • the first and second methods above are FL (Federated Learning) schemes, which use gradient exchange, MPC (Secure multi-Party Computation, multi-party secure computing) and TEE (Trusted Execution Environment, Trusted Execution Environment), etc.;
  • the solution is not particularly efficient in the context of cross-agency cooperation.
  • the above-mentioned method 3 is an isolated domain computing solution. In the case that the two parties cannot trust each other, in order to perform data security fusion, find an independent and trusted third-party environment, and build an isolated domain environment based on network security technology. Both parties put their own data into the isolation domain, and both parties can monitor and audit data operations to ensure that all behaviors of data analysts comply with the principles pre-agreed by both parties; in the scenario of cross-agency cooperation, Nor is it particularly efficient.
  • the present disclosure provides a data processing method, device, electronic device and storage medium, which can more efficiently realize data cooperation across institutions, realize the safety and controllability of data cooperation, save the cost of data cooperation, and avoid waste of resources.
  • a data processing method applied to a computing party, the method comprising:
  • the data analysis logic of the data to be processed is generated on the federated computing agent of the computing party;
  • Another data processing method is provided, applied to a data party, the method comprising:
  • the data analysis logic based on the data to be processed calculates the result output corresponding to the data to be processed under at least one data processing tool; wherein the result output includes: result data and output log.
  • a data processing device comprising: a data acquisition module, a logic generation module and a logic transmission module; wherein,
  • the data acquisition module is configured to acquire metadata of the data to be processed from the data party through the federated computing agent of the data party; wherein, the metadata represents the characteristics of the data to be processed;
  • the logic generation module is configured to generate the data analysis logic of the data to be processed on the federated computing agent of the computing party according to the metadata of the data to be processed;
  • the logic sending module is configured to send the data analysis logic of the data to be processed to the data party; so that the data party calculates the data analysis logic of the data to be processed based on the data analysis logic of the data to be processed.
  • another data processing device includes: a data sending module, a logic receiving module, and a result computing module; wherein,
  • the data sending module is configured to send metadata of the data to be processed to the computing party through the federated computing agent of the computing party; wherein, the metadata represents the characteristics of the data to be processed;
  • the logic receiving module is configured to receive the data analysis logic of the data to be processed and generated based on the metadata of the data to be processed and sent by the computing party;
  • the result calculation module is configured to calculate the result output corresponding to the to-be-processed data under at least one data processing tool based on the data analysis logic of the to-be-processed data; wherein, the result output includes: result data and product output. log out.
  • an electronic device comprising:
  • the memory is provided with instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the data processing method described above.
  • a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to implement the above-mentioned data processing method.
  • a computer program product comprising a computer program, the computer program implementing the above-mentioned data processing method when executed by a processor.
  • Fig. 1 is the structural schematic diagram of the cross-agency data cooperation mode in the related art
  • FIG. 2 is a schematic flowchart of a data processing method provided by an embodiment of the present disclosure
  • FIG. 3 is a schematic structural diagram of a micro-isolation system provided by an embodiment of the present disclosure.
  • FIG. 4 is a schematic flowchart of another data processing method provided by an embodiment of the present disclosure.
  • FIG. 5 is a schematic flowchart of another data processing method provided by an embodiment of the present disclosure.
  • FIG. 6 is a schematic structural diagram of a data processing apparatus provided by an embodiment of the present disclosure.
  • FIG. 7 is a schematic structural diagram of another data processing apparatus provided by an embodiment of the present disclosure.
  • FIG. 8 is a block diagram of an electronic device for implementing a data processing method provided by an embodiment of the present disclosure.
  • FIG. 2 is a schematic flowchart of a data processing method provided by an embodiment of the present disclosure.
  • the method may be executed by a data processing apparatus or an electronic device of the computing side, and the apparatus or the electronic equipment of the computing side may be performed by software and/or hardware.
  • the device or the electronic device of the computing side can be integrated into any smart device with a network communication function.
  • the data processing method may include the following steps:
  • the computing party can obtain metadata of the data to be processed from the data party through the federated computing agent of the data party; wherein, the metadata represents the characteristics of the data to be processed.
  • the metadata in this embodiment of the present disclosure includes, but is not limited to: the name of the data warehouse, the name of the data table stored in the data warehouse, the field name of the data table, the field type (String, Integer, etc.), and the number of rows.
  • the computing party may generate data analysis logic of the data to be processed on the federated computing agent of the computing party according to the metadata of the data to be processed.
  • the computing party can analyze the received metadata of the data to be processed to obtain the analysis result of the metadata of the data to be processed; and then generate the data analysis logic of the data to be processed according to the analysis result of the metadata of the data to be processed.
  • S203 send the data analysis logic of the data to be processed to the data party; make the data party calculate the result output corresponding to the data to be processed under at least one data processing tool based on the data analysis logic of the data to be processed; wherein, the result output includes: Results data and output logs.
  • the computing side can send the data analysis logic of the data to be processed to the data side; the data side can calculate the result output corresponding to the data to be processed under multiple data processing tools based on the data analysis logic of the data to be processed; wherein , the result output includes: result data and output log.
  • the data party can first analyze the syntax and semantics of the data analysis logic of the data to be processed through the scheduler of the federated computing agent of the data party to obtain at least one data processing tool used by the data analysis logic of the data to be processed; then use the at least one data processing tool.
  • the data processing tool processes the data to be processed, and obtains a result output corresponding to the data to be processed under the at least one data processing tool.
  • FIG. 3 is a schematic structural diagram of a micro-isolation system provided by an embodiment of the present disclosure.
  • the micro-isolation system may include: a computing party (Agent A), a data party (Agent B), and a coordinator (Coordinator); the computing party and the data party transmit and receive data through their respective federated computing agents; the computing party The federated computing proxy of the data party is Proxy A; the federated computing proxy of the data party is Proxy B.
  • the computing party and the data party transmit and receive data with the coordinator through their respective collaborative computing agents.
  • the micro-isolation system in the present disclosure adopts SaaS (Software-as-a-Service, software as a service) and privatized distributed deployment architecture, wherein the coordinator is SaaS deployment, while Agent A and Agent B adopt privatized deployment, When deploying, users only need to deploy their own Agent, and then follow the system guidance to complete the necessary system configuration.
  • SaaS Software-as-a-Service, software as a service
  • Agent A and Agent B adopt privatized deployment
  • the micro-isolation scheme proposed in this disclosure is a new type of isolated domain technology, which does not involve privacy computing technologies such as multi-party secure computing and federated learning.
  • the micro-isolation system provides a unified entry for remote and secure operations on isolated domain data, and realizes unified management of lightweight and fast access to task submission, authorized execution, and authorized result output of isolated domain data through the WEB interface.
  • For the business side it is only necessary to deploy computing nodes for federated computing, and it does not require a lot of business transformation costs to realize multi-party secure collaborative computing on the basis that the data is not out of the domain and the data is available and invisible.
  • the micro-isolation solution in this disclosure is recommended to be implemented together with technologies such as "gatekeeper" to achieve a higher level of data security protection.
  • the data party's sensitive data and data processing facilities are placed in the "gatekeeper", and the "gatekeeper” can ensure that data can only enter and exit.
  • the micro-isolation system is the only way for data from the isolated domain to go out of the domain, so if it needs to go out of the domain, it needs to be authorized and approved by the micro-isolation system.
  • the data processing method proposed by the embodiment of the present disclosure first obtains metadata of the data to be processed from the data party through the federated computing agent of the data party; and then generates the data to be processed on the federated computing agent of the computing party according to the metadata of the data to be processed. Then send the data analysis logic of the data to be processed to the data party; make the data party calculate the corresponding result output of the data to be processed under at least one data processing tool based on the data analysis logic of the data to be processed. That is to say, the computing side in the present disclosure does not need to obtain the sensitive data in the data side. The computing side first generates data analysis logic according to the metadata of the sensitive data, and then the data side can calculate the sensitive data in at least one based on the data analysis logic.
  • the federated learning scheme and the isolated domain computing scheme are not particularly efficient in the scenario of cross-institution cooperation.
  • the present disclosure adopts the technical means that the computing party first generates data analysis logic according to the metadata of the sensitive data, and then the data party calculates the corresponding result output of the sensitive data under at least one data processing tool based on the data analysis logic, which overcomes the problems in the related art.
  • the federated learning scheme and the isolated domain computing scheme are not particularly efficient technical problems in the scenario of cross-institution cooperation.
  • the technical scheme provided by this disclosure can more efficiently realize cross-institution data cooperation, and realize the safety and reliability of data cooperation.
  • the technical solutions of the embodiments of the present disclosure are simple and convenient to implement, easy to popularize, and have a wider application range.
  • FIG. 4 is a schematic flowchart of another data processing method provided by an embodiment of the present disclosure.
  • the method may be executed by a data processing apparatus or an electronic device of a data party, and the apparatus or the electronic device of a data party may be executed by software and/or hardware.
  • the device or the electronic device of the data party can be integrated in any smart device with network communication function.
  • the data processing method may include the following steps:
  • the data party may send metadata of the data to be processed to the computing party through the federated computing agent of the computing party; wherein, the metadata represents the characteristics of the data to be processed.
  • the metadata in this embodiment of the present disclosure includes, but is not limited to: the name of the data warehouse, the name of the data table stored in the data warehouse, the field name of the data table, the field type (String, Integer, etc.), and the number of rows.
  • S402 Receive the data analysis logic of the data to be processed generated based on the metadata of the data to be processed and sent by the computing party.
  • the data party may receive the data analysis logic of the to-be-processed data generated based on the metadata of the to-be-processed data and sent by the computing party.
  • the computing party can analyze the received metadata of the data to be processed to obtain the analysis result of the metadata of the data to be processed; and then generate the data analysis logic of the data to be processed according to the analysis result of the metadata of the data to be processed.
  • the data party can calculate the result output corresponding to the to-be-processed data under at least one data processing tool based on the data analysis logic of the to-be-processed data; wherein, the result output includes: the result data and the output log.
  • the data party can first analyze the syntax and semantics of the data analysis logic of the data to be processed through the scheduler of the federated computing agent of the data party to obtain at least one data processing tool used by the data analysis logic of the data to be processed; then use the at least one data processing tool.
  • the data processing tool processes the data to be processed, and obtains a result output corresponding to the data to be processed under at least one data processing tool.
  • the data party can receive the data transmission instruction sent by the computing party through the pre-deployed micro-isolation system; wherein, the data transmission instruction includes at least an identifier corresponding to the data processing tool; If the approval is passed, the result data corresponding to the data to be processed under at least one data processing tool is transmitted to the computing party through a pre-created file transmission channel.
  • the data party can also receive the log viewing application sent by the computing party through the pre-deployed micro-isolation system; wherein, the log viewing application includes at least one identifier corresponding to the data processing tool; if the data party approves the log viewing application, the pre- The set application program interface provides the computing party with the output log corresponding to the data to be processed under at least one data processing tool.
  • the data party sends the metadata of the data to be processed to the computing party through the federated computing agent of the computing party; and then receives the data of the data to be processed generated based on the metadata of the data to be processed sent by the computing party.
  • Analysis logic based on the data analysis logic of the data to be processed, the corresponding result output of the data to be processed under at least one data processing tool is calculated. That is to say, the computing side in the present disclosure does not need to obtain the sensitive data in the data side.
  • the computing side first generates data analysis logic according to the metadata of the sensitive data, and then the data side can calculate the sensitive data in at least one based on the data analysis logic. The result output corresponding to the data processing tool.
  • the federated learning scheme and the isolated domain computing scheme are not particularly efficient in the scenario of cross-institution cooperation.
  • the present disclosure adopts the technical means that the computing party first generates data analysis logic according to the metadata of the sensitive data, and then the data party calculates the corresponding result output of the sensitive data under at least one data processing tool based on the data analysis logic, which overcomes the problems in the related art.
  • the federated learning scheme and the isolated domain computing scheme are not particularly efficient technical problems in the scenario of cross-institution cooperation.
  • the technical scheme provided by this disclosure can more efficiently realize cross-institution data cooperation, and realize the safety and reliability of data cooperation.
  • the technical solutions of the embodiments of the present disclosure are simple and convenient to implement, easy to popularize, and have a wider application range.
  • FIG. 5 is a schematic flowchart of another data processing method provided by an embodiment of the present disclosure. The description is based on the above technical solutions, and can be combined with the above optional embodiments. As shown in Figure 5, the data processing method may include the following steps:
  • S502 Receive the data analysis logic of the data to be processed generated based on the metadata of the data to be processed and sent by the computing party.
  • the data party can analyze the syntax and semantics of the data analysis logic of the data to be processed through the scheduler of the data party's federated computing agent to obtain at least one data processing tool used by the data analysis logic of the data to be processed.
  • the data party can also approve the data analysis logic of the data to be processed through the approval unit of the data party. If the data analysis logic of the data is approved, the data analysis logic of the data to be processed is submitted to the scheduler.
  • the data party (Agent B) accesses its own data to the micro-isolation system through the federated computing agent, and converts the metadata of the data to be processed (for example: which fields in the data table, each field The data type is String or Integer, etc.) to tell the calculator (Agent A); the calculator (Agent A) refers to the metadata of the data to be processed sent by the data party (Agent B), and writes the data analysis logic on its own federated computing agent , and submit the data analysis logic through the federated computing agent.
  • the data party (Agent B) reviews the data analysis logic of the computing party (Agent A).
  • the data analysis logic is handed over to the scheduler (Scheduler) of the federated computing agent for analysis and scheduling.
  • the scheduler analyzes the syntax and semantics of the data analysis logic, analyzes the data processing tools (Hadoop, Spark, TensorFlow, etc.) used by the data analysis logic, and automatically invokes the data processing tools involved according to the context semantics of the data analysis logic , to analyze and process the DB (Data Base, database) involved in the data analysis logic.
  • DB Data Base, database
  • the result output of the data party (Agent B) may include two parts, one is the final result data (Result), and the other is the log (Logs) generated by the data processing tool. If the calculation party (Agent A) wants to obtain the result data (Result) of the data party (Agent B), it needs to create a clear data transmission instruction through the micro-isolation system, wait for the approval of the data party (Agent B), and pass the micro-isolation system. The file transfer channel of the data transfer channel transmits the result data (Result) from the data side (Agent B) to the computing side (Agent A).
  • the computing party (Agent A) wants to obtain the logs (Logs) of the data party (Agent B), it needs to apply for the log viewing authorization from the data party (Agent B) through the micro-isolation system.
  • Application Programming Interface to view logs (Logs), which can realize real-time viewing of logs and facilitate system debugging.
  • the micro-isolation scheme allows the data side’s isolated domain environment without large-scale transformation, and the computing side
  • the isolation system remotely models and develops isolated data, analyzes big data, and satisfies the security isolation mechanism of the isolation domain.
  • the federated computing product process implements WEB interface operations, supports business processes such as task submission, authorization, execution and result generation, and provides a lower threshold for multi-party security data collaboration.
  • the deployment of the micro-isolation system can be completed in 2 hours, while the traditional isolation domain deployment and delivery takes about 4 months. 2.
  • the traditional isolation domain needs to rebuild a set of big data infrastructure such as Hadoop and Spark in a third party, which requires a lot of hardware resources such as servers, and BFC (Best Fit with Coalescing, the most suitable for merging) Then, the existing resources of the data party can be fully utilized, and data security and controllability can be realized based on the BFC security mechanism.
  • big data infrastructure such as Hadoop and Spark
  • BFC Best Fit with Coalescing, the most suitable for merging
  • the data platform is natively supported, and can be run without modifying the code: the micro-isolation mode can make full use of various data parties.
  • Basic data processing platforms including big data processing platforms (Hadoop, Spark, etc.), deep learning and machine learning platforms (Paddle, TensorFlow, etc.), traditional data analysis software (SAS (Statistical Analysis System), SPSS (Statistical Product) and Service Solutions, statistical products and service solutions), Mathematics, etc.), the user code can be run almost without modification, and the compatibility is good. 2.
  • Micro-isolation mode does not rely on cryptography, TEE and other technologies, so it can make full use of local hardware resources (GPU (Graphics Processing Unit, graphics processing unit), FPGA (Field Programmable Gate Array, Field Programmable Gate Arrays), etc.) and distributed frameworks for large-scale training and iteration, resulting in rapid results.
  • GPU Graphics Processing Unit
  • FPGA Field Programmable Gate Array
  • Field Programmable Gate Arrays Field Programmable Gate Arrays
  • the data party sends the metadata of the data to be processed to the computing party through the federated computing agent of the computing party; and then receives the data of the data to be processed generated based on the metadata of the data to be processed sent by the computing party.
  • Analysis logic based on the data analysis logic of the data to be processed, the corresponding result output of the data to be processed under at least one data processing tool is calculated. That is to say, the computing side in the present disclosure does not need to obtain the sensitive data in the data side.
  • the computing side first generates data analysis logic according to the metadata of the sensitive data, and then the data side can calculate the sensitive data in at least one based on the data analysis logic. The result output corresponding to the data processing tool.
  • the federated learning scheme and the isolated domain computing scheme are not particularly efficient in the scenario of cross-institution cooperation.
  • the present disclosure adopts the technical means that the computing party first generates data analysis logic according to the metadata of the sensitive data, and then the data party calculates the corresponding result output of the sensitive data under at least one data processing tool based on the data analysis logic, which overcomes the problems in the related art.
  • the federated learning scheme and the isolated domain computing scheme are not particularly efficient technical problems in the scenario of cross-institution cooperation.
  • the technical scheme provided by this disclosure can more efficiently realize cross-institution data cooperation, and realize the safety and reliability of data cooperation.
  • the technical solutions of the embodiments of the present disclosure are simple and convenient to implement, easy to popularize, and have a wider application range.
  • FIG. 6 is a schematic structural diagram of a data processing apparatus provided by an embodiment of the present disclosure.
  • the apparatus 600 includes: a data acquisition module 601, a logic generation module 602 and a logic transmission module 603; wherein, the data acquisition module 601 is configured to obtain data from the data party through the federated computing agent of the data party.
  • the metadata represents the characteristics of the data to be processed;
  • the logic generation module 602 is configured to perform federated computing on the computing side according to the metadata of the data to be processed Generate the data analysis logic of the data to be processed on the agent;
  • the logic sending module 603 is configured to send the data analysis logic of the data to be processed to the data party; make the data party based on the data to be processed
  • the data analysis logic calculates the result output corresponding to the data to be processed under at least one data processing tool; wherein, the result output includes: result data and output log.
  • the apparatus 600 further includes: a result receiving module (not shown in the figure), configured to create a data transmission instruction through a pre-deployed micro-isolation system; and send the data transmission instruction to the data party;
  • the data transmission instruction includes an identifier corresponding to at least one data processing tool; if the data party approves the data transmission instruction, the data to be processed is received in the at least one data processing tool through a pre-created file transmission channel.
  • the corresponding result data below.
  • the apparatus 600 further includes: a log viewing module (not shown in the figure), configured to send a log viewing application to the data party through a pre-deployed micro-isolation system; wherein the log viewing application includes at least one data processing tool The corresponding identifier; if the data party approves the log viewing application, the output log corresponding to the data to be processed under the at least one data processing tool is viewed through a preset application program interface.
  • a log viewing module (not shown in the figure), configured to send a log viewing application to the data party through a pre-deployed micro-isolation system; wherein the log viewing application includes at least one data processing tool The corresponding identifier; if the data party approves the log viewing application, the output log corresponding to the data to be processed under the at least one data processing tool is viewed through a preset application program interface.
  • the above-mentioned data processing apparatus can execute the method provided by the first embodiment of the present disclosure, and has corresponding functional modules and effects for executing the method.
  • the data processing method provided in Embodiment 1 of the present disclosure can execute the method provided by the first embodiment of the present disclosure, and has corresponding functional modules and effects for executing the method.
  • FIG. 7 is a schematic structural diagram of another data processing apparatus provided by an embodiment of the present disclosure.
  • the apparatus 700 includes: a data sending module 701, a logic receiving module 702 and a result calculating module 703; wherein, the data sending module 701 is configured to send data to the computing party through the federated computing agent of the computing party.
  • the metadata represents the characteristics of the data to be processed
  • the logic receiving module 702 is configured to receive the metadata generated based on the metadata of the data to be processed sent by the computing party
  • the result calculation module 703 is configured to calculate the corresponding result output of the data to be processed under at least one data processing tool based on the data analysis logic of the data to be processed
  • the result output includes: result data and output log.
  • the result calculation module 703 is configured to analyze the syntax and semantics of the data analysis logic of the data to be processed through the scheduler of the federated computing agent of the data party, and obtain the data used by the data analysis logic of the data to be processed. at least one data processing tool; use the at least one data processing tool to process the to-be-processed data to obtain a corresponding result output of the to-be-processed data under the at least one data processing tool.
  • the result calculation module 703 is further configured to approve the data analysis logic of the data to be processed through the approval unit of the data party; if the data analysis logic of the data to be processed is approved, Data analysis logic that processes data is submitted to the scheduler.
  • the apparatus 700 further includes: a result sending module (not shown in the figure), configured to receive a data transmission instruction sent by the computing party through a pre-deployed micro-isolation system; wherein the data transmission instruction includes at least one data processing The identifier corresponding to the tool; if the data party approves the data transmission instruction, the result data corresponding to the data to be processed under the at least one data processing tool is transmitted to the Calculator.
  • a result sending module (not shown in the figure), configured to receive a data transmission instruction sent by the computing party through a pre-deployed micro-isolation system; wherein the data transmission instruction includes at least one data processing The identifier corresponding to the tool; if the data party approves the data transmission instruction, the result data corresponding to the data to be processed under the at least one data processing tool is transmitted to the Calculator.
  • the apparatus 700 further includes: a log providing module (not shown in the figure), configured to receive a log viewing application sent by the computing party through a pre-deployed micro-isolation system; wherein the log viewing application includes at least one data processing The identifier corresponding to the tool; if the data party approves the log viewing application, it will provide the computing party with the product corresponding to the data to be processed under the at least one data processing tool through a preset application program interface. log out.
  • a log providing module (not shown in the figure), configured to receive a log viewing application sent by the computing party through a pre-deployed micro-isolation system; wherein the log viewing application includes at least one data processing The identifier corresponding to the tool; if the data party approves the log viewing application, it will provide the computing party with the product corresponding to the data to be processed under the at least one data processing tool through a preset application program interface. log out.
  • the above-mentioned data processing apparatus can execute the methods provided by the second and third embodiments of the present disclosure, and has functional modules and effects corresponding to the execution methods.
  • the data processing methods provided in Embodiment 2 and Embodiment 3 of the present disclosure.
  • the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.
  • FIG. 8 is a block diagram of an electronic device for implementing a data processing method provided by an embodiment of the present disclosure.
  • Electronic device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.
  • Electronic device 800 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices.
  • the components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.
  • the electronic device 800 includes a computing unit 801, which can be loaded into a RAM (Random Access Memory, random access) according to a computer program stored in a ROM (Read-Only Memory) 802 or from a storage unit 808 memory) 803 to perform various appropriate actions and processes.
  • a RAM Random Access Memory, random access
  • ROM Read-Only Memory
  • various programs and data required for the operation of the electronic device 800 can also be stored.
  • the computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804.
  • An I/O (Input/Output) interface 805 is also connected to the bus 804 .
  • Various components in the electronic device 800 are connected to the I/O interface 805, including: an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, and the like.
  • the communication unit 809 allows the electronic device 800 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.
  • Computing unit 801 may be a variety of general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, CPU (Central Processing Unit, central processing unit), GPU, various dedicated AI (Artificial Intelligence, artificial intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any suitable processor, controller, microcontroller, etc.
  • the computing unit 801 performs the above-described methods and processes, such as data processing methods.
  • a data processing method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 808 .
  • part or all of the computer program may be loaded and/or installed on the electronic device 800 via the ROM 802 and/or the communication unit 809 .
  • a computer program When a computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of the data processing method described above may be performed.
  • the computing unit 801 may be configured to perform the data processing method by any other suitable means (eg, by means of firmware).
  • These various embodiments may include implementation in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that
  • the processor which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.
  • Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented.
  • the program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing.
  • machine-readable storage media examples include one or more wire-based electrical connections, portable computer disks, hard disks, RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), or flash memory Memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage device, magnetic storage device, or any suitable combination of the above.
  • the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (Cathode Ray Tube) or an LCD (Liquid ray tube) configured to display information to the user Crystal Display (liquid crystal display) monitor); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer.
  • a display device eg, a CRT (Cathode Ray Tube) or an LCD (Liquid ray tube) configured to display information to the user Crystal Display (liquid crystal display) monitor
  • a keyboard and pointing device eg, a mouse or trackball
  • Other kinds of devices may also be configured to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and may be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.
  • the systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system.
  • the components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), blockchain network, and the Internet.
  • a computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other.
  • the server can be a cloud server, also known as a cloud computing server or a cloud host. It is a host product in the cloud computing service system to solve the problems existing in traditional physical hosts and VPS (Virtual Private Server, virtual private server) services. The management is difficult and the business expansion is weak.
  • the server can also be a server of a distributed system, or a server combined with a blockchain.
  • Steps can be reordered, added, or removed using the various forms of flow shown above.
  • steps described in the present disclosure can be performed in parallel, sequentially or in different orders, and as long as the desired results of the technical solutions provided in the present disclosure can be achieved, no limitation is imposed herein.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Library & Information Science (AREA)
  • Software Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Hardware Design (AREA)
  • General Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • Health & Medical Sciences (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本公开提供了一种数据处理方法、装置、电子设备及存储介质。数据处理方法，包括：通过数据方的联邦计算代理从数据方获取待处理数据的元数据；其中，元数据表示待处理数据的特征；根据待处理数据的元数据，在计算方的联邦计算代理上生成待处理数据的数据分析逻辑；将待处理数据的数据分析逻辑发送至数据方；使得数据方基于待处理数据的数据分析逻辑计算待处理数据在至少一个数据处理工具下对应的结果产出；其中，结果产出包括：结果数据和产出日志。

Description

数据处理方法、装置、电子设备及存储介质
本申请要求在2021年03月02日提交中国专利局、申请号为202110231850.2的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本公开涉及数据处理技术领域，例如涉及联邦学习技术，尤其涉及一种数据处理方法、装置、电子设备及存储介质。
背景技术
数据是一种重要的生产资料，为了挖掘数据价值，需要将不同机构的数据进行联合分析或者联合建模，以产生新的数据价值，创造新的业务形态。考虑到数据隐私、数据安全等监管需求，跨机构数据合作会面临很多新的挑战。
图1是相关技术中跨机构数据合作方式的结构示意图。如图1所示，常见的跨机构的数据合作方式主要有三种，分别是：方式一：纵向联邦XXY式，一方提供数据X1、X2，另外一方提供数据X3、X4和Y。方式二：纵向联邦XY式，一方提供数据X，另外一方提供数据Y。方式三：受控隔离式，一方提供数据，另外一方不提供数据，仅分析数据。
上述方式一和方式二为FL(Federated Learning，联邦学习)方案，该方案选用梯度交换、MPC(Secure multi-Party Computation，多方安全计算)和TEE(Trusted Execution Environment，可信执行环境)等；该方案在跨机构合作的场景下，并不是特别高效。上述方式三为隔离域计算方案，合作双方在相互之间无法信任的情况下，为了进行数据安全融合，寻找一个独立的可信的第三方环境，并基于网络安全技术构建一套隔离域环境，双方将各自拥有的数据放到隔离域中，双方均可以对数据操作进行监控和审计，以确保数据分析人员的所有行为均符合双方预先约定好的原则；该方案在跨机构合作的场景下，也不是特别高效。
发明内容
本公开提供了一种数据处理方法、装置、电子设备及存储介质，可以更加高效地实现跨机构的数据合作，实现数据合作的安全可控，节省数据合作的成本，避免资源浪费。
根据本公开的第一方面，提供了一种数据处理方法，应用于计算方，所述 方法包括：
通过数据方的联邦计算代理从所述数据方获取待处理数据的元数据；其中，所述元数据表示所述待处理数据的特征；
根据所述待处理数据的元数据，在所述计算方的联邦计算代理上生成所述待处理数据的数据分析逻辑；
将所述待处理数据的数据分析逻辑发送至所述数据方；使得所述数据方基于所述待处理数据的数据分析逻辑计算所述待处理数据在至少一个数据处理工具下对应的结果产出；其中，所述结果产出包括：结果数据和产出日志。
根据本公开的第二方面，提供了另一种数据处理方法，应用于数据方，所述方法包括：
通过计算方的联邦计算代理向所述计算方发送待处理数据的元数据；其中，所述元数据表示所述待处理数据的特征；
接收所述计算方发送的基于所述待处理数据的元数据生成的所述待处理数据的数据分析逻辑；
基于所述待处理数据的数据分析逻辑计算所述待处理数据在至少一个数据处理工具下对应的结果产出；其中，所述结果产出包括：结果数据和产出日志。
根据本公开的第三方面，提供了一种数据处理装置，所述装置包括：数据获取模块、逻辑生成模块和逻辑发送模块；其中，
所述数据获取模块，设置为通过数据方的联邦计算代理从所述数据方获取待处理数据的元数据；其中，所述元数据表示所述待处理数据的特征；
所述逻辑生成模块，设置为根据所述待处理数据的元数据，在所述计算方的联邦计算代理上生成所述待处理数据的数据分析逻辑；
所述逻辑发送模块，设置为将所述待处理数据的数据分析逻辑发送至所述数据方；使得所述数据方基于所述待处理数据的数据分析逻辑计算所述待处理数据在至少一个数据处理工具下对应的结果产出；其中，所述结果产出包括：结果数据和产出日志。
根据本公开的第四方面，提供了另一种数据处理装置，所述装置包括：数据发送模块、逻辑接收模块和结果计算模块；其中，
所述数据发送模块，设置为通过计算方的联邦计算代理向所述计算方发送待处理数据的元数据；其中，所述元数据表示所述待处理数据的特征；
所述逻辑接收模块，设置为接收所述计算方发送的基于所述待处理数据的元数据生成的所述待处理数据的数据分析逻辑；
所述结果计算模块，设置为基于所述待处理数据的数据分析逻辑计算所述待处理数据在至少一个数据处理工具下对应的结果产出；其中，所述结果产出包括：结果数据和产出日志。
根据本公开的第五方面，提供了一种电子设备，包括：
至少一个处理器；以及
与所述至少一个处理器通信连接的存储器；其中，
所述存储器设置为可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够实现上述的数据处理方法。
根据本公开的第六方面，提供了一种存储有计算机指令的非瞬时计算机可读存储介质，其中，所述计算机指令用于使所述计算机实现上述的数据处理方法。
根据本公开的第七方面，提供了一种计算机程序产品，包括计算机程序，所述计算机程序在被处理器执行时实现上述的数据处理方法。
附图说明
图1是相关技术中跨机构数据合作方式的结构示意图；
图2是本公开实施例提供的一种数据处理方法的流程示意图；
图3是本公开实施例提供的一种微隔离系统的结构示意图；
图4是本公开实施例提供的另一种数据处理方法的流程示意图；
图5是本公开实施例提供的另一种数据处理方法的流程示意图；
图6是本公开实施例提供的一种数据处理装置的结构示意图；
图7是本公开实施例提供的另一种数据处理装置的结构示意图；
图8是本公开实施例提供的一种用来实现数据处理方法的电子设备的框图。
具体实施方式
以下结合附图对本公开的示范性实施例做出说明，其中包括本公开实施例的多种细节以助于理解，应当将它们认为仅仅是示范性的。以下的描述中省略了对公知功能和结构的描述。
实施例一
图2是本公开实施例提供的一种数据处理方法的流程示意图，该方法可以由数据处理装置或者计算方的电子设备来执行，该装置或者计算方的电子设备 可以由软件和/或硬件的方式实现，该装置或者计算方的电子设备可以集成在任何具有网络通信功能的智能设备中。如图2所示，数据处理方法可以包括以下步骤：
S201、通过数据方的联邦计算代理从数据方获取待处理数据的元数据；其中，元数据表示待处理数据的特征。
在本步骤中，计算方可以通过数据方的联邦计算代理从数据方获取待处理数据的元数据；其中，元数据表示待处理数据的特征。本公开实施例中的元数据包括但不限于：数据仓库的名称、存储于数据仓库中的数据表的名称、数据表的字段名称、字段类型(String、Integer等)和行数等。
S202、根据待处理数据的元数据，在计算方的联邦计算代理上生成待处理数据的数据分析逻辑。
在本步骤中，计算方可以根据待处理数据的元数据，在计算方的联邦计算代理上生成待处理数据的数据分析逻辑。计算方可以对接收到的待处理数据的元数据进行分析，得到待处理数据的元数据的分析结果；然后根据待处理数据的元数据的分析结果，生成待处理数据的数据分析逻辑。
S203、将待处理数据的数据分析逻辑发送至数据方；使得数据方基于待处理数据的数据分析逻辑计算待处理数据在至少一个数据处理工具下对应的结果产出；其中，结果产出包括：结果数据和产出日志。
在本步骤中，计算方可以将待处理数据的数据分析逻辑发送至数据方；使得数据方基于待处理数据的数据分析逻辑计算待处理数据在多个数据处理工具下对应的结果产出；其中，结果产出包括：结果数据和产出日志。数据方可以先通过数据方的联邦计算代理的调度器对待处理数据的数据分析逻辑的语法和语义进行分析，得到待处理数据的数据分析逻辑所使用的至少一个数据处理工具；然后使用该至少一个数据处理工具对待处理数据进行处理，得到待处理数据在该至少一个数据处理工具下对应的结果产出。
图3是本公开实施例提供的一种微隔离系统的结构示意图。如图3所示，微隔离系统可以包括：计算方(Agent A)、数据方(Agent B)和协调方(Coordinator)；计算方和数据方通过各自的联邦计算代理传送和接收数据；计算方的联邦计算代理为Proxy A；数据方的联邦计算代理为Proxy B。同时，计算方和数据方通过各自的协同计算代理分别与协调方传送和接收数据。本公开中的微隔离系统采用SaaS(Software-as-a-Service，软件即服务)和私有化的分布式部署架构，其中协调方是SaaS部署，而Agent A和Agent B则采用私有化部署，用户部署的时候仅需要部署属于自己的Agent，然后按照系统引导完成必 要的系统配置即可。
本公开提出的微隔离方案是一种新型的隔离域技术，其中不涉及多方安全计算、联邦学习等隐私计算技术。微隔离系统提供对隔离域数据进行远程安全操作的统一入口，对隔离域数据的任务提交、授权执行、授权结果输出通过WEB界面实现轻量化、快速接入的统一管理。对于业务方而言只需部署联邦计算的计算节点，不需要很大的业务改造成本，即可实现数据不出域、数据可用不可见基础上的多方安全协同计算。本公开中的微隔离方案推荐配合“网闸”等技术一起实现，以实现更高的数据安全保护级别。在这种情况下，数据方的敏感数据和数据处理设施(Hadoop、Spark、Paddle等)均放置于“网闸”内，“网闸”可以保证数据只进不出。微隔离系统作为隔离域数据出域的唯一途径，这样如果需要出域则需经过微隔离系统的授权审批。
本公开实施例提出的数据处理方法，先通过数据方的联邦计算代理从数据方获取待处理数据的元数据；然后根据待处理数据的元数据，在计算方的联邦计算代理上生成待处理数据的数据分析逻辑；再将待处理数据的数据分析逻辑发送至数据方；使得数据方基于待处理数据的数据分析逻辑计算待处理数据在至少一个数据处理工具下对应的结果产出。也就是说，本公开中的计算方无需获取数据方中的敏感数据，计算方先根据敏感数据的元数据生成数据分析逻辑，然后数据方基于该数据分析逻辑即可计算出敏感数据在至少一个数据处理工具对应的结果产出。而在数据合作方式中，联邦学习方案和隔离域计算方案，在跨机构合作的场景下，都不是特别高效。本公开采用了计算方先根据敏感数据的元数据生成数据分析逻辑，然后数据方基于该数据分析逻辑计算敏感数据在至少一个数据处理工具下对应的结果产出的技术手段，克服了相关技术中的联邦学习方案和隔离域计算方案，在跨机构合作的场景下，都不是特别高效的技术问题，本公开提供的技术方案，可以更加高效地实现跨机构的数据合作，实现数据合作的安全可控，节省数据合作的成本，避免资源浪费；并且，本公开实施例的技术方案实现简单方便、便于普及，适用范围更广。
实施例二
图4是本公开实施例提供的另一种数据处理方法的流程示意图，该方法可以由数据处理装置或者数据方的电子设备来执行，该装置或者数据方的电子设备可以由软件和/或硬件的方式实现，该装置或者数据方的电子设备可以集成在任何具有网络通信功能的智能设备中。如图4所示，数据处理方法可以包括以下步骤：
S401、通过计算方的联邦计算代理向计算方发送待处理数据的元数据；其 中，元数据表示待处理数据的特征。
在本步骤中，数据方可以通过计算方的联邦计算代理向计算方发送待处理数据的元数据；其中，元数据表示待处理数据的特征。本公开实施例中的元数据包括但不限于：数据仓库的名称、存储于数据仓库中的数据表的名称、数据表的字段名称、字段类型(String、Integer等)和行数等。
S402、接收计算方发送的基于待处理数据的元数据生成的待处理数据的数据分析逻辑。
在本步骤中，数据方可以接收计算方发送的基于待处理数据的元数据生成的待处理数据的数据分析逻辑。计算方可以对接收到的待处理数据的元数据进行分析，得到待处理数据的元数据的分析结果；然后根据待处理数据的元数据的分析结果，生成待处理数据的数据分析逻辑。
S403、基于待处理数据的数据分析逻辑计算待处理数据在至少一个数据处理工具下对应的结果产出；其中，结果产出包括：结果数据和产出日志。
在本步骤中，数据方可以基于待处理数据的数据分析逻辑计算待处理数据在至少一个数据处理工具下对应的结果产出；其中，结果产出包括：结果数据和产出日志。数据方可以先通过数据方的联邦计算代理的调度器对待处理数据的数据分析逻辑的语法和语义进行分析，得到待处理数据的数据分析逻辑所使用的至少一个数据处理工具；然后使用该至少一个数据处理工具对待处理数据进行处理，得到待处理数据在至少一个数据处理工具下对应的结果产出。
在本公开的具体实施例中，数据方可以通过预先部署的微隔离系统接收计算方发送的数据传输指令；其中，数据传输指令包括至少一个数据处理工具对应的标识；若数据方对数据传输指令审批通过，则通过预先创建的文件传输通道将待处理数据在至少一个数据处理工具下对应的结果数据传送至计算方。此外，数据方还可以通过预先部署的微隔离系统接收计算方发送的日志查看申请；其中，日志查看申请包括至少一个数据处理工具对应的标识；若数据方对日志查看申请审批通过，则通过预先设置的应用程序接口向计算方提供待处理数据在至少一个数据处理工具下对应的产出日志。
本公开实施例提出的数据处理方法，数据方通过计算方的联邦计算代理向计算方发送待处理数据的元数据；然后接收计算方发送的基于待处理数据的元数据生成的待处理数据的数据分析逻辑；再基于待处理数据的数据分析逻辑计算待处理数据在至少一个数据处理工具下对应的结果产出。也就是说，本公开中的计算方无需获取数据方中的敏感数据，计算方先根据敏感数据的元数据生成数据分析逻辑，然后数据方基于该数据分析逻辑即可计算出敏感数据在至少 一个数据处理工具对应的结果产出。而在数据合作方式中，联邦学习方案和隔离域计算方案，在跨机构合作的场景下，都不是特别高效。本公开采用了计算方先根据敏感数据的元数据生成数据分析逻辑，然后数据方基于该数据分析逻辑计算敏感数据在至少一个数据处理工具下对应的结果产出的技术手段，克服了相关技术中的联邦学习方案和隔离域计算方案，在跨机构合作的场景下，都不是特别高效的技术问题，本公开提供的技术方案，可以更加高效地实现跨机构的数据合作，实现数据合作的安全可控，节省数据合作的成本，避免资源浪费；并且，本公开实施例的技术方案实现简单方便、便于普及，适用范围更广。
实施例三
图5是本公开实施例提供的另一种数据处理方法的流程示意图。基于上述技术方案进行说明，并可以与上述可选实施方式进行结合。如图5所示，数据处理方法可以包括以下步骤：
S501、通过计算方的联邦计算代理向计算方发送待处理数据的元数据；其中，元数据表示待处理数据的特征。
S502、接收计算方发送的基于待处理数据的元数据生成的待处理数据的数据分析逻辑。
S503、通过数据方的联邦计算代理的调度器对待处理数据的数据分析逻辑的语法和语义进行分析，得到待处理数据的数据分析逻辑所使用的至少一个数据处理工具。
在本步骤中，数据方可以通过数据方的联邦计算代理的调度器对待处理数据的数据分析逻辑的语法和语义进行分析，得到待处理数据的数据分析逻辑所使用的至少一个数据处理工具。数据方在通过数据方的联邦计算代理的调度器对待处理数据的数据分析逻辑的语法和语义进行分析之前，还可以先通过数据方的审批单元对待处理数据的数据分析逻辑进行审批；若对待处理数据的数据分析逻辑审批通过，则将待处理数据的数据分析逻辑提交给调度器。
S504、使用至少一个数据处理工具对待处理数据进行处理，得到待处理数据在至少一个数据处理工具下对应的结果产出；其中，结果产出包括：结果数据和产出日志。
在本公开的一个实施例中，数据方(Agent B)将自己的数据通过联邦计算代理接入到微隔离系统，并将待处理数据的元数据(例如：数据表有哪些字段、每个字段的数据类型是String还是Integer等)告诉计算方(Agent A)；计算方(Agent A)参照数据方(Agent B)发送的待处理数据的元数据，在自己的联邦 计算代理上编写数据分析逻辑，并将该数据分析逻辑通过联邦计算代理进行提交。数据方(Agent B)审核计算方(Agent A)的数据分析逻辑，审批(Approval)通过后，该数据分析逻辑被交给联邦计算代理的调度器(Scheduler)进行解析并调度。调度器通过分析数据分析逻辑的语法和语义，分析出数据分析逻辑所使用的数据处理工具(Hadoop、Spark、Tensor Flow等)，并按照数据分析逻辑的上下文语义自动调起所涉及的数据处理工具，对数据分析逻辑所涉及的DB(Data Base，数据库)进行分析处理。
在本公开的具体实施例中，数据方(Agent B)的结果产出可以包括两个部分，一个是最终结果数据(Result)，另外一个是数据处理工具所产生的日志(Logs)。如果计算方(Agent A)想获取数据方(Agent B)的结果数据(Result)，则需要通过微隔离系统创建一个明确的数据传输指令，等待数据方(Agent B)审批后，通过微隔离系统的文件传输通道将结果数据(Result)从数据方(Agent B)传送至计算方(Agent A)。
此外，如果计算方(Agent A)想获取数据方(Agent B)的日志(Logs)，则需要通过微隔离系统向数据方(Agent B)申请日志查看授权，获批后可以通过特定的API(Application Programming Interface，应用程序接口)查看日志(Logs)，这样可实现日志实时查看，为系统调试等提供便利。
与传统的隔离域方案相比，本公开提出的数据处理方法的主要优势是：1、轻量化部署，可视化易用：微隔离方案允许数据方的隔离域环境无需大规模改造，计算方通过微隔离系统远程对隔离数据进行建模开发、大数据挖分析，同时满足隔离域的安全隔离机制。联邦计算产品流程实现WEB界面操作，支持任务提交、授权、执行生成结果等业务流程，为多方安全数据协同提供更低门槛的方式。微隔离系统的部署可以在2个小时内完成，而传统隔离域部署交付大约需要4个月时间。2、“白盒”审查，安全可控：微隔离域内所执行的所有代码，均需要数据所有者事前审批，且能够实现“一事一批”。数据所有者知晓并审批数据分析者对数据的所有操作。相对于传统隔离域仅审批最终数据结果，而数据操作采用事后审计的做法，微隔离属于“白盒”安全。3、资源复用，避免浪费：传统隔离域需要在第三方重建一套Hadoop、Spark等大数据基础实施，这需要大量的服务器等硬件资源投入，而BFC(Best Fit with Coalescing，最适合合并)则可充分利用数据方已有的资源，基于BFC安全机制以实现数据安全可控。
与联邦学习、多方安全计算等隐私计算相比，本公开提出的数据处理方法的主要优势是：1、数据平台原生支持，无需修改代码即可运行：微隔离模式能够充分利用数据方的多种数据处理基础平台，包括大数据处理平台(Hadoop、 Spark等)、深度学习和机器学习平台(Paddle、TensorFlow等)、传统数据分析软件(SAS(Statistical Analysis System，统计分析系统)、SPSS(Statistical Product and Service Solutions，统计产品与服务解决方案)、Mathematics等)，用户代码几乎不需要修改即可运行，兼容性好。2、充分利用硬件资源，运行效率高：微隔离模式不依赖密码学、TEE等技术，因此可以充分利用本地的硬件资源(GPU(Graphics Processing Unit，图形处理单元)、FPGA(Field Programmable Gate Array，场可编程门阵列)等)和分布式框架，进行大规模训练与迭代，快速产生结果。
本公开实施例提出的数据处理方法，数据方通过计算方的联邦计算代理向计算方发送待处理数据的元数据；然后接收计算方发送的基于待处理数据的元数据生成的待处理数据的数据分析逻辑；再基于待处理数据的数据分析逻辑计算待处理数据在至少一个数据处理工具下对应的结果产出。也就是说，本公开中的计算方无需获取数据方中的敏感数据，计算方先根据敏感数据的元数据生成数据分析逻辑，然后数据方基于该数据分析逻辑即可计算出敏感数据在至少一个数据处理工具对应的结果产出。而在数据合作方式中，联邦学习方案和隔离域计算方案，在跨机构合作的场景下，都不是特别高效。本公开采用了计算方先根据敏感数据的元数据生成数据分析逻辑，然后数据方基于该数据分析逻辑计算敏感数据在至少一个数据处理工具下对应的结果产出的技术手段，克服了相关技术中的联邦学习方案和隔离域计算方案，在跨机构合作的场景下，都不是特别高效的技术问题，本公开提供的技术方案，可以更加高效地实现跨机构的数据合作，实现数据合作的安全可控，节省数据合作的成本，避免资源浪费；并且，本公开实施例的技术方案实现简单方便、便于普及，适用范围更广。
实施例四
图6是本公开实施例提供的一种数据处理装置的结构示意图。如图6所示，所述装置600包括：数据获取模块601、逻辑生成模块602和逻辑发送模块603；其中，所述数据获取模块601，设置为通过数据方的联邦计算代理从所述数据方获取待处理数据的元数据；其中，所述元数据表示所述待处理数据的特征；所述逻辑生成模块602，设置为根据所述待处理数据的元数据，在所述计算方的联邦计算代理上生成所述待处理数据的数据分析逻辑；所述逻辑发送模块603，设置为将所述待处理数据的数据分析逻辑发送至所述数据方；使得所述数据方基于所述待处理数据的数据分析逻辑计算所述待处理数据在至少一个数据处理工具下对应的结果产出；其中，所述结果产出包括：结果数据和产出日志。
所述装置600还包括：结果接收模块(图中未示出)，设置为通过预先部 署的微隔离系统创建一个数据传输指令；并将所述数据传输指令发送至所述数据方；其中，所述数据传输指令包括至少一个数据处理工具对应的标识；若所述数据方对所述数据传输指令审批通过，则通过预先创建的文件传输通道接收所述待处理数据在所述至少一个数据处理工具下对应的结果数据。
所述装置600还包括：日志查看模块(图中未示出)，设置为通过预先部署的微隔离系统向所述数据方发送日志查看申请；其中，所述日志查看申请包括至少一个数据处理工具对应的标识；若所述数据方对所述日志查看申请审批通过，则通过预先设置的应用程序接口查看所述待处理数据在所述至少一个数据处理工具下对应的产出日志。
上述数据处理装置可执行本公开实施例一所提供的方法，具备执行方法相应的功能模块和效果。未在本实施例中详尽描述的技术细节，可参见本公开实施例一提供的数据处理方法。
实施例五
图7是本公开实施例提供的另一种数据处理装置的结构示意图。如图7所示，所述装置700包括：数据发送模块701、逻辑接收模块702和结果计算模块703；其中，所述数据发送模块701，设置为通过计算方的联邦计算代理向所述计算方发送待处理数据的元数据；其中，所述元数据表示所述待处理数据的特征；所述逻辑接收模块702，设置为接收所述计算方发送的基于所述待处理数据的元数据生成的所述待处理数据的数据分析逻辑；所述结果计算模块703，设置为基于所述待处理数据的数据分析逻辑计算所述待处理数据在至少一个数据处理工具下对应的结果产出；其中，所述结果产出包括：结果数据和产出日志。
所述结果计算模块703，设置为通过所述数据方的联邦计算代理的调度器对所述待处理数据的数据分析逻辑的语法和语义进行分析，得到所述待处理数据的数据分析逻辑所使用的至少一个数据处理工具；使用所述至少一个数据处理工具对所述待处理数据进行处理，得到所述待处理数据在至少一个数据处理工具下对应的结果产出。
所述结果计算模块703，还设置为通过所述数据方的审批单元对所述待处理数据的数据分析逻辑进行审批；若对所述待处理数据的数据分析逻辑审批通过，则将所述待处理数据的数据分析逻辑提交给所述调度器。
所述装置700还包括：结果发送模块(图中未示出)，设置为通过预先部署的微隔离系统接收所述计算方发送的数据传输指令；其中，所述数据传输指令包括至少一个数据处理工具对应的标识；若所述数据方对所述数据传输指令 审批通过，则通过预先创建的文件传输通道将所述待处理数据在所述至少一个数据处理工具下对应的结果数据传送至所述计算方。
所述装置700还包括：日志提供模块(图中未示出)，设置为通过预先部署的微隔离系统接收所述计算方发送的日志查看申请；其中，所述日志查看申请包括至少一个数据处理工具对应的标识；若所述数据方对所述日志查看申请审批通过，则通过预先设置的应用程序接口向所述计算方提供所述待处理数据在所述至少一个数据处理工具下对应的产出日志。
上述数据处理装置可执行本公开实施例二和实施例三所提供的方法，具备执行方法相应的功能模块和效果。未在本实施例中详尽描述的技术细节，可参见本公开实施例二和实施例三提供的数据处理方法。
本公开的技术方案中，所涉及的用户个人信息的获取，存储和应用等，均符合相关法律法规的规定，且不违背公序良俗。
实施例六
根据本公开的实施例，本公开还提供了一种电子设备、一种可读存储介质和一种计算机程序产品。
图8是本公开实施例提供的一种用来实现数据处理方法的电子设备的框图。电子设备800旨在表示多种形式的数字计算机，诸如，膝上型计算机、台式计算机、工作台、个人数字助理、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备800还可以表示多种形式的移动装置，诸如，个人数字处理、蜂窝电话、智能电话、可穿戴设备和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和/或者要求的本公开的实现。
如图8所示，电子设备800包括计算单元801，其可以根据存储在ROM(Read-Only Memory，只读存储器)802中的计算机程序或者从存储单元808加载到RAM(Random Access Memory，随机访问存储器)803中的计算机程序，来执行多种适当的动作和处理。在RAM 803中，还可存储电子设备800操作所需的多种程序和数据。计算单元801、ROM 802以及RAM 803通过总线804彼此相连。I/O(Input/Output，输入/输出)接口805也连接至总线804。
电子设备800中的多个部件连接至I/O接口805，包括：输入单元806，例如键盘、鼠标等；输出单元807，例如多种类型的显示器、扬声器等；存储单元808，例如磁盘、光盘等；以及通信单元809，例如网卡、调制解调器、无线通信收发机等。通信单元809允许电子设备800通过诸如因特网的计算机网络和/ 或多种电信网络与其他设备交换信息/数据。
计算单元801可以是多种具有处理和计算能力的通用和/或专用处理组件。计算单元801的一些示例包括但不限于CPU(Central Processing Unit，中央处理单元)、GPU、多种专用的AI(Artificial Intelligence，人工智能)计算芯片、多种运行机器学习模型算法的计算单元、DSP(Digital Signal Processor，数字信号处理器)、以及任何适当的处理器、控制器、微控制器等。计算单元801执行上文所描述的方法和处理，例如数据处理方法。例如，在一些实施例中，数据处理方法可被实现为计算机软件程序，其被有形地包含于机器可读介质，例如存储单元808。在一些实施例中，计算机程序的部分或者全部可以经由ROM802和/或通信单元809而被载入和/或安装到电子设备800上。当计算机程序加载到RAM 803并由计算单元801执行时，可以执行上文描述的数据处理方法的一个或多个步骤。备选地，在其他实施例中，计算单元801可以通过其他任何适当的方式(例如，借助于固件)而被配置为执行数据处理方法。
本文中以上描述的系统和技术的多种实施方式可以在数字电子电路系统、集成电路系统、FPGA、ASIC(Application Specific Integrated Circuit，专用集成电路)、ASSP(Application Specific Standard Parts，专用标准产品)、SoC(System on Chip，芯片上系统的系统)、CPLD(Complex Programming Logic Device，复杂可编程逻辑设备)、计算机硬件、固件、软件、和/或它们的组合中实现。这些多种实施方式可以包括：实施在一个或者多个计算机程序中，该一个或者多个计算机程序可在包括至少一个可编程处理器的可编程系统上执行和/或解释，该可编程处理器可以是专用或者通用可编程处理器，可以从存储系统、至少一个输入装置、和至少一个输出装置接收数据和指令，并且将数据和指令传输至该存储系统、该至少一个输入装置、和该至少一个输出装置。
用于实施本公开的方法的程序代码可以采用一个或多个编程语言的任何组合来编写。这些程序代码可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理器或控制器，使得程序代码当由处理器或控制器执行时使流程图和/或框图中所规定的功能/操作被实施。程序代码可以完全在机器上执行、部分地在机器上执行，作为独立软件包部分地在机器上执行且部分地在远程机器上执行或完全在远程机器或服务器上执行。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质 的示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、RAM、ROM、EPROM(Erasable Programmable Read-Only Memory，可擦除可编程只读存储器)或快闪存储器、光纤、CD-ROM(Compact Disc Read-Only Memory，便捷式紧凑盘只读存储器)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
为了提供与用户的交互，可以在计算机上实施此处描述的系统和技术，该计算机具有：设置为向用户显示信息的显示装置(例如，CRT(Cathode Ray Tube，阴极射线管)或者LCD(Liquid Crystal Display，液晶显示器)监视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给计算机。其它种类的装置还可以设置为提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。
可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：LAN(Local Area Network，局域网)、WAN(Wide Area Network，广域网)、区块链网络和互联网。
计算机系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。服务器可以是云服务器，又称为云计算服务器或云主机，是云计算服务体系中的一项主机产品，以解决了传统物理主机与VPS(Virtual Private Server，虚拟专用服务器)服务中，存在的管理难度大，业务扩展性弱的缺陷。服务器也可以为分布式系统的服务器，或者是结合了区块链的服务器。
可以使用上面所示的多种形式的流程，重新排序、增加或删除步骤。例如，本公开中记载的多个步骤可以并行地执行也可以顺序地执行也可以不同的次序执行，只要能够实现本公开提供的技术方案所期望的结果，本文在此不进行限制。

Claims (19)

  1. 一种数据处理方法，应用于计算方，包括：
    通过数据方的联邦计算代理从所述数据方获取待处理数据的元数据；其中，所述元数据表示所述待处理数据的特征；
    根据所述待处理数据的元数据，在所述计算方的联邦计算代理上生成所述待处理数据的数据分析逻辑；
    将所述待处理数据的数据分析逻辑发送至所述数据方；使得所述数据方基于所述待处理数据的数据分析逻辑计算所述待处理数据在至少一个数据处理工具下对应的结果产出；其中，所述结果产出包括：结果数据和产出日志。
  2. 根据权利要求1所述的方法，还包括：
    通过预先部署的微隔离系统创建一个数据传输指令；并将所述数据传输指令发送至所述数据方；其中，所述数据传输指令包括至少一个数据处理工具对应的标识；
    在所述数据方对所述数据传输指令审批通过的情况下，通过预先创建的文件传输通道接收所述待处理数据在所述数据传输指令中的所述至少一个数据处理工具下对应的结果数据。
  3. 根据权利要求1所述的方法，还包括：
    通过预先部署的微隔离系统向所述数据方发送日志查看申请；其中，所述日志查看申请包括至少一个数据处理工具对应的标识；
    在所述数据方对所述日志查看申请审批通过的情况下，通过预先设置的应用程序接口查看所述待处理数据在所述日志查看申请中的所述至少一个数据处理工具下对应的产出日志。
  4. 一种数据处理方法，应用于数据方，包括：
    通过计算方的联邦计算代理向所述计算方发送待处理数据的元数据；其中，所述元数据表示所述待处理数据的特征；
    接收所述计算方发送的基于所述待处理数据的元数据生成的所述待处理数据的数据分析逻辑；
    基于所述待处理数据的数据分析逻辑计算所述待处理数据在至少一个数据处理工具下对应的结果产出；其中，所述结果产出包括：结果数据和产出日志。
  5. 根据权利要求4所述的方法，其中，所述基于所述待处理数据的数据分析逻辑计算所述待处理数据在至少一个数据处理工具下对应的结果产出，包括：
    通过所述数据方的联邦计算代理的调度器对所述待处理数据的数据分析逻 辑的语法和语义进行分析，得到所述待处理数据的数据分析逻辑所使用的所述至少一个数据处理工具；
    使用所述至少一个数据处理工具对所述待处理数据进行处理，得到所述待处理数据在所述至少一个数据处理工具下对应的结果产出。
  6. 根据权利要求5所述的方法，在所述通过所述数据方的联邦计算代理的调度器对所述待处理数据的数据分析逻辑的语法和语义进行分析之前，还包括：
    通过所述数据方的审批单元对所述待处理数据的数据分析逻辑进行审批；在对所述待处理数据的数据分析逻辑审批通过的情况下，将所述待处理数据的数据分析逻辑提交给所述调度器。
  7. 根据权利要求4所述的方法，还包括：
    通过预先部署的微隔离系统接收所述计算方发送的数据传输指令；其中，所述数据传输指令包括至少一个数据处理工具对应的标识；
    在所述数据方对所述数据传输指令审批通过的情况下，通过预先创建的文件传输通道将所述待处理数据在所述数据传输指令中的所述至少一个数据处理工具下对应的结果数据传送至所述计算方。
  8. 根据权利要求4所述的方法，还包括：
    通过预先部署的微隔离系统接收所述计算方发送的日志查看申请；其中，所述日志查看申请包括至少一个数据处理工具对应的标识；
    在所述数据方对所述日志查看申请审批通过的情况下，通过预先设置的应用程序接口向所述计算方提供所述待处理数据在所述日志查看申请中的所述至少一个数据处理工具下对应的产出日志。
  9. 一种数据处理装置，包括：数据获取模块、逻辑生成模块和逻辑发送模块；其中，
    所述数据获取模块，设置为通过数据方的联邦计算代理从所述数据方获取待处理数据的元数据；其中，所述元数据表示所述待处理数据的特征；
    所述逻辑生成模块，设置为根据所述待处理数据的元数据，在所述计算方的联邦计算代理上生成所述待处理数据的数据分析逻辑；
    所述逻辑发送模块，设置为将所述待处理数据的数据分析逻辑发送至所述数据方；使得所述数据方基于所述待处理数据的数据分析逻辑计算所述待处理数据在各个数据处理工具下对应的结果产出；其中，所述结果产出包括：结果数据和产出日志。
  10. 根据权利要求9所述的装置，还包括：结果接收模块，设置为通过预先 部署的微隔离系统创建一个数据传输指令；并将所述数据传输指令发送至所述数据方；其中，所述数据传输指令包括至少一个数据处理工具对应的标识；若所述数据方对所述数据传输指令审批通过，则通过预先创建的文件传输通道接收所述待处理数据在所述至少一个数据处理工具下对应的结果数据。
  11. 根据权利要求9所述的装置，还包括：日志查看模块，设置为通过预先部署的微隔离系统向所述数据方发送日志查看申请；其中，所述日志查看申请包括至少一个数据处理工具对应的标识；若所述数据方对所述日志查看申请审批通过，则通过预先设置的应用程序接口查看所述待处理数据在所述至少一个数据处理工具下对应的产出日志。
  12. 一种数据处理装置，包括：数据发送模块、逻辑接收模块和结果计算模块；其中，
    所述数据发送模块，设置为通过计算方的联邦计算代理向所述计算方发送待处理数据的元数据；其中，所述元数据表示所述待处理数据的特征；
    所述逻辑接收模块，设置为接收所述计算方发送的基于所述待处理数据的元数据生成的所述待处理数据的数据分析逻辑；
    所述结果计算模块，设置为基于所述待处理数据的数据分析逻辑计算所述待处理数据在各个数据处理工具下对应的结果产出；其中，所述结果产出包括：结果数据和产出日志。
  13. 根据权利要求12所述的装置，其中，所述结果计算模块，设置为通过所述数据方的联邦计算代理的调度器对所述待处理数据的数据分析逻辑的语法和语义进行分析，得到所述待处理数据的数据分析逻辑所使用的至少一个数据处理工具；使用所述至少一个数据处理工具对所述待处理数据进行处理，得到所述待处理数据在各个数据处理工具下对应的结果产出。
  14. 根据权利要求13所述的装置，其中，所述结果计算模块，还设置为通过所述数据方的审批单元对所述待处理数据的数据分析逻辑进行审批；若对所述待处理数据的数据分析逻辑审批通过，则将所述待处理数据的数据分析逻辑提交给所述调度器。
  15. 根据权利要求12所述的装置，还包括：结果发送模块，其中，通过预先部署的微隔离系统接收所述计算方发送的数据传输指令；其中，所述数据传输指令包括至少一个数据处理工具对应的标识；若所述数据方对所述数据传输指令审批通过，则通过预先创建的文件传输通道将所述待处理数据在所述至少一个数据处理工具下对应的结果数据传送至所述计算方。
  16. 根据权利要求12所述的装置，还包括：日志提供模块，其中，通过预 先部署的微隔离系统接收所述计算方发送的日志查看申请；其中，所述日志查看申请包括至少一个数据处理工具对应的标识；若所述数据方对所述日志查看申请审批通过，则通过预先设置的应用程序接口向所述计算方提供所述待处理数据在所述至少一个数据处理工具下对应的产出日志。
  17. 一种电子设备，包括：
    至少一个处理器；以及
    与所述至少一个处理器通信连接的存储器；其中，
    所述存储器设置为存储可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行权利要求1-3或者4-8中任一项所述的方法。
  18. 一种存储有计算机指令的非瞬时计算机可读存储介质，其中，所述计算机指令用于使所述计算机执行根据权利要求1-3或者4-8中任一项所述的方法。
  19. 一种计算机程序产品，包括计算机程序，所述计算机程序在被处理器执行时实现根据权利要求1-3或者4-8中任一项所述的方法。
PCT/CN2021/103669 2021-03-02 2021-06-30 数据处理方法、装置、电子设备及存储介质 Ceased WO2022183644A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110231850.2 2021-03-02
CN202110231850.2A CN112860970B (zh) 2021-03-02 2021-03-02 一种数据处理方法、装置、电子设备及存储介质

Publications (1)

Publication Number Publication Date
WO2022183644A1 true WO2022183644A1 (zh) 2022-09-09

Family

ID=75991055

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/103669 Ceased WO2022183644A1 (zh) 2021-03-02 2021-06-30 数据处理方法、装置、电子设备及存储介质

Country Status (2)

Country Link
CN (1) CN112860970B (zh)
WO (1) WO2022183644A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116069997A (zh) * 2022-09-27 2023-05-05 广东电网有限责任公司 元数据分析著录方法、装置、电子设备和存储介质
WO2025130307A1 (zh) * 2023-12-21 2025-06-26 华为技术有限公司 一种安全多方计算方法及装置
CN120653258A (zh) * 2025-08-18 2025-09-16 启元实验室 基于动态代码生成的数据处理方法及装置

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112860970B (zh) * 2021-03-02 2024-03-12 百度在线网络技术（北京）有限公司 一种数据处理方法、装置、电子设备及存储介质
CN113537508B (zh) * 2021-06-18 2024-02-02 百度在线网络技术（北京）有限公司 联邦计算的处理方法、装置、电子设备和存储介质
CN113704314A (zh) * 2021-08-04 2021-11-26 百度在线网络技术（北京）有限公司 数据的分析方法及装置、电子设备和存储介质
CN115730574A (zh) * 2022-10-31 2023-03-03 蚂蚁区块链科技(上海)有限公司 生成数据表的方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108241722A (zh) * 2016-12-23 2018-07-03 北京金山云网络技术有限公司 一种数据处理系统、方法及装置
US20190361772A1 (en) * 2018-05-24 2019-11-28 International Business Machines Corporation Metadata hardening and parity accumulation for log-structured arrays
CN110781165A (zh) * 2019-10-10 2020-02-11 支付宝(杭州)信息技术有限公司 一种业务数据的处理方法、装置和设备
CN111914038A (zh) * 2020-09-21 2020-11-10 北京百度网讯科技有限公司 联邦计算方法、装置、设备及存储介质
CN112860970A (zh) * 2021-03-02 2021-05-28 百度在线网络技术（北京）有限公司 一种数据处理方法、装置、电子设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108241722A (zh) * 2016-12-23 2018-07-03 北京金山云网络技术有限公司 一种数据处理系统、方法及装置
US20190361772A1 (en) * 2018-05-24 2019-11-28 International Business Machines Corporation Metadata hardening and parity accumulation for log-structured arrays
CN110781165A (zh) * 2019-10-10 2020-02-11 支付宝(杭州)信息技术有限公司 一种业务数据的处理方法、装置和设备
CN111914038A (zh) * 2020-09-21 2020-11-10 北京百度网讯科技有限公司 联邦计算方法、装置、设备及存储介质
CN112860970A (zh) * 2021-03-02 2021-05-28 百度在线网络技术（北京）有限公司 一种数据处理方法、装置、电子设备及存储介质

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116069997A (zh) * 2022-09-27 2023-05-05 广东电网有限责任公司 元数据分析著录方法、装置、电子设备和存储介质
WO2025130307A1 (zh) * 2023-12-21 2025-06-26 华为技术有限公司 一种安全多方计算方法及装置
CN120653258A (zh) * 2025-08-18 2025-09-16 启元实验室 基于动态代码生成的数据处理方法及装置

Also Published As

Publication number Publication date
CN112860970B (zh) 2024-03-12
CN112860970A (zh) 2021-05-28

Similar Documents

Publication Publication Date Title
CN112860970B (zh) 一种数据处理方法、装置、电子设备及存储介质
CN114675965B (zh) 联邦学习方法、装置、设备和介质
US10025620B2 (en) Incremental parallel processing of data
US20240037436A1 (en) System for processing quantum task, method for processing quantum task and related apparatuses
KR20230006033A (ko) 연합 계산 처리 방법, 장치, 전자 기기 및 저장 매체
CN110704290A (zh) 日志分析方法及装置
CN113691374B (zh) 数据加密方法及装置、存储介质及电子设备
WO2023103390A1 (zh) 任务处理方法、任务处理装置、电子设备以及存储介质
US20220100777A1 (en) Transaction request construction method and apparatus, transaction request processing method and apparatus, device and storage medium
CN113032817B (zh) 基于区块链的数据对齐方法、装置、设备和介质
KR20220151132A (ko) 네트워크 기기의 테스트 방법 및 장치
CN112231336B (zh) 识别用户的方法、装置、存储介质及电子设备
CN108984202B (zh) 一种电子资源分享方法、装置和存储介质
CN113935070A (zh) 基于区块链的数据处理方法、装置、设备以及存储介质
CN114548983A (zh) 一种区块链隐私数据处理方法、碳交易实现方法及系统
CN113037489B (zh) 数据处理方法、装置、设备和存储介质
Al-Khafaji et al. Secured environment for cloud integrated fog and mist architecture
CN112486460A (zh) 自动导入接口文档的方法、系统、设备和介质
CN116541224A (zh) 性能测试方法、装置、电子设备与可读存储介质
CN114338051B (zh) 区块链获取随机数的方法、装置、设备和介质
CN110728009A (zh) 仿真方法、装置和系统
KR20220054556A (ko) 응용 프로그램 통신 방법, 기기 및 저장 매체
CN115129554A (zh) 一种共识交易追踪方法、装置、设备及存储介质
Deng et al. [Retracted] Security Risk and Preventive Measures of Multimedia Database System under Remote Control of Network Robot
CN114978626B (zh) 基于区块链的可信计算方法、装置、设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21928722

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21928722

Country of ref document: EP

Kind code of ref document: A1