CN112860970A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112860970A CN112860970A CN202110231850.2A CN202110231850A CN112860970A CN 112860970 A CN112860970 A CN 112860970A CN 202110231850 A CN202110231850 A CN 202110231850A CN 112860970 A CN112860970 A CN 112860970A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- processing tool
- analysis logic
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 33
- 238000003860 storage Methods 0.000 title claims abstract description 16
- 238000012545 processing Methods 0.000 claims abstract description 105
- 238000007405 data analysis Methods 0.000 claims abstract description 97
- 238000004364 calculation method Methods 0.000 claims abstract description 58
- 238000002955 isolation Methods 0.000 claims description 54
- 230000005540 biological transmission Effects 0.000 claims description 33
- 238000000034 method Methods 0.000 claims description 31
- 238000004590 computer program Methods 0.000 claims description 11
- 238000013215 result calculation Methods 0.000 claims description 6
- 238000005516 engineering process Methods 0.000 abstract description 8
- 239000002699 waste material Substances 0.000 abstract description 6
- 239000003795 chemical substances by application Substances 0.000 description 42
- 238000010586 diagram Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 10
- 238000004458 analytical method Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000012550 audit Methods 0.000 description 2
- 238000013475 authorization Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/907—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/908—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
- G06F16/212—Schema design and management with details for data modelling support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Library & Information Science (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- General Health & Medical Sciences (AREA)
- Bioethics (AREA)
- Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The disclosure discloses a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, in particular to a federal learning technology. The specific implementation scheme is as follows: acquiring metadata of data to be processed from a data party through a federal calculation agent of the data party; wherein the metadata represents characteristics of the data to be processed; generating data analysis logic of the data to be processed on a federal calculation agent of a calculator according to metadata of the data to be processed; sending the data analysis logic of the data to be processed to a data side; enabling the data side to calculate corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log. The embodiment of the application can more efficiently realize cross-mechanism data cooperation, is safe and controllable, saves cost and avoids resource waste.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and further relates to a federal learning technology, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.
Background
Data is an important production data, and in order to further mine data value, joint analysis or joint modeling needs to be performed on data of different organizations to generate new data value and create new business forms. Given regulatory requirements for data privacy, data security, etc., cross-organization data collaboration faces many new challenges.
Fig. 1 is a schematic diagram of a prior art cross-facility data collaboration approach. As shown in fig. 1, there are three common cross-organization data collaboration modes, which are: mode 1: longitudinal federal formula XXY, one provides X1, X2, and the other provides X3, X4, and Y. Mode 2: vertical federal XY, one party provides X and the other provides Y. Mode 3: in the controlled isolation mode, one side provides data, the other side does not provide data, and only the data is analyzed.
The above-mentioned mode 1 and mode 2 are federal Learning schemes, and the schemes select gradient exchange (FL), multi-party Secure computing (MPC), executable Environment (TEE), and so on; this scheme is not particularly efficient in a cross-organization collaboration scenario. The above mode 3 is an isolation domain calculation scheme, where two cooperating parties cannot trust each other, to perform data security fusion, an independent and trusted third-party environment is found, and a set of isolation domain environment is constructed based on a network security technology, and the two parties place their own data in an isolation domain, and both parties can monitor and audit data operations, so as to ensure that all behaviors of a data analyst conform to a principle agreed in advance by both parties. This solution is also not particularly efficient in a cross-organization collaboration scenario.
Disclosure of Invention
The application provides a data processing method, a data processing device, an electronic device and a storage medium, which can more efficiently realize cross-mechanism data cooperation, are safe and controllable, save cost and avoid resource waste.
According to a first aspect of the present application, there is provided a data processing method applied to a computing side, the method including:
acquiring metadata of data to be processed from a data party through a federal calculation agent of the data party; wherein the metadata represents characteristics of the data to be processed;
generating data analysis logic of the data to be processed on a federal calculation agent of the calculator according to the metadata of the data to be processed;
sending the data analysis logic of the data to be processed to the data side; enabling the data side to calculate corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log.
According to a second aspect of the present application, there is provided another data processing method applied to a data side, the method including:
sending metadata of data to be processed to a computer side through a federal calculation agent of the computer side; wherein the metadata represents characteristics of the data to be processed;
receiving data analysis logic of the data to be processed, which is sent by the computer and generated based on metadata of the data to be processed;
calculating corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log.
According to a third aspect of the present application, there is provided a data processing apparatus comprising: the device comprises a data acquisition module, a logic generation module and a logic sending module; wherein,
the data acquisition module is used for acquiring metadata of data to be processed from a data party through a federal calculation agent of the data party; wherein the metadata represents characteristics of the data to be processed;
the logic generation module is used for generating data analysis logic of the data to be processed on a federal calculation agent of the calculator according to the metadata of the data to be processed;
the logic sending module is used for sending the data analysis logic of the data to be processed to the data side; enabling the data side to calculate corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log.
According to a fourth aspect of the present application, there is provided another data processing apparatus comprising: the system comprises a data sending module, a logic receiving module and a result calculating module; wherein,
the data sending module is used for sending metadata of data to be processed to the computer party through a federal calculation agent of the computer party; wherein the metadata represents characteristics of the data to be processed;
the logic receiving module is used for receiving data analysis logic of the data to be processed, which is sent by the computer and generated based on the metadata of the data to be processed;
the result calculation module is used for calculating corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log.
According to a fifth aspect of the present application, there is provided an electronic device comprising:
one or more processors;
a memory for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any embodiment of the present application.
According to a sixth aspect of the present application, there is provided a storage medium having stored thereon a computer program which, when executed by a processor, implements the data processing method described in any of the embodiments of the present application.
According to a seventh aspect of the present application, there is provided a computer program product which, when executed by a computer device, implements the data processing method of any of the embodiments of the present application.
According to the technical scheme, the technical problem that a federal learning scheme and an isolation domain calculation scheme in the prior art are not particularly efficient in a cross-organization cooperation scene is solved, and the cross-organization data cooperation can be realized more efficiently, so that the method is safe and controllable, saves the cost and avoids resource waste.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is a schematic diagram of a prior art cross-facility data collaboration approach;
fig. 2 is a first flowchart of a data processing method according to an embodiment of the present application;
FIG. 3 is a schematic structural diagram of a micro-isolation system provided in an embodiment of the present application;
fig. 4 is a second flowchart of a data processing method provided in an embodiment of the present application;
fig. 5 is a third flow chart of a data processing method according to an embodiment of the present application;
FIG. 6 is a schematic diagram of a first structure of a data processing apparatus according to an embodiment of the present application;
fig. 7 is a second schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
fig. 8 is a block diagram of an electronic device for implementing the data processing method according to the embodiment of the present application.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Example one
Fig. 2 is a first flowchart of a data processing method provided in an embodiment of the present application, where the method may be performed by a data processing apparatus or an electronic device of a computing party, where the apparatus or the electronic device of the computing party may be implemented by software and/or hardware, and the apparatus or the electronic device of the computing party may be integrated in any intelligent device with a network communication function. As shown in fig. 2, the data processing method may include the steps of:
s201, obtaining metadata of data to be processed from a data party through a federal calculation agent of the data party; wherein the metadata represents characteristics of the data to be processed.
In the step, the computer side can obtain the metadata of the data to be processed from the data side through the federal computer agency of the data side; wherein the metadata represents characteristics of the data to be processed. Specifically, the metadata in the embodiment of the present application includes, but is not limited to: the name of the data warehouse, the name of the data table stored in the data warehouse, the field name of the data table, the field type (String, Integer, etc.), the number of rows, etc.
S202, generating data analysis logic of the data to be processed on a federal calculation agent of a calculator according to the metadata of the data to be processed.
In this step, the computer may generate a data analysis logic of the data to be processed on the federal computer proxy of the computer according to the metadata of the data to be processed. Specifically, the calculator may analyze the received metadata of the data to be processed to obtain an analysis result of the metadata of the data to be processed; and then generating data analysis logic of the data to be processed according to the analysis result of the metadata of the data to be processed.
S203, sending data analysis logic of the data to be processed to a data side; enabling the data side to calculate corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log.
In this step, the calculator may send the data analysis logic of the data to be processed to the data calculator; enabling the data side to calculate corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log. Specifically, the data side may first analyze syntax and semantics of the data analysis logic of the data to be processed by using a scheduler of the federal computer commission of the data side to obtain at least one data processing tool used by the data analysis logic of the data to be processed; and then, processing the data to be processed by using the at least one data processing tool to obtain the corresponding result output of the data to be processed under each data processing tool.
Fig. 3 is a schematic structural diagram of a micro-isolation system provided in an embodiment of the present application. As shown in fig. 3, the micro-isolation system may include: a calculator (Agent A), a data party (Agent B) and a Coordinator (Coordinator); the computer and the data side transmit and receive data through respective federal computing agency; the federal calculation agency of the calculator is Proxy A; the federal calculation agency of the data side is Proxy B. Meanwhile, the calculator and the data side transmit and receive data with the coordinator respectively through respective cooperative calculation agents. The micro-isolation system in the application adopts Software-as-a-Service (SaaS) and a privatized distributed deployment architecture, wherein a coordinator is SaaS deployment, agents A and agents B adopt privatized deployment, and users only need to deploy agents belonging to the micro-isolation system when deploying, and then necessary system configuration is completed according to system guidance.
The micro-isolation scheme provided by the application is a novel isolation domain technology, and privacy computing technologies such as multi-party security computing and federal learning are not involved. The micro-isolation system provides a uniform entrance for performing remote safe operation on the isolation domain data, and realizes uniform management of lightweight and quick access on task submission, authorized execution and authorized result output of the isolation domain data through a WEB interface. For a business side, only the calculation node of federal calculation is deployed, and multi-party safety cooperative calculation on the basis that data cannot be out of a domain and data is available and invisible can be realized without great business transformation cost. The micro-isolation scheme in the application is recommended to be matched with technologies such as 'gatekeeper' and the like to be realized together so as to realize higher data security protection level. In this case, sensitive data on the data side and data processing facilities (Hadoop, Spark, Paddle, etc.) are all placed in the "gatekeeper" which can guarantee that data can only go in and out. The micro-isolation system is used as a unique way for isolating domain data, so that authorized approval of the micro-isolation system is required if domain data is required to be exported.
The data processing method provided by the embodiment of the application comprises the steps of firstly obtaining metadata of data to be processed from a data party through a federal calculation agent of the data party; then according to the metadata of the data to be processed, generating data analysis logic of the data to be processed on a federal calculation agent of a calculator; then sending the data analysis logic of the data to be processed to a data side; and enabling the data side to calculate the corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed. That is to say, the calculator in the present application does not need to acquire the sensitive data in the data side, the calculator first generates the data analysis logic according to the metadata of the sensitive data, and then the data side can calculate the result output of the sensitive data corresponding to each data processing tool based on the data analysis logic. In the existing data cooperation mode, the federal learning scheme and the isolation domain calculation scheme are not particularly efficient in the cross-organization cooperation scene. Because the technical means that the calculator generates the data analysis logic according to the metadata of the sensitive data and then the data calculator calculates the corresponding result output of the sensitive data under each data processing tool based on the data analysis logic is adopted, the technical problems that a federal learning scheme and an isolation domain calculation scheme in the prior art are not particularly efficient in a cross-organization cooperation scene are solved, and the technical scheme provided by the application can more efficiently realize cross-organization data cooperation, is safe and controllable, saves the cost and avoids resource waste; moreover, the technical scheme of the embodiment of the application is simple and convenient to implement, convenient to popularize and wide in application range.
Example two
Fig. 4 is a second flowchart of a data processing method provided in an embodiment of the present application, where the method may be performed by a data processing apparatus or an electronic device of a data side, where the apparatus or the electronic device of the data side may be implemented by software and/or hardware, and the apparatus or the electronic device of the data side may be integrated in any intelligent device with a network communication function. As shown in fig. 4, the data processing method may include the steps of:
s401, sending metadata of data to be processed to a calculator through a federal calculation agent of the calculator; wherein the metadata represents characteristics of the data to be processed.
In this step, the data side may send metadata of the data to be processed to the computation side through the federal computation agent of the computation side; wherein the metadata represents characteristics of the data to be processed. Specifically, the metadata in the embodiment of the present application includes, but is not limited to: the name of the data warehouse, the name of the data table stored in the data warehouse, the field name of the data table, the field type (String, Integer, etc.), the number of rows, etc.
S402, receiving data analysis logic of the data to be processed, which is generated based on the metadata of the data to be processed and sent by the calculator.
In this step, the data side may receive data analysis logic of the to-be-processed data generated based on the metadata of the to-be-processed data, which is sent by the computation side. Specifically, the calculator may analyze the received metadata of the data to be processed to obtain an analysis result of the metadata of the data to be processed; and then generating data analysis logic of the data to be processed according to the analysis result of the metadata of the data to be processed.
S403, calculating corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log.
In this step, the data side may calculate, based on the data analysis logic of the data to be processed, a corresponding result output of the data to be processed under each data processing tool; wherein the outcome yield comprises: result data and a yield log. Specifically, the data side may first analyze syntax and semantics of the data analysis logic of the data to be processed by using a scheduler of the federal computer commission of the data side to obtain at least one data processing tool used by the data analysis logic of the data to be processed; and then, processing the data to be processed by using the at least one data processing tool to obtain the corresponding result output of the data to be processed under each data processing tool.
In a specific embodiment of the present application, a data party may receive a data transmission instruction sent by a computer party through a pre-deployed micro-isolation system; wherein, the data transmission instruction comprises an identifier corresponding to at least one data processing tool; and if the data transmission instruction is approved by the data side, transmitting the corresponding result data of the data to be processed under at least one data processing tool to the calculating side through a pre-established file transmission channel. In addition, the data side can also receive a log viewing application sent by the computer side through a pre-deployed micro-isolation system; the log viewing application comprises an identifier corresponding to at least one data processing tool; and if the log checking application approval of the data side is passed, providing a corresponding output log of the data to be processed under at least one data processing tool for the calculating side through a preset application program interface.
According to the data processing method provided by the embodiment of the application, a data side sends metadata of data to be processed to a calculation side through a federal calculation agent of the calculation side; then receiving data analysis logic of the data to be processed, which is generated based on the metadata of the data to be processed and sent by a calculator; and calculating corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed. That is to say, the calculator in the present application does not need to acquire the sensitive data in the data side, the calculator first generates the data analysis logic according to the metadata of the sensitive data, and then the data side can calculate the result output of the sensitive data corresponding to each data processing tool based on the data analysis logic. In the existing data cooperation mode, the federal learning scheme and the isolation domain calculation scheme are not particularly efficient in the cross-organization cooperation scene. Because the technical means that the calculator generates the data analysis logic according to the metadata of the sensitive data and then the data calculator calculates the corresponding result output of the sensitive data under each data processing tool based on the data analysis logic is adopted, the technical problems that a federal learning scheme and an isolation domain calculation scheme in the prior art are not particularly efficient in a cross-organization cooperation scene are solved, and the technical scheme provided by the application can more efficiently realize cross-organization data cooperation, is safe and controllable, saves the cost and avoids resource waste; moreover, the technical scheme of the embodiment of the application is simple and convenient to implement, convenient to popularize and wide in application range.
EXAMPLE III
Fig. 5 is a third flow chart of the data processing method according to the embodiment of the present application. Further optimization and expansion are performed based on the technical scheme, and the method can be combined with the various optional embodiments. As shown in fig. 5, the data processing method may include the steps of:
s501, sending metadata of data to be processed to a calculator through a federal calculation agent of the calculator; wherein the metadata represents characteristics of the data to be processed.
And S502, receiving data analysis logic of the data to be processed, which is generated based on the metadata of the data to be processed and sent by the calculator.
S503, analyzing the grammar and the semantics of the data analysis logic of the data to be processed through the dispatcher of the federal computer agency of the data side to obtain at least one data processing tool used by the data analysis logic of the data to be processed.
In this step, the data side may analyze the syntax and semantics of the data analysis logic of the data to be processed through a scheduler of the federal computer commission of the data side, to obtain at least one data processing tool used by the data analysis logic of the data to be processed. Preferably, before analyzing the syntax and semantics of the data analysis logic of the data to be processed through the scheduler of the federal computing agency of the data party, the data party can also examine and approve the data analysis logic of the data to be processed through the examination and approval unit of the data party; and if the data analysis logic of the data to be processed passes the approval, submitting the data analysis logic of the data to be processed to the scheduler.
S504, processing the data to be processed by using at least one data processing tool to obtain corresponding result output of the data to be processed under each data processing tool; wherein the outcome yield comprises: result data and a yield log.
In one embodiment of the application, a data party (Agent B) accesses own data to a micro-isolation system through a federal computing agency, and informs a computing party (Agent A) of metadata of data to be processed (such as which fields of a data table are, whether the data type of each field is String or Integer, and the like); and the calculating party (Agent A) writes data analysis logic on the own federal calculation Agent according to the metadata of the data to be processed, which is sent by the data party (Agent B), and submits the data analysis logic through the federal calculation Agent. The data side (Agent B) checks the data analysis logic of the calculator (Agent a), and after the Approval (Approval) is passed, the data analysis logic is handed to a Scheduler (Scheduler) of the federal calculation agency to be analyzed and scheduled. The scheduler analyzes the data processing tools (Hadoop, Spark, sensor Flow, etc.) used by the data analysis logic by analyzing the syntax and semantics of the data analysis logic, automatically calls the related data processing tools according to the context semantics of the data analysis logic, and analyzes and processes the Data (DB) related to the data analysis logic.
In a specific embodiment of the present application, the data side (Agent B) outcome generation may include two parts, one is the final outcome data (Result) and the other is the log (Logs) generated by the data processing tool. If the calculating party (Agent A) wants to obtain the Result data (Result) of the data party (Agent B), a definite data transmission instruction needs to be created through the micro-isolation system, and after the data party (Agent B) is approved, the Result data (Result) is transmitted from the data party (Agent B) to the calculating party (Agent A) through a file transmission channel of the micro-isolation system.
In addition, if the calculator (Agent a) wants to obtain the log (Logs) of the data side (Agent B), the calculator (Agent a) needs to apply log viewing authorization to the data side (Agent B) through the micro-isolation system, and the log (Logs) can be viewed through a specific Application Programming Interface (API for short) after approval is obtained, so that real-time log viewing can be realized, and convenience is provided for system debugging and the like.
Compared with the traditional isolation domain scheme, the data processing method provided by the application has the main advantages that: 1. light-weighted deployment, visual easy-to-use: the micro-isolation scheme allows the isolation domain environment of the data side not to need large-scale transformation, and the computer side carries out modeling development and big data mining analysis on the isolation data remotely through the micro-isolation system and meets the safety isolation mechanism of the isolation domain. The federal calculation product flow realizes WEB interface operation, supports business flows of task submission, authorization, execution generation result and the like, and provides a lower threshold mode for cooperation of multi-party security data. Deployment of micro-isolated systems can be completed in 2 hours, whereas traditional isolated domain deployment delivery takes approximately 4 months. 2. And (3) white box examination, safety and controllability: all code executed within the micro-isolated domain requires prior approval by the data owner and is able to implement a "one-out-of-a-lot". The data owner knows and approves all operations of the data by the data analyst. Compared with the traditional isolation domain which only examines and approves the final data result, the data operation adopts a method of audit afterwards, and the micro-isolation belongs to 'white box' safety. 3. Resource reuse, avoid extravagant: a set of Hadoop, Spark and other big data basic implementation needs to be rebuilt in a third party in the traditional isolation domain, a large amount of hardware resources such as servers need to be invested, the BFC can fully utilize existing resources of a data party, and the data security and the data control are achieved based on a BFC security mechanism.
Compared with privacy calculation such as federal learning and multi-party safety calculation, the data processing method provided by the application has the main advantages that: 1. the data platform is native, and can run without modifying codes: the micro-isolation mode can fully utilize various data processing basic platforms of a data party, including the existing big data processing platforms (Hadoop, Spark, and the like), deep learning and machine learning platforms (Paddle, Tensflow, and the like), and traditional data analysis software (SAS, SPSS, Mathemetics, and the like), and user codes can operate almost without modification, so that the compatibility is good. 2. Hardware resources are fully utilized, and the operation efficiency is high: the micro-isolation mode does not depend on technologies such as cryptography, TEE and the like, so that local hardware resources (GPU, FPGA and the like) and a distributed framework can be fully utilized to carry out large-scale training and iteration and quickly generate results.
According to the data processing method provided by the embodiment of the application, a data side sends metadata of data to be processed to a calculation side through a federal calculation agent of the calculation side; then receiving data analysis logic of the data to be processed, which is generated based on the metadata of the data to be processed and sent by a calculator; and calculating corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed. That is to say, the calculator in the present application does not need to acquire the sensitive data in the data side, the calculator first generates the data analysis logic according to the metadata of the sensitive data, and then the data side can calculate the result output of the sensitive data corresponding to each data processing tool based on the data analysis logic. In the existing data cooperation mode, the federal learning scheme and the isolation domain calculation scheme are not particularly efficient in the cross-organization cooperation scene. Because the technical means that the calculator generates the data analysis logic according to the metadata of the sensitive data and then the data calculator calculates the corresponding result output of the sensitive data under each data processing tool based on the data analysis logic is adopted, the technical problems that a federal learning scheme and an isolation domain calculation scheme in the prior art are not particularly efficient in a cross-organization cooperation scene are solved, and the technical scheme provided by the application can more efficiently realize cross-organization data cooperation, is safe and controllable, saves the cost and avoids resource waste; moreover, the technical scheme of the embodiment of the application is simple and convenient to implement, convenient to popularize and wide in application range.
Example four
Fig. 6 is a schematic diagram of a first structure of a data processing apparatus according to an embodiment of the present application. As shown in fig. 6, the apparatus 600 includes: a data acquisition module 601, a logic generation module 602 and a logic sending module 603; wherein,
the data acquisition module 601 is configured to acquire metadata of data to be processed from a data party through a federal calculation agent of the data party; wherein the metadata represents characteristics of the data to be processed;
the logic generation module 602 is configured to generate a data analysis logic of the to-be-processed data on a federal calculation agent of the calculator according to the metadata of the to-be-processed data;
the logic sending module 603 is configured to send the data analysis logic of the data to be processed to the data side; enabling the data side to calculate corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log.
Further, the apparatus further comprises: a result receiving module 604 (not shown) for creating a data transmission instruction through the pre-deployed micro-isolation system; sending the data transmission instruction to the data side; wherein the data transmission instruction comprises an identifier corresponding to at least one data processing tool; and if the data transmission instruction is approved by the data side, receiving corresponding result data of the data to be processed under the at least one data processing tool through a pre-established file transmission channel.
Further, the apparatus further comprises: a log viewing module 605 (not shown in the figure) for sending a log viewing application to the data side through a pre-deployed micro-isolation system; wherein the log viewing application comprises an identifier corresponding to at least one data processing tool; and if the data side passes the log viewing application approval, viewing a corresponding output log of the data to be processed under the at least one data processing tool through a preset application program interface.
The data processing device can execute the method provided by the first embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method. For details of the data processing method provided in the first embodiment of the present application, reference may be made to the following description.
EXAMPLE five
Fig. 7 is a schematic diagram of a second structure of a data processing apparatus according to an embodiment of the present application. As shown in fig. 7, the apparatus 700 includes: a data sending module 701, a logic receiving module 702 and a result calculating module 703; wherein,
the data sending module 701 is configured to send metadata of data to be processed to a computing party through a federal computing agency of the computing party; wherein the metadata represents characteristics of the data to be processed;
the logic receiving module 702 is configured to receive data analysis logic, which is sent by the computing party and generated based on metadata of the to-be-processed data, of the to-be-processed data;
the result calculation module 703 is configured to calculate, based on the data analysis logic of the to-be-processed data, a corresponding result output of the to-be-processed data under each data processing tool; wherein the outcome yield comprises: result data and a yield log.
Further, the result calculation module 703 is specifically configured to analyze syntax and semantics of the data analysis logic of the to-be-processed data by using a scheduler of the federal calculation agent of the data party, so as to obtain at least one data processing tool used by the data analysis logic of the to-be-processed data; and processing the data to be processed by using the at least one data processing tool to obtain corresponding result output of the data to be processed under each data processing tool.
Further, the result calculating module 703 is further configured to examine and approve the data analysis logic of the data to be processed through an examination and approval unit of the data side; and if the data analysis logic of the data to be processed passes the approval, submitting the data analysis logic of the data to be processed to the scheduler.
Further, the apparatus further comprises: a result sending module 704 (not shown in the figure) for receiving a data transmission instruction sent by the computing party through a pre-deployed micro-isolation system; wherein the data transmission instruction comprises an identifier corresponding to at least one data processing tool; and if the data transmission instruction is approved by the data side, transmitting the corresponding result data of the data to be processed under the at least one data processing tool to the calculation side through a pre-established file transmission channel.
Further, the apparatus further comprises: a log providing module 705 (not shown in the figure) for receiving a log viewing application sent by the computing party through a pre-deployed micro-isolation system; wherein the log viewing application comprises an identifier corresponding to at least one data processing tool; and if the log viewing application is approved by the data side, providing a corresponding output log of the data to be processed under the at least one data processing tool for the calculation side through a preset application program interface.
The data processing device can execute the methods provided by the second embodiment and the third embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method. For details of the data processing method provided in the second embodiment and the third embodiment of the present application, reference may be made to the technical details not described in detail in this embodiment.
EXAMPLE six
The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.
FIG. 8 illustrates a schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 8, the apparatus 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)802 or a computer program loaded from a storage unit 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The calculation unit 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.
A number of components in the device 800 are connected to the I/O interface 805, including: an input unit 806, such as a keyboard, a mouse, or the like; an output unit 807 such as various types of displays, speakers, and the like; a storage unit 808, such as a magnetic disk, optical disk, or the like; and a communication unit 809 such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.
The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.
Claims (19)
1. A data processing method applied to a calculator, the method comprising:
acquiring metadata of data to be processed from a data party through a federal calculation agent of the data party; wherein the metadata represents characteristics of the data to be processed;
generating data analysis logic of the data to be processed on a federal calculation agent of the calculator according to the metadata of the data to be processed;
sending the data analysis logic of the data to be processed to the data side; enabling the data side to calculate corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log.
2. The method of claim 1, further comprising:
creating a data transmission instruction through a pre-deployed micro-isolation system; sending the data transmission instruction to the data side; wherein the data transmission instruction comprises an identifier corresponding to at least one data processing tool;
and if the data transmission instruction is approved by the data side, receiving corresponding result data of the data to be processed under the at least one data processing tool through a pre-established file transmission channel.
3. The method of claim 1, further comprising:
sending a log checking application to the data side through a pre-deployed micro-isolation system; wherein the log viewing application comprises an identifier corresponding to at least one data processing tool;
and if the data side passes the log viewing application approval, viewing a corresponding output log of the data to be processed under the at least one data processing tool through a preset application program interface.
4. A data processing method applied to a data side, the method comprising:
sending metadata of data to be processed to a computer side through a federal calculation agent of the computer side; wherein the metadata represents characteristics of the data to be processed;
receiving data analysis logic of the data to be processed, which is sent by the computer and generated based on metadata of the data to be processed;
calculating corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log.
5. The method of claim 4, the calculating, based on the data analysis logic of the data to be processed, a corresponding resulting outcome of the data to be processed under each data processing tool, comprising:
analyzing the syntax and semantics of the data analysis logic of the data to be processed through a dispatcher of a federal calculation agent of the data side to obtain at least one data processing tool used by the data analysis logic of the data to be processed;
and processing the data to be processed by using the at least one data processing tool to obtain corresponding result output of the data to be processed under each data processing tool.
6. The method of claim 5, prior to the analyzing syntax and semantics of the data analysis logic of the data to be processed by a scheduler of a federated computing agent of the data party, the method further comprising:
examining and approving the data analysis logic of the data to be processed through an examination and approval unit of the data side; and if the data analysis logic of the data to be processed passes the approval, submitting the data analysis logic of the data to be processed to the scheduler.
7. The method of claim 4, further comprising:
receiving a data transmission instruction sent by the calculator through a pre-deployed micro-isolation system; wherein the data transmission instruction comprises an identifier corresponding to at least one data processing tool;
and if the data transmission instruction is approved by the data side, transmitting the corresponding result data of the data to be processed under the at least one data processing tool to the calculation side through a pre-established file transmission channel.
8. The method of claim 4, further comprising:
receiving a log viewing application sent by the calculator through a pre-deployed micro-isolation system; wherein the log viewing application comprises an identifier corresponding to at least one data processing tool;
and if the log viewing application is approved by the data side, providing a corresponding output log of the data to be processed under the at least one data processing tool for the calculation side through a preset application program interface.
9. A data processing apparatus, the apparatus comprising: the device comprises a data acquisition module, a logic generation module and a logic sending module; wherein,
the data acquisition module is used for acquiring metadata of data to be processed from a data party through a federal calculation agent of the data party; wherein the metadata represents characteristics of the data to be processed;
the logic generation module is used for generating data analysis logic of the data to be processed on a federal calculation agent of the calculator according to the metadata of the data to be processed;
the logic sending module is used for sending the data analysis logic of the data to be processed to the data side; enabling the data side to calculate corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log.
10. The apparatus of claim 9, the apparatus further comprising: the result receiving module is used for creating a data transmission instruction through a pre-deployed micro-isolation system; sending the data transmission instruction to the data side; wherein the data transmission instruction comprises an identifier corresponding to at least one data processing tool; and if the data transmission instruction is approved by the data side, receiving corresponding result data of the data to be processed under the at least one data processing tool through a pre-established file transmission channel.
11. The apparatus of claim 9, the apparatus further comprising: the log viewing module is used for sending a log viewing application to the data side through a pre-deployed micro-isolation system; wherein the log viewing application comprises an identifier corresponding to at least one data processing tool; and if the data side passes the log viewing application approval, viewing a corresponding output log of the data to be processed under the at least one data processing tool through a preset application program interface.
12. A data processing apparatus, the apparatus comprising: the system comprises a data sending module, a logic receiving module and a result calculating module; wherein,
the data sending module is used for sending metadata of data to be processed to the computer party through a federal calculation agent of the computer party; wherein the metadata represents characteristics of the data to be processed;
the logic receiving module is used for receiving data analysis logic of the data to be processed, which is sent by the computer and generated based on the metadata of the data to be processed;
the result calculation module is used for calculating corresponding result output of the data to be processed under each data processing tool based on the data analysis logic of the data to be processed; wherein the outcome yield comprises: result data and a yield log.
13. The apparatus according to claim 12, wherein the result calculation module is specifically configured to analyze syntax and semantics of the data analysis logic of the to-be-processed data by a scheduler of a federal computer commission of the data party to obtain at least one data processing tool used by the data analysis logic of the to-be-processed data; and processing the data to be processed by using the at least one data processing tool to obtain corresponding result output of the data to be processed under each data processing tool.
14. The apparatus of claim 13, the result calculation module further configured to approve the data analysis logic of the data to be processed through an approval unit of the data side; and if the data analysis logic of the data to be processed passes the approval, submitting the data analysis logic of the data to be processed to the scheduler.
15. The apparatus of claim 12, the apparatus further comprising: the result sending module is used for receiving a data transmission instruction sent by the calculator through a pre-deployed micro-isolation system; wherein the data transmission instruction comprises an identifier corresponding to at least one data processing tool; and if the data transmission instruction is approved by the data side, transmitting the corresponding result data of the data to be processed under the at least one data processing tool to the calculation side through a pre-established file transmission channel.
16. The apparatus of claim 12, the apparatus further comprising: the log providing module is used for receiving a log viewing application sent by the calculator through a pre-deployed micro-isolation system; wherein the log viewing application comprises an identifier corresponding to at least one data processing tool; and if the log viewing application is approved by the data side, providing a corresponding output log of the data to be processed under the at least one data processing tool for the calculation side through a preset application program interface.
17. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3 or 4-8.
18. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any of claims 1-3 or 4-8.
19. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-3 or 4-8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110231850.2A CN112860970B (en) | 2021-03-02 | 2021-03-02 | Data processing method and device, electronic equipment and storage medium |
| PCT/CN2021/103669 WO2022183644A1 (en) | 2021-03-02 | 2021-06-30 | Data processing method and apparatus, and electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110231850.2A CN112860970B (en) | 2021-03-02 | 2021-03-02 | Data processing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112860970A true CN112860970A (en) | 2021-05-28 |
| CN112860970B CN112860970B (en) | 2024-03-12 |
Family
ID=75991055
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110231850.2A Active CN112860970B (en) | 2021-03-02 | 2021-03-02 | Data processing method and device, electronic equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112860970B (en) |
| WO (1) | WO2022183644A1 (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113537508A (en) * | 2021-06-18 | 2021-10-22 | 百度在线网络技术（北京）有限公司 | Federal calculation processing method and device, electronic equipment and storage medium |
| CN113704314A (en) * | 2021-08-04 | 2021-11-26 | 百度在线网络技术（北京）有限公司 | Data analysis method and device, electronic equipment and storage medium |
| WO2022183644A1 (en) * | 2021-03-02 | 2022-09-09 | 百度在线网络技术（北京）有限公司 | Data processing method and apparatus, and electronic device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108241722A (en) * | 2016-12-23 | 2018-07-03 | 北京金山云网络技术有限公司 | A kind of data processing system, method and device |
| US10437670B1 (en) * | 2018-05-24 | 2019-10-08 | International Business Machines Corporation | Metadata hardening and parity accumulation for log-structured arrays |
| CN110781165A (en) * | 2019-10-10 | 2020-02-11 | 支付宝(杭州)信息技术有限公司 | Method, device and equipment for processing service data |
| CN111914038A (en) * | 2020-09-21 | 2020-11-10 | 北京百度网讯科技有限公司 | Federal calculation method, device, equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112860970B (en) * | 2021-03-02 | 2024-03-12 | 百度在线网络技术（北京）有限公司 | Data processing method and device, electronic equipment and storage medium |
-
2021
- 2021-03-02 CN CN202110231850.2A patent/CN112860970B/en active Active
- 2021-06-30 WO PCT/CN2021/103669 patent/WO2022183644A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108241722A (en) * | 2016-12-23 | 2018-07-03 | 北京金山云网络技术有限公司 | A kind of data processing system, method and device |
| US10437670B1 (en) * | 2018-05-24 | 2019-10-08 | International Business Machines Corporation | Metadata hardening and parity accumulation for log-structured arrays |
| CN110781165A (en) * | 2019-10-10 | 2020-02-11 | 支付宝(杭州)信息技术有限公司 | Method, device and equipment for processing service data |
| CN111914038A (en) * | 2020-09-21 | 2020-11-10 | 北京百度网讯科技有限公司 | Federal calculation method, device, equipment and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022183644A1 (en) * | 2021-03-02 | 2022-09-09 | 百度在线网络技术（北京）有限公司 | Data processing method and apparatus, and electronic device and storage medium |
| CN113537508A (en) * | 2021-06-18 | 2021-10-22 | 百度在线网络技术（北京）有限公司 | Federal calculation processing method and device, electronic equipment and storage medium |
| CN113537508B (en) * | 2021-06-18 | 2024-02-02 | 百度在线网络技术（北京）有限公司 | Processing method and device for federal calculation, electronic equipment and storage medium |
| CN113704314A (en) * | 2021-08-04 | 2021-11-26 | 百度在线网络技术（北京）有限公司 | Data analysis method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022183644A1 (en) | 2022-09-09 |
| CN112860970B (en) | 2024-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112860970B (en) | Data processing method and device, electronic equipment and storage medium | |
| US10628212B2 (en) | Incremental parallel processing of data | |
| CN112422638A (en) | Data real-time stream processing method, system, computer device and storage medium | |
| CN108984202B (en) | Electronic resource sharing method and device and storage medium | |
| KR20220151132A (en) | Method and device for testing network equipment | |
| CN110780879A (en) | Decision execution method, device, equipment and medium based on intelligent compiling technology | |
| CN114157701A (en) | Task testing method, device, equipment and storage medium | |
| CN113360144A (en) | Auxiliary processing method, device, storage medium and program product for software development | |
| CN113037489B (en) | Data processing method, device, equipment and storage medium | |
| CN115509522A (en) | Interface arranging method and system for low-code scene and electronic equipment | |
| US20150058279A1 (en) | Control data driven modifications and generation of new schema during runtime operations | |
| US12056492B2 (en) | Method and apparatus for generating dependence graph, device, storage medium and program product | |
| CN113032817B (en) | Data alignment method, device, equipment and medium based on block chain | |
| CN116305324A (en) | Host safety protection method, device, equipment and storage medium | |
| CN113590217B (en) | Function management method and device based on engine, electronic equipment and storage medium | |
| CN113535574B (en) | Automatic generation method, device, equipment and medium for test user data | |
| CN115481594A (en) | Score board implementation method, score board, electronic device and storage medium | |
| CN114237686A (en) | Installation package generation method and device, electronic equipment and storage medium | |
| CN112486460A (en) | Method, system, device and medium for automatically importing interface document | |
| CN111158649A (en) | Method and device for configuring multi-level parameters | |
| US20240037436A1 (en) | System for processing quantum task, method for processing quantum task and related apparatuses | |
| CN116470968B (en) | Ground test method and device for communication function of aerospace science system | |
| CN116701053B (en) | Method, device, equipment and medium for restoring data backup of production environment database | |
| CN114416166A (en) | Mock debugging method, apparatus, medium and device | |
| CN114185864A (en) | Operation log management method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |