CN113935390A - Data processing method, system, device and storage medium - Google Patents
Data processing method, system, device and storage medium Download PDFInfo
- Publication number
- CN113935390A CN113935390A CN202010606801.8A CN202010606801A CN113935390A CN 113935390 A CN113935390 A CN 113935390A CN 202010606801 A CN202010606801 A CN 202010606801A CN 113935390 A CN113935390 A CN 113935390A
- Authority
- CN
- China
- Prior art keywords
- model
- training
- data
- data processing
- strategy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 49
- 238000012549 training Methods 0.000 claims abstract description 241
- 238000012545 processing Methods 0.000 claims abstract description 146
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000004891 communication Methods 0.000 claims abstract description 13
- 230000004927 fusion Effects 0.000 claims description 46
- 238000004422 calculation algorithm Methods 0.000 claims description 38
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000011156 evaluation Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims 2
- 230000006835 compression Effects 0.000 description 36
- 238000007906 compression Methods 0.000 description 36
- 238000007405 data analysis Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 230000002093 peripheral effect Effects 0.000 description 4
- 230000003044 adaptive effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000010248 power generation Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 238000007477 logistic regression Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013468 resource allocation Methods 0.000 description 2
- 208000033748 Device issues Diseases 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/20—Ensemble learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2113/00—Details relating to the application field
- G06F2113/04—Power grid distribution networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Human Resources & Organizations (AREA)
- General Engineering & Computer Science (AREA)
- Strategic Management (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Entrepreneurship & Innovation (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Computer Hardware Design (AREA)
- Geometry (AREA)
- Economics (AREA)
- Educational Administration (AREA)
- Development Economics (AREA)
- Data Mining & Analysis (AREA)
- Medical Informatics (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Game Theory and Decision Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the technical field of communication, and discloses a data processing method, a data processing system and a storage medium. In the invention, the data processing method is applied to the data processing equipment and the N data control equipment in the model training system, and the data processing equipment is respectively in communication connection with the N data control equipment. The method applied to the data processing device comprises the following steps: receiving N intermediate models uploaded by N data control devices; wherein, the N intermediate models are obtained by training N data processing devices; and generating a comprehensive model according to the N intermediate models. The method applied to the data control device comprises the following steps: training a basic model to be trained by using local training data to obtain an intermediate model; and reporting the intermediate model to the data control equipment for the data control equipment to generate a comprehensive model. By the technical means, the data processing equipment is prevented from directly contacting the data of all the data control equipment, the model training precision is guaranteed, and meanwhile the privacy of the model training data is improved.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a data processing method, system, device, and storage medium.
Background
As communication technologies develop, data collection becomes easier and more data can be analyzed and processed by network users. From the viewpoint of the quality of model training of the data processing apparatus, the larger the amount of data used for model training, the better the accuracy and generalization index of the model. Currently, the data processing device will acquire raw collected data from different data control devices through a network to train a model according to needs, wherein the different data control devices can be distributed in different regions and used for collecting and storing data of the regions where the different data control devices are located. For example, the power internet of things systems in different regions can be understood as different data control devices, and the electronic devices of the power internet of things systems for actually executing the power scheduling scheme can be understood as data processing devices; for another example, medical systems in different regions may be understood as different data control devices, and a dispatch department of a medical system may be understood as a data processing device.
Since the data processing apparatus can be exposed to a large amount of raw collected data uploaded by the data control apparatus, a series of problems may be caused, such as private information of an individual or confidential information of a region being leaked.
Disclosure of Invention
The embodiment of the invention aims to provide a data processing method, a data processing system and a storage medium.
In order to solve the above technical problem, an embodiment of the present invention provides a data processing method, which is applied to a data processing device, where the data processing device is in communication connection with N data control devices respectively; the method comprises the following steps: receiving N intermediate models uploaded by N data control devices; the N intermediate models are obtained by training a base model to be trained by N data processing devices by using local training data; and generating a comprehensive model according to the N intermediate models.
The embodiment of the invention also provides a data processing method, which is applied to the data control equipment, and the data control equipment is in communication connection with the data processing equipment; the method comprises the following steps: training a basic model to be trained by using local training data to obtain an intermediate model; and reporting the intermediate model to the data control equipment for the data control equipment to generate a comprehensive model.
The embodiment of the present invention further provides a model training system, including: the data processing system comprises a central module arranged in the data processing equipment and N agent modules respectively arranged in N data control equipment; the central module is in communication connection with the N agent modules respectively; the central module is used for receiving N intermediate models uploaded by N data control devices; the N intermediate models are obtained by training a base model to be trained by N data processing devices by using local training data; generating a comprehensive model according to the N intermediate models; the agent module is used for training a basic model to be trained by utilizing local training data to obtain an intermediate model; and reporting the intermediate model to the data control equipment for the data control equipment to generate a comprehensive model.
An embodiment of the present invention further provides an apparatus, including: at least one processor, and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data processing method as described above.
Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which when executed by a processor implements the data processing method as described above.
Compared with the prior art, the method and the device have the advantages that firstly, N data control devices respectively use stored training data to train a basic model to be trained to obtain an intermediate model, then the N data control devices respectively upload the intermediate model obtained through training to the data processing devices, and the data processing devices fuse the N received intermediate models to obtain the comprehensive model. In the whole training process of the model, a data processor does not directly contact any training data, so that the risk of disclosure of privacy data is avoided. Meanwhile, the model can still be trained by using the data of all the data control devices, and the precision and generalization index of the comprehensive model obtained by training are not influenced.
Drawings
One or more embodiments are illustrated by the corresponding figures in the drawings, which are not meant to be limiting.
Fig. 1 is a flowchart of a data processing method applied to a data processing apparatus according to a first embodiment of the present invention;
fig. 2 is a flowchart of a data processing method applied to a data processing apparatus according to a second embodiment of the present invention;
fig. 3 is a flowchart of a data processing method applied to a data control apparatus according to a third embodiment of the present invention;
fig. 4 is a flowchart of a data processing method applied to a data control apparatus according to a fourth embodiment of the present invention;
fig. 5 is a flowchart of a data processing method applied to a data control apparatus according to a fifth embodiment of the present invention;
fig. 6 is a flowchart of a data processing method applied to a data control apparatus according to a fifth embodiment of the present invention;
FIG. 7 is a schematic structural diagram of a model training system according to a sixth embodiment of the present invention;
fig. 8 is a schematic structural diagram of a data control device according to a seventh embodiment and an eighth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that in various embodiments of the invention, numerous technical details are set forth in order to provide a better understanding of the present application. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments. The following embodiments are divided for convenience of description, and should not constitute any limitation to the specific implementation of the present invention, and each of them is reasonably incorporated into each other and referred to each other without contradiction.
It should be noted that although functional blocks are partitioned in a schematic diagram of an apparatus and a logical order is shown in a flowchart, in some cases, the steps shown or described may be performed in a different order than the partitioning of blocks in the apparatus or the order in the flowchart. The terms "first," "second," and the like in the description and in the above-described figures are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
A first embodiment of the present invention relates to a data processing method applied to a data processing device, where the data processing device is a terminal device for performing data analysis processing by using a model trained from mass data; the data control device is a terminal device for model training and managing stored data. The training data mentioned in this embodiment are distributed in N data control devices, respectively, where N is a natural number greater than or equal to 1. The data processing equipment is respectively in communication connection with the N data control equipment; the data processing equipment receives N intermediate models uploaded by N data control equipment; the N intermediate models are obtained by training a basic model to be trained by N data control devices by using local training data; and generating a comprehensive model according to the N intermediate models.
The following describes the implementation details of the data processing method in this embodiment with reference to the drawings, and the following description is only provided for the convenience of understanding and is not necessary for implementing the present embodiment. The embodiment can be widely applied to a distributed data storage architecture and is used for ensuring the data privacy safety of a large amount of data; the distributed data storage structure may be used to store patient information in a medical system, student information in an educational system, vehicle information in a traffic prediction system, environmental data in an electrical power system, and the like. Taking a power grid architecture as an example, different regions are provided with independent power internet of things systems, the power internet of things systems are formed based on internet of things devices, and power scheduling in the regions is achieved by reading and storing information collected by the internet of things devices such as a camera, a sound sensor, a gas sensor, a temperature and humidity sensor, a pressure sensor, an optical sensor, a magnetic field sensor, a motion sensor, an RFRD radio frequency identification system and a GPS terminal. Because the electric power system scheduling needs a reasonable scheduling scheme, a large amount of data is adopted to train an electric power scheduling model as the scheduling scheme, so that the use efficiency of the power grid can be greatly improved. In the prior art, an electric power internet of things system acquires characteristic data of each region, and trains a model by using the acquired characteristic data of each region; but there is a risk of data compromise in the transmission of the characteristic data.
The data processing apparatus may be either another apparatus independent of the data control apparatus or one data control apparatus selected from a plurality of data control apparatuses, the selected data control apparatus performing the function of the data processing apparatus. Similarly, taking the power internet of things system as an example, the power internet of things system of each region can be used as a data control device, and the electronic device in the power dispatching department responsible for executing the power dispatching scheme can be a device completely independent of the power internet of things system of each region; or, one power internet of things system can be selected from power internet of things systems in various regions to serve as a power dispatching department, namely, the function of the data processing equipment is executed. If the data processing device is selected from a plurality of data control devices, a suitable data control device can be selected as the data processing device according to a preset rule, for example, the data control device with the best hardware performance is selected as the data processing device.
An execution subject of the data processing method in this embodiment is a data processing device, and a specific flow is shown in fig. 1, and includes:
Specifically, the N intermediate models received by the data processing apparatus are trained by the N data control apparatuses, respectively. The data for training are distributed in N data control devices, each data control device trains a basic model to be trained by using training data stored locally, and the intermediate model is uploaded to the data processing device after the intermediate model is obtained. The basic model to be trained can be carried by the data processing equipment through a model training task in advance and issued to the data control equipment, and the model training task is used for indicating the data control equipment to carry out model training.
And a step 102 of a basic model to be trained, fusing the N intermediate models based on a fusion algorithm to generate a comprehensive model.
Specifically, since the training data stored in each data control device is different, the accuracy of the intermediate model obtained by training is different from the generalization index, and in order to obtain a model in which both the accuracy and the generalization index can satisfy the requirement, the data processing device generates a comprehensive model for finally performing data analysis processing by fusing the N intermediate models. Compared with the prior art, the training data for model training in the embodiment are respectively stored in different data control devices in a decentralized mode according to the main bodies of the data, the process setting of the model training is carried out by the data control devices, the data processing devices only carry out fusion generation of the comprehensive models and cannot directly contact the training data managed by the data control devices, and therefore physical isolation of the training data is achieved, and the purpose of data privacy protection is effectively achieved. Meanwhile, the total amount of training data used for model training is not changed, and the precision and generalization index of the model finally obtained by the data processing equipment can be ensured.
A second embodiment of the present invention relates to a data processing method, and the execution subject of this embodiment is a data processing apparatus. In this embodiment, the data processing apparatus may evaluate an index such as accuracy of the integrated model after generating the integrated model.
The following describes in detail the implementation of the data processing method in this embodiment with reference to the accompanying drawings, and the specific flow is shown in fig. 2, and includes:
Specifically, the data processing device issues a model training task to the data control device, where the model training task is used to instruct the data control device to perform model training, and the model training task includes at least one of the following: strategy guide information and a basic model to be trained. The strategy guidance information is used for guiding the data processing equipment to determine a training strategy adopted for training the basic model to be trained.
In one example, the strategy guidance information in the model training task is composed of parsed operation instructions. Before the data processing equipment issues the model training tasks to the N data control equipment, the data processing equipment determines the data dictionaries corresponding to the N data control equipment according to the mapping relation between the pre-stored data dictionaries and the data control equipment. And then analyzing a preset operation instruction according to a data dictionary corresponding to the data control equipment to obtain strategy guidance information.
Further, the parsed operation instruction is an operation primitive, for example:
primitive for redundant data removal operation: data table name: a UserName; redundant data removal ratio: adaptive redundant data removal parameters: value1, value 2; expected completion time: 1 second;
primitive to perform compression point generation operation: executing a data table: UserInfo; compression ratio setting: self-adapting; expected completion time: 1 second;
the primitives for performing a data training operation for a data table are: data table name: list of UserInfoDemo fields: f1, F2, F3; a label field: l; the algorithm is as follows: performing logistic regression; expected completion time: for 10 seconds.
Since the data bodies of the respective data control apparatuses are different in actual applications, the types of operation primitives that can be recognized by the respective data control apparatuses are also different. Therefore, before the data processing device analyzes the preset operation instruction, the data dictionary of each data control device needs to be acquired, then the mapping relationship between the data control device and the data dictionary is stored, when the data processing device analyzes the preset operation instruction, the corresponding data dictionary is searched in the mapping relationship between the data control device and the data dictionary, and then the operation instruction is converted into the operation primitive which can be recognized by the data control device, so that the data processing device can be compatible with all different data control devices.
This step is similar to step 101 in the first embodiment of the present invention, and details of the implementation have been specifically described in the first embodiment, and are not described herein again.
And step 203, fusing the N intermediate models based on a fusion algorithm to generate a comprehensive model.
Specifically, in this embodiment, the data processing device merges the intermediate models uploaded by the data control device according to a preset merging algorithm, and a specific merging process is affected by the merging parameters.
In one example, the data processing device receives training strategies of N intermediate models uploaded by the data control device before merging the intermediate models, where the training strategies of the intermediate models refer to training strategies adopted for training the basic model to be trained. In the process of combining the intermediate models, firstly, fusion parameters of the N intermediate models in a preset fusion algorithm are determined according to training strategies of the N intermediate models. And then fusing the N intermediate models according to the fusion parameters of the N intermediate models and a fusion algorithm to obtain a comprehensive model. When the data processing equipment merges the intermediate models, the training strategy of the intermediate models of the data processing equipment is taken into consideration, and the fusion parameters of the intermediate models in the fusion algorithm are determined according to the training strategy of the intermediate models, so that the final precision and generalization index of the models are influenced by the actual training strategy of the data control equipment, and the accuracy of the integrated model obtained by fusion can be improved.
In another example, the data control device may upload the training strategy at the same time as uploading the intermediate model to the data processing device, i.e. the data processing device receives the intermediate model and the training strategy of the intermediate model at the same time.
In one example, the fusion algorithm may adopt a weighting algorithm, where the fusion parameter is a weight value of each intermediate model in the weighting algorithm, and the weight value of each intermediate model is determined according to a training strategy. When the training strategy adopted by the data control equipment is a single strategy, determining the weight value of each intermediate model in the weighting algorithm according to the single training strategy; if multiple strategies are adopted for model training, the weight values of the intermediate models in the weighting algorithm are determined by comprehensively considering all the training strategies. The integrated model obtained by fusion can have better precision by adopting a weighting algorithm.
Further, the training strategy of the intermediate model may include: the method comprises a privacy protection strategy, a data training strategy, a compression point generation strategy, a redundant data removal strategy, a performance index ratio calculation strategy and an adaptive resource allocation strategy.
The data control device may adopt a performance index ratio calculation strategy to obtain the current available hardware resource index of each data control device. The strategy monitors various available resources of the current data control equipment, such as network bandwidth, hard disk I/O conditions, CPU utilization rate and the like, by using a system API (application programming interface), so as to obtain the performance index of the current data control equipment. In order to obtain the performance condition of the current data control equipment relative to other data control equipment, the efficiency of the whole system is improved better. The agent module sends the local hardware resource status to the data processing device at regular time, after the data processing device collects all resource indexes, the relative resource index, namely the performance index ratio, of each agent module is calculated, and finally the relative resource index is sent back to the data control device. When the model fusion is performed by using the weighting algorithm, if the performance index of the data control device is higher, the weight value of the intermediate model obtained by training the data control device in the fusion algorithm is higher.
In addition, if the data control device performs training of the intermediate model, other training strategies are also used, such as a compression point generation strategy for compressing data. The compression points include two types, namely coarse-grained compression points and fine-grained compression points. And when the coarse-grained compression points are not enough to judge the influence of the current compression point on updating of the fusion parameters, calculating the fine-grained compression points corresponding to the coarse-grained compression points to obtain more accurate fusion parameters.
In one example, the adopted fusion algorithm is a weighting algorithm, and the training strategy based on the intermediate model sets the weight value of the intermediate model, which may include the following ways: for the intermediate model obtained by training the compression point generation strategy, the weight value of the intermediate model can be set to be relatively lower; or, for the two intermediate models, if both are trained by using the compression point generation strategy, the intermediate model trained by using the fine-grained compression point mode is set to have a smaller weight value than the intermediate model trained by using the coarse-grained compression point mode.
Other training strategies mentioned in this example are explained below:
and (3) a data training strategy: an algorithm used by the agent module in data training may be specified and executed. The specific choice of the strategy is various, such as selecting a convolutional neural network, performing logistic regression, performing reinforcement learning and other algorithms. The data controller can set according to the requirement and receive the strategy guidance information sent by the data processing equipment. Firstly, the strategy needs to determine the current training strategy and determine the data training algorithm which needs to be used currently. And then allocating resources according to the setting, and carrying out training operation. And finally generating an intermediate model through a data training strategy.
Redundant data removal strategy: and performing a redundant data removal strategy by using the coarse-grained compression points and the fine-grained compression points generated by the compression point generation strategy. Firstly, calculating the influence value of all coarse-grained compression points on the updating of the data training parameters, and judging whether the data of the compression points are redundant or not according to the influence value. And traversing all coarse-grained compression points, and reserving the compression points with larger influence values. And meanwhile, performing fine-grained compression on the coarse-grained compression points with smaller influence values to generate a plurality of fine-grained compression points, then judging the influence values of the current fine-grained compression points on parameter updating, removing redundant data and reserving valuable data.
Adaptive resource allocation strategy: and calculating the resources and the training data amount which should be used when the current agent module carries out data training by using the performance index ratio of the current data control equipment. The strategy can solve the problem that the data training progress between the agent modules is not synchronous, and the overall training efficiency is improved.
In particular, in order to enable the comprehensive model ultimately used for the data analysis process to meet the requirements. After the data processing equipment merges the intermediate models to obtain the comprehensive model, the comprehensive model is evaluated, and whether the precision or generalization index of the model meets a preset threshold value or not is judged. When the precision of the comprehensive model meets a preset threshold value, the comprehensive model is reserved, the comprehensive model can be issued to each data control device, and the data control device can adopt the comprehensive model to perform data analysis processing. And when the precision of the comprehensive model does not meet the preset threshold value, adding the comprehensive model serving as a basic model to be trained into the model training task and issuing the basic model to the data control equipment for the data control equipment to perform model training again so as to further improve the precision or generalization index of the model and the like.
In a specific implementation, when the embodiment is applied to different service scenarios, the model precision evaluation modes are not necessarily the same, and if the scenario is an index prediction scenario, evaluation indexes such as RMSE (root mean square error), MAE (mean absolute error), R-square and the like can be adopted; if the scene is classified, evaluation indexes such as accuracy, recall rate, accuracy, error rate, F1 and the like can be adopted; if the clustering is performed, the indexes can be evaluated by Davies-Bouldin Index (Davison Bouldin Index), Dunn Validity Index (Dunn Index) and the like. The above indexes are purely mathematical, and in practical applications, there are many variations of precision evaluation indexes, for example, in the application scenario of the power internet of things system mentioned below, the fluctuation rate of the invalid power generation amount in a period of time can be used as the evaluation index of the power scheme. In terms of the characteristics of the power system, that is, when the invalid power generation amount of the power system in a period of time is small and there is no large fluctuation, it means that the accuracy of the power scheduling scheme as the comprehensive model is good.
In one example, if the model training task includes the strategy guidance information, the data processing device may continue to use the strategy guidance information in the previous model training task while adding the previously generated integrated model as the basic model to be trained in the model training task; the strategy guidance information can be modified according to the requirement of data analysis and processing so as to better improve the efficiency of model training and the precision or generalization index of the intermediate model obtained by training.
The present embodiment is further explained below with the power internet of things system as an application scenario.
In this example, the data processing device is an electronic device that executes a power scheduling scheme in the power internet of things system; the data control devices are power systems in different regions.
Firstly, the data control equipment acquires information such as primitive definition supporting data operation, an access identifier of a data processor, a policy descriptor and the like from the data processing equipment; the data control equipment sets the level of the data processing equipment, the execution action flow corresponding to the operation primitive, the data acquisition shielding rule and other strategies; the data processing apparatus acquires data dictionary information from a plurality of data control apparatuses.
The data processing equipment takes the current power scheduling scheme as a basic model to be trained, and model training tasks carrying the basic model to be trained are sent to the data control equipment. Specifically, for a plurality of regions, since the power generation amount is determined by the local power plants, it is necessary to reasonably distribute the power scheduling to adapt to the domestic power of the general public and the production power of various production departments (such as factories). Therefore, a default power scheduling scheme is used as a basic model to be trained, and the basic model to be trained and task starting information are sent to each data control device.
After receiving the basic model to be trained issued by the data processing equipment, the data control equipment deploys the power scheduling scheme of the local area according to the power scheduling method in the model, so that the power scheduling scheme can be deployed in time after the data control equipment obtains a new model each time. And then, carrying out data training according to strategy guide information in the model training task and local rules. The used data set is information collected by a sensing layer of a local area power Internet of things system; the data training strategy is selected by the data control device in a self-adaptive mode, and the available strategies comprise the following strategies: a compression point generation strategy, a redundant data removal strategy, a performance index ratio calculation strategy and the like; the data methods are uniformly specified by the data processing equipment, such as a classification algorithm, a clustering algorithm, a multi-layer neural network algorithm and the like.
After one data training, the data control device obtains the currently trained intermediate model and the strategy used by the training. Firstly, the data control equipment carries out privacy protection operation on the intermediate model by using a self-adaptive privacy protection strategy, and the purpose of doing so is to prevent local power use condition and a power scheduling scheme from being leaked out in the transmission process, so that the privacy is unsafe. Then each data control device transmits the local intermediate model to the data processing device, so that the data processing device combines each intermediate model, and the generalization of the finally obtained comprehensive model is ensured.
And then merging the models of the data processing equipment and evaluating the models, merging the intermediate models after the data processing equipment obtains the intermediate models of all the current data control equipment, and obtaining a comprehensive model. And evaluating the comprehensive model (namely the power dispatching scheme) by using the test data of the data processing equipment. If the comprehensive model meets the requirements, the comprehensive model is issued; otherwise, the model training task is issued again, and the data control equipment retrains a new intermediate model according to the new strategy guidance information.
It should be noted that the above examples in the present embodiment are only for easy understanding, and do not limit the technical scheme of the present invention.
Compared with the prior art, in the embodiment, the training data for model training is respectively stored in different data control devices in a decentralized manner according to the main bodies of the data, the process setting of the model training is performed by the data control devices, and the data processing devices only perform fusion generation of the comprehensive model and cannot directly contact the training data managed by the data control devices. In addition, when the intermediate models are combined, the training strategies of the intermediate models are taken into consideration, and the fusion parameters of the intermediate models in the fusion algorithm are determined according to the training strategies of the intermediate models, so that the final precision and generalization indexes of the models are controlled by the data control equipment, and the privacy safety of the data stored by the data control equipment is further improved.
A third embodiment of the present invention relates to a data processing method applied to a data control device, which is a terminal device for model training and managing stored data; the data processing device is a terminal device for performing data analysis processing by using a model trained by mass data. The training data mentioned in this embodiment are distributed in N data control devices, respectively, where N is a natural number greater than or equal to 1. The data control equipment is in communication connection with the data processing equipment; the method comprises the following steps: training a basic model to be trained by using local training data to obtain an intermediate model; and reporting the intermediate model to the data control equipment for the data control equipment to generate a comprehensive model.
The following describes the implementation details of the data processing method in this embodiment with reference to the drawings, and the following description is only provided for the convenience of understanding and is not necessary for implementing the present embodiment.
The main execution body of the data processing method in this embodiment is a data control device, and a specific flow is shown in fig. 3, and includes:
Specifically, the data control device stores locally a large amount of training data for model training. The basic model to be trained is an initial model pre-stored in the data control equipment, and an intermediate model is obtained after training data is input into the basic model to be trained for training.
In one example, the basic model to be trained is generated and issued by the data processing device, and the data control device trains the basic model to be trained to obtain an intermediate model.
Specifically, since the data subjects of the plurality of data control apparatuses are different and the samples of the training data are also different, the accuracy and the generalization index of the intermediate model obtained by training of different data control apparatuses are different. In order to meet the requirements of the data processing equipment on the accuracy of the finally adopted model and the analysis and processing of the generalized index data, the data control equipment uploads the trained intermediate models to the data processing equipment, and the data processing equipment combines the intermediate models to generate a comprehensive model.
It can be seen that this embodiment is a data processing method applied to a data control device, which is implemented in cooperation with the first embodiment and the second embodiment of the present invention, and the technical details mentioned in the first embodiment of the present invention can also be implemented in this embodiment, and are not described herein again in order to reduce the repetition.
Compared with the prior art, the training data for model training in the embodiment are respectively stored in different data control devices in a decentralized mode according to the main bodies of the data, the process setting of the model training is carried out by the data control devices, the data processing devices only carry out fusion generation of the comprehensive models and cannot directly contact the training data managed by the data control devices, and therefore physical isolation of the training data is achieved, and the purpose of data privacy protection is effectively achieved. Meanwhile, the total amount of training data used for model training is not changed, and the precision and generalization index of the model finally obtained by the data processing equipment can be ensured.
A fourth embodiment of the present invention relates to a data processing method, and the main difference between this embodiment and the third embodiment of the present invention is that when a data control device trains a basic model to be trained by using training data, a training strategy of an intermediate model is determined according to a local strategy execution rule, and after the training is completed, the training strategy of the intermediate model obtained by the training is uploaded to a data processing module.
The main execution body of the data processing method in this embodiment is a data control device, and a specific flow is shown in fig. 3, and includes:
Specifically, in this embodiment, the data control device receives a model training task issued by the data processing device before training the basic model to be trained, and the model training task may carry policy guidance information and the basic model to be trained. And when the basic model to be trained by the data control equipment is generated by the data processing equipment, the basic model to be trained is carried and issued by the model training task.
Specifically, after the data control device receives the model training task, if the model training task only contains the strategy guidance information, the training strategy of the basic model to be trained is determined according to the strategy guidance information and the locally preset strategy execution rule, and at this time, the basic model to be trained is the locally preset model of the data control device.
In one example, if the model training task only includes the basic model to be trained, the data control device determines the training strategy of the basic model to be trained according to a locally preset strategy execution rule, and then trains the basic model to be trained in the model training task according to the training strategy.
In one example, if the model training task includes both the strategy guidance information and the basic model to be trained, the data control device determines the training strategy of the basic model to be trained according to the strategy guidance information and the locally preset strategy execution rule, and then trains the basic model to be trained in the model training task according to the training strategy.
And step 403, reporting the intermediate model to the data processing equipment for the data processing equipment to generate a comprehensive model.
Specifically, in this embodiment, the data control device reports both the intermediate model and the training strategy of the intermediate model to the data processing device. The data control device may report the training strategy of the intermediate model to the data processing device in advance, or may report the training strategies of the intermediate model and the intermediate model to the data processing device at the same time. And the data processing equipment determines the fusion parameters of the intermediate model in the fusion algorithm according to the training strategy of the intermediate model. Namely, when the data processing device merges the intermediate models, the training strategy of the intermediate models of the data processing device is taken into consideration, and the fusion parameters of the intermediate models in the fusion algorithm are determined according to the training strategy of the intermediate models, so that the final precision and generalization index of the models are influenced by the actual training strategy of the data control device, and the privacy security of the data stored in the data control device is further improved.
A fifth embodiment of the present invention relates to a data processing method, and the execution subject of this embodiment is a data control apparatus. The difference between this embodiment and the third embodiment of the present invention is that after the data control device generates the intermediate model, the data control device evaluates the accuracy of the intermediate model to determine whether to continue to train the intermediate model with higher accuracy to perform the data analysis processing local to the data control device.
The following describes in detail the implementation of the data processing method in this embodiment with reference to the accompanying drawings, and a specific flow is shown in fig. 5, and includes:
Specifically, the initial basic model to be trained in this embodiment is preset locally by the data control device, and the data control device trains the locally preset basic model to be trained according to the requirement of edge intelligence to obtain the intermediate model.
In one embodiment, a training strategy for model training includes: performance index ratio calculation strategy, compression point generation strategy, redundant data removal strategy, privacy protection strategy and the like. The efficiency of model training of the whole system can be better improved by adopting a performance index than a calculation strategy; the training data can be compressed by adopting a compression point generation strategy, and the compression points have two types, including: coarse grain compression points and fine grain compression points; specifically, the influence values of all coarse-grained compression points on updating of the fusion parameters need to be judged, and the influence values can be obtained through the fusion parameters during model training, for example, the gradient of the model parameters, the coarse-grained compression points with high influence values are reserved, and the influence values are removed. For compression points which are difficult to judge, considering the influence values of fine-grained compression points on fusion parameter updating, reserving the high influence values, removing the low influence values, and finally obtaining the training data samples with redundant data removed; when the privacy protection strategy is adopted to upload the intermediate model, the privacy protection operation is carried out on the intermediate model, for example, the noise adding operation of the intermediate model is carried out.
The data processing module can carry the model training task in the issued model training task, so that the data processing equipment is guided to determine the training strategy adopted for training the basic model to be trained. However, the specific training process of the data is only controlled by the data control device, so that the strategy guidance information only has a reference function, and finally, the actual execution of the training strategy does not necessarily completely meet the requirements of the data processing device, so that the local strategy execution rule of the data control device can have a certain influence on the precision of the intermediate model obtained by training.
Specifically, after the data processing device trains the intermediate model each time, the intermediate model is uploaded to the data processing device, and the data processing device generates the comprehensive model according to the intermediate model uploaded by each data control device.
In one example, when the data control device reports the intermediate model to the data processing device, the training strategy adopted by the training of the basic model to be trained is also uploaded to the data processing device, so that the data processing device determines the fusion parameters of the intermediate model in the fusion algorithm according to the training strategy. Taking a weighting algorithm as an example, the fusion parameter is a weight value of the weighting algorithm, and on the premise that other training strategies are the same, the performance index ratio determined by the performance index ratio calculation strategy influences the weight value of the intermediate model in the weighting algorithm.
Specifically, in the scenario of edge intelligence, all edge intelligent nodes need to assume responsibility for data processing, so that the data control device in this embodiment needs to perform analysis processing on data by using the intermediate model obtained by training, in addition to training the base model to be trained by using local training data to obtain the intermediate model. Therefore, each data control device has a certain requirement for the accuracy or generalization index of the local intermediate model. After the intermediate model is obtained through training, indexes such as the precision of the intermediate model can be evaluated, and if the precision of the intermediate model meets a threshold value, the intermediate model is stored for data analysis processing; and when the precision of the intermediate model does not meet the threshold value, pulling the comprehensive model from the data processing equipment, and performing model training by taking the pulled comprehensive model as a basic model to be trained and reusing local training data.
In another example, when the precision of the intermediate model satisfies a preset threshold after the precision evaluation of the intermediate model, the data processing method in this embodiment is shown in fig. 6, and includes:
This step is similar to step 601 in this embodiment, and details of implementation have already been described, and are not described herein again.
And step 602, when the precision of the intermediate model meets a preset threshold, performing low-precision processing on the intermediate model to obtain a low-precision intermediate model.
Specifically, the low-precision model has the main function of submitting the trained model to the central module on the premise of not revealing local data so as to ensure the generalization and accuracy of the finally trained model. And the high-precision model is used for edge intelligence in actual applications such as local prediction, classification and the like.
It should be noted that the above examples in the present embodiment are only for easy understanding, and do not limit the technical scheme of the present invention.
Compared with the prior art, in the embodiment, the training data for model training is respectively stored in different data control devices in a decentralized manner according to the main bodies of the data, the process setting of the model training is performed by the data control devices, and the data processing devices only perform fusion generation of the comprehensive model and cannot directly contact the training data managed by the data control devices. In addition, the data processing equipment uploads the training strategy of the intermediate model while uploading the intermediate model, the training strategy of the intermediate model is taken into consideration when the intermediate models are combined, and the fusion parameters of the intermediate models in the fusion algorithm are determined according to the training strategy of the intermediate model, so that the final precision and generalization indexes of the models are controlled by the data control equipment, and the privacy safety of data stored by the data control equipment is further improved.
A sixth embodiment of the present invention is directed to a data processing system, as shown in fig. 6, including:
a data processing apparatus 601, and a center module 6011 provided in the data processing apparatus;
the N agent modules 6021 are respectively disposed on the N data control devices 602, wherein the center module 6011 is respectively in communication with the N agent modules.
The central module 6011 is configured to receive the N intermediate models uploaded by the N proxy modules; the N intermediate models are obtained by training the basic model to be trained by the N agent modules 6021 by using local training data; and generating a comprehensive model according to the N intermediate models.
The agent module 6021 is used for training a basic model to be trained by using local training data to obtain an intermediate model; and reporting the intermediate model to a central module 6011, so that the central module 6011 generates a comprehensive model.
A seventh embodiment of the present invention relates to a data processing apparatus, as shown in fig. 8, including at least one processor 801; and, at least one memory 802; the memory 802 stores instructions executable by the at least one processor 701, and the instructions are executed by the at least one processor 801, so that the at least one processor 801 can execute the data processing method in the first or second embodiment.
The memory 802 and the processor 801 are coupled by a bus, which may include any number of interconnecting buses and bridges that couple one or more of the various circuits of the processor 801 and the memory 802 together. The bus may also connect various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor 801 is transmitted over a wireless medium through an antenna, which receives the data and transmits the data to the processor 801. The processor 801 is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And memory 802 may be used to store data used by processor 801 in performing operations.
An eighth embodiment of the present invention relates to a data control apparatus, also shown in fig. 8, including at least one processor 801; and, at least one memory 802; the memory 802 stores instructions executable by the at least one processor 801, and the instructions are executed by the at least one processor 801, so that the at least one processor 801 can execute the data processing method in the first, second, third, fourth, or fifth embodiment.
The memory 802 and the processor 801 are coupled by a bus, which may include any number of interconnecting buses and bridges that couple one or more of the various circuits of the processor 801 and the memory 802 together. The bus may also connect various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor 801 is transmitted over a wireless medium through an antenna, which receives the data and transmits the data to the processor 801. The processor 801 is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And memory 802 may be used to store data used by processor 801 in performing operations.
An eighth embodiment of the present invention relates to a computer-readable storage medium storing a computer program. The computer program realizes the above-described method embodiments when executed by a processor.
That is, those skilled in the art can understand that all or part of the steps in the above method embodiments may be implemented by a program to instruct related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It will be understood by those of ordinary skill in the art that the foregoing is a specific embodiment for practicing the present invention, and that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.
Claims (17)
1. The data processing method is applied to data processing equipment, and the data processing equipment is respectively in communication connection with N data control equipment; the method comprises the following steps:
receiving N intermediate models uploaded by the N data control devices through a network; the N intermediate models are obtained by training a basic model to be trained by the N data control devices by using local training data;
and fusing the N intermediate models based on a fusion algorithm to generate a comprehensive model.
2. The data processing method according to claim 1, wherein before the receiving the N intermediate models uploaded by the N data control devices, the method further comprises:
issuing at least one of the following to the N data control devices: strategy guide information and the basic model to be trained;
and the strategy guidance information is used for guiding the data control equipment to select a training strategy adopted for training the basic model to be trained.
3. The data processing method according to claim 1, further comprising, after the fusing the N intermediate models based on the fusion algorithm to generate a comprehensive model:
performing precision evaluation on the comprehensive model;
when the precision index of the comprehensive model does not meet a preset threshold value, taking the comprehensive model as the basic model to be trained, issuing the basic model to be trained and new strategy guidance information to the N data control devices, and receiving N new intermediate models obtained by the N data control devices by training the basic model to be trained by using local training data; and generating a comprehensive model according to the N new intermediate models.
4. The data processing method according to claim 1, further comprising, before the fusing the N intermediate models to generate a comprehensive model based on a fusion algorithm:
receiving training strategies of the N intermediate models uploaded by the N data control devices; the training strategy of the intermediate model is a strategy adopted for training the basic model to be trained;
the fusion algorithm-based fusion of the N intermediate models to generate a comprehensive model comprises:
determining fusion parameters of the N intermediate models in a preset fusion algorithm according to the training strategies of the N intermediate models;
and fusing the N intermediate models according to the fusion parameters of the N intermediate models and the fusion algorithm to obtain the comprehensive model.
5. The data processing method of claim 4, wherein the fusion algorithm is a weighting algorithm and the fusion parameters are weight values.
6. The data processing method of claim 2, wherein the model training task includes the policy guidance information;
before the model training task is issued to the N data control devices, the method further includes:
determining a data dictionary corresponding to each data control device according to a mapping relation between a pre-stored data dictionary and the data control device;
and analyzing the strategy guidance information in the model training task issued to each data control device into a form capable of being recognized by each data control device according to the data dictionary corresponding to each data control device.
7. The data processing method is applied to data control equipment, and the data control equipment is in communication connection with the data processing equipment; the method comprises the following steps:
training a basic model to be trained by using local training data to obtain an intermediate model;
and reporting the intermediate model to the data processing equipment for the data processing equipment to generate a comprehensive model.
8. The data processing method of claim 7, further comprising, before the training the base model to be trained using the local training data:
receiving policy guidance information issued by the data processing equipment;
the training of the basic model to be trained by using the local training data to obtain the intermediate model comprises the following steps:
determining a training strategy for training the basic model to be trained according to the strategy guide information and a local strategy execution rule;
and training the basic model to be trained by using local training data according to the training strategy to obtain the intermediate model.
9. The data processing method of claim 7, wherein the training of the basic model to be trained using the local training data to obtain the intermediate model comprises:
determining a training strategy for training the basic model to be trained according to a local strategy execution rule;
determining data training operation according to the training strategy;
and executing the data training operation on the training data to obtain the intermediate model.
10. The data processing method of claim 9, wherein the training strategy of the base model to be trained comprises: a performance index ratio calculation strategy and a strategy for preprocessing the training data;
the determining of the training strategy for training the basic model to be trained according to the local strategy execution rule comprises:
determining a local performance index ratio according to the performance index ratio calculation strategy;
and determining a strategy for preprocessing the training data according to the performance index ratio.
11. The data processing method of claim 10, wherein obtaining the local performance indicator ratio according to the performance indicator ratio calculation policy comprises:
acquiring the use condition of local hardware resources, and uploading the use condition of the hardware resources to the data processing equipment; the data processing equipment generates the performance index ratio according to the hardware resource use condition uploaded by the N data control equipment;
and acquiring the performance index ratio sent by the data processing equipment.
12. The data processing method according to claim 9, wherein after the training of the basic model to be trained by using the local training data to obtain the intermediate model, the method further comprises:
and when the precision of the intermediate model does not meet a preset threshold value, pulling the comprehensive model to the data processing equipment, taking the pulled comprehensive model as the basic model to be trained, and executing the training of the basic model to be trained by using local training data to obtain the intermediate model.
13. The data processing method of claim 12, wherein reporting the intermediate model to the data processing device comprises:
when the precision of the intermediate model meets a preset threshold value, performing low-precision processing on the intermediate model to obtain the intermediate model with low precision;
and reporting the low-precision intermediate model to the data processing equipment.
14. The data processing method according to claim 8 or 9, wherein after the training the basic model to be trained by using the local training data to obtain the intermediate model, the method further comprises:
uploading the training strategy of the intermediate model to the data processing equipment; the training strategy is a strategy adopted for training the basic model to be trained.
15. A data processing system, comprising: the data processing system comprises a central module arranged in the data processing equipment and N agent modules respectively arranged in N data control equipment; the central module is in communication connection with the N agent modules respectively;
the central module is used for receiving the N intermediate models uploaded by the N agent modules; generating a comprehensive model according to the N intermediate models;
the agent module is used for training a basic model to be trained by using local training data to obtain an intermediate model; and reporting the intermediate model to the central module.
16. An apparatus, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a data processing method as claimed in claims 1 to 6 and/or a data processing method as claimed in any one of claims 7 to 14.
17. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the data training method of one of claims 1 to 6 and/or the data processing method of one of claims 7 to 14.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010606801.8A CN113935390A (en) | 2020-06-29 | 2020-06-29 | Data processing method, system, device and storage medium |
| PCT/CN2021/103193 WO2022002068A1 (en) | 2020-06-29 | 2021-06-29 | Data processing method, system and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010606801.8A CN113935390A (en) | 2020-06-29 | 2020-06-29 | Data processing method, system, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113935390A true CN113935390A (en) | 2022-01-14 |
Family
ID=79273048
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010606801.8A Pending CN113935390A (en) | 2020-06-29 | 2020-06-29 | Data processing method, system, device and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113935390A (en) |
| WO (1) | WO2022002068A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115611393B (en) * | 2022-11-07 | 2023-04-07 | 中节能晶和智慧城市科技(浙江)有限公司 | Multi-end cooperative coagulant feeding method and system for multiple water plants |
| CN117194991B (en) * | 2023-11-03 | 2024-02-13 | 四川并济科技有限公司 | High-dimensional data recommendation system and method based on GPU cluster |
| CN117272688B (en) * | 2023-11-20 | 2024-02-13 | 四川省交通勘察设计研究院有限公司 | Compression and decompression method, device and system for structural mechanics simulation data |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110378488A (en) * | 2019-07-22 | 2019-10-25 | 深圳前海微众银行股份有限公司 | Federal training method, device, training terminal and the storage medium of client variation |
| CN110442457A (en) * | 2019-08-12 | 2019-11-12 | 北京大学深圳研究生院 | Model training method, device and server based on federation's study |
| CN110825970A (en) * | 2019-11-07 | 2020-02-21 | 浙江同花顺智能科技有限公司 | Information recommendation method, device, equipment and computer readable storage medium |
| WO2020042658A1 (en) * | 2018-08-31 | 2020-03-05 | 华为技术有限公司 | Data processing method, device, apparatus, and system |
| CN110995488A (en) * | 2019-12-03 | 2020-04-10 | 电子科技大学 | Multi-mechanism collaborative learning system and method based on hierarchical parameter server |
| US20200143206A1 (en) * | 2018-11-05 | 2020-05-07 | Royal Bank Of Canada | System and method for deep reinforcement learning |
| CN111178503A (en) * | 2019-12-16 | 2020-05-19 | 北京邮电大学 | Mobile terminal-oriented decentralized target detection model training method and system |
| CN111245903A (en) * | 2019-12-31 | 2020-06-05 | 烽火通信科技股份有限公司 | Joint learning method and system based on edge calculation |
| CN111310932A (en) * | 2020-02-10 | 2020-06-19 | 深圳前海微众银行股份有限公司 | Method, device and equipment for optimizing horizontal federated learning system and readable storage medium |
| CN111339553A (en) * | 2020-02-14 | 2020-06-26 | 云从科技集团股份有限公司 | Task processing method, system, device and medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104143327B (en) * | 2013-07-10 | 2015-12-09 | 腾讯科技（深圳）有限公司 | A kind of acoustic training model method and apparatus |
| CN105575389B (en) * | 2015-12-07 | 2019-07-30 | 百度在线网络技术（北京）有限公司 | Model training method, system and device |
| EP3430526B1 (en) * | 2016-03-18 | 2024-09-25 | Microsoft Technology Licensing, LLC | Method and apparatus for training a learning machine |
| US20200027009A1 (en) * | 2018-07-23 | 2020-01-23 | Kabushiki Kaisha Toshiba | Device and method for optimising model performance |
| CN109600255A (en) * | 2018-12-04 | 2019-04-09 | 中山大学 | A kind of parameter server optimization algorithm of decentralization |
| CN110795477A (en) * | 2019-09-20 | 2020-02-14 | 平安科技（深圳）有限公司 | Data training method, device and system |
| CN111081337B (en) * | 2020-03-23 | 2020-06-26 | 腾讯科技（深圳）有限公司 | Collaborative task prediction method and computer readable storage medium |
-
2020
- 2020-06-29 CN CN202010606801.8A patent/CN113935390A/en active Pending
-
2021
- 2021-06-29 WO PCT/CN2021/103193 patent/WO2022002068A1/en active Application Filing
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020042658A1 (en) * | 2018-08-31 | 2020-03-05 | 华为技术有限公司 | Data processing method, device, apparatus, and system |
| US20200143206A1 (en) * | 2018-11-05 | 2020-05-07 | Royal Bank Of Canada | System and method for deep reinforcement learning |
| CN110378488A (en) * | 2019-07-22 | 2019-10-25 | 深圳前海微众银行股份有限公司 | Federal training method, device, training terminal and the storage medium of client variation |
| CN110442457A (en) * | 2019-08-12 | 2019-11-12 | 北京大学深圳研究生院 | Model training method, device and server based on federation's study |
| CN110825970A (en) * | 2019-11-07 | 2020-02-21 | 浙江同花顺智能科技有限公司 | Information recommendation method, device, equipment and computer readable storage medium |
| CN110995488A (en) * | 2019-12-03 | 2020-04-10 | 电子科技大学 | Multi-mechanism collaborative learning system and method based on hierarchical parameter server |
| CN111178503A (en) * | 2019-12-16 | 2020-05-19 | 北京邮电大学 | Mobile terminal-oriented decentralized target detection model training method and system |
| CN111245903A (en) * | 2019-12-31 | 2020-06-05 | 烽火通信科技股份有限公司 | Joint learning method and system based on edge calculation |
| CN111310932A (en) * | 2020-02-10 | 2020-06-19 | 深圳前海微众银行股份有限公司 | Method, device and equipment for optimizing horizontal federated learning system and readable storage medium |
| CN111339553A (en) * | 2020-02-14 | 2020-06-26 | 云从科技集团股份有限公司 | Task processing method, system, device and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022002068A1 (en) | 2022-01-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113935390A (en) | Data processing method, system, device and storage medium | |
| EP3644581B1 (en) | Edge-cloud collaboration system for analyzing internet of things data and operating method thereof | |
| CN113286315B (en) | Load balance judging method, device, equipment and storage medium | |
| CN113255953B (en) | RRU undervoltage risk prediction method, device, system, equipment and medium | |
| CN116761194B (en) | Police affair cooperative communication optimization system and method in wireless communication network | |
| CN112232985B (en) | Power distribution and utilization data monitoring method and device for ubiquitous power Internet of things | |
| CN115529315B (en) | Cloud edge cooperative system | |
| CN115221982B (en) | Traction power supply operation and maintenance method and device, terminal and storage medium | |
| CN112804287A (en) | Intelligent network slice template generation method and system for power Internet of things | |
| CN115543577A (en) | Kubernetes resource scheduling optimization method based on covariates, storage medium and equipment | |
| CN117595504A (en) | Intelligent monitoring and early warning method for power grid running state | |
| CN114548416A (en) | Data model training method and device | |
| CN117175664A (en) | Energy storage charging equipment output power self-adaptive adjusting system based on use scene | |
| CN117633592B (en) | Intelligent monitoring disc system of new energy power station | |
| CN111769987B (en) | Network information security testing system and method based on big data management model | |
| CN116346640A (en) | Network index prediction method and device, electronic equipment and storage medium | |
| RU2532714C2 (en) | Method of acquiring data when evaluating network resources and apparatus therefor | |
| CN116384240A (en) | Server energy consumption prediction method, device and storage medium | |
| CN107517474B (en) | Network analysis optimization method and device | |
| CN115827232A (en) | Method, device, system and equipment for determining configuration for service model | |
| CN104462756A (en) | Service credibility assessment method oriented to internet of things manufacturing | |
| CN115037625A (en) | Network slice processing method and device, electronic equipment and readable storage medium | |
| CN111092755B (en) | Edge service migration simulation method based on resource occupation | |
| Dorenskyi | The methodology of evaluating the test cases quality for simple IT monoprojects software testing | |
| CN110084511B (en) | Unmanned aerial vehicle configuration method, device, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |