WO2022002068A1 - 数据处理方法、系统、设备及存储介质 - Google Patents

数据处理方法、系统、设备及存储介质 Download PDF

Info

Publication number
WO2022002068A1
WO2022002068A1 PCT/CN2021/103193 CN2021103193W WO2022002068A1 WO 2022002068 A1 WO2022002068 A1 WO 2022002068A1 CN 2021103193 W CN2021103193 W CN 2021103193W WO 2022002068 A1 WO2022002068 A1 WO 2022002068A1
Authority
WO
WIPO (PCT)
Prior art keywords
model
data
training
data processing
trained
Prior art date
Application number
PCT/CN2021/103193
Other languages
English (en)
French (fr)
Inventor
洪科
裴应明
韩锐
刘驰
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2022002068A1 publication Critical patent/WO2022002068A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/20—Design optimisation, verification or simulation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/20—Design optimisation, verification or simulation
    • G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/20—Ensemble learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2113/00—Details relating to the application field
    • G06F2113/04—Power grid distribution networks

Definitions

  • the present application relates to the field of communication technologies, and in particular, to a data processing method, system, device and storage medium.
  • the data processing equipment can access a large amount of raw collected data uploaded by the data control equipment, it may lead to a series of problems, such as leakage of personal privacy information or regional confidential information.
  • An embodiment of the present application provides a data processing method, which is applied to a data processing device, and the data processing device is respectively connected to N data control devices in communication; the method includes: receiving N intermediate models uploaded by the N data control devices; wherein, The N intermediate models are obtained by training the basic model to be trained by using the local training data by N data processing devices; and a comprehensive model is generated according to the N intermediate models.
  • the embodiments of the present application also provide a data processing method, which is applied to a data control device, and the data control device is connected to the data processing device in communication; the method includes: using local training data to train a basic model to be trained to obtain an intermediate model; The model is reported to the data control device for the data control device to generate a comprehensive model.
  • the embodiment of the present application also provides a model training system, including: a central module arranged in a data processing device and N proxy modules respectively arranged in N data control devices; wherein the central module and the N proxy modules are respectively Communication connection; the central module is used to receive N intermediate models uploaded by N data control devices; wherein, the N intermediate models are obtained by training the basic model to be trained by N data processing devices using local training data; according to the N intermediate models
  • the model generates a comprehensive model; the proxy module is used to train the basic model to be trained by using the local training data to obtain an intermediate model; the intermediate model is reported to the data control device for the data control device to generate a comprehensive model.
  • Embodiments of the present application also provide a device, comprising: at least one processor, and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are processed by the at least one processor The processor executes to enable at least one processor to execute the data processing method as described above.
  • Embodiments of the present application further provide a computer-readable storage medium, storing a computer program, and when the computer program is executed by a processor, the above-mentioned data processing method is implemented.
  • FIG. 1 is a flowchart of a data processing method applied to a data processing device according to the first embodiment of the present application
  • FIG. 2 is a flowchart of a data processing method applied to a data processing device according to a second embodiment of the present application
  • FIG. 3 is a flowchart of a data processing method applied to a data control device according to a third embodiment of the present application.
  • FIG. 4 is a flowchart of a data processing method applied to a data control device according to a fourth embodiment of the present application.
  • FIG. 5 is a flowchart of a data processing method applied to a data control device according to a fifth embodiment of the present application.
  • FIG. 6 is a flowchart of a data processing method applied to a data control device according to a fifth embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a model training system according to a sixth embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a data control device according to the seventh and eighth embodiments of the present application.
  • the data processing equipment will obtain the original collection data from different data control equipment through the network to train the model according to the needs, wherein the different data control equipment can be distributed in different regions and used to collect and store the data of the region where they are located.
  • the power IoT systems in different regions can be understood as different data control equipment, and the electronic equipment used by the power IoT system to actually implement the power dispatching scheme can be understood as data processing equipment; another example, the medical systems in different regions can be understood as For different data control equipment, the dispatching department of the medical system can be understood as data processing equipment.
  • the data processing equipment can access a large amount of raw collected data uploaded by the data control equipment, it may lead to a series of problems, such as leakage of personal privacy information or regional confidential information.
  • the purpose of the embodiments of the present application is to provide a data processing method, system, device, and storage medium, so as to prevent the data processing device from directly contacting the data of all data control devices, and to improve the privacy of model training data while ensuring the accuracy of model training.
  • the first embodiment of the present application relates to a data processing method, which is applied to a data processing device.
  • the data processing device is a terminal device used for data analysis and processing using a model trained with massive data; a data control device is used for the model Terminal equipment for training and management of stored data.
  • the training data mentioned in this embodiment are respectively distributed in N data control devices, where N is a natural number greater than or equal to 1.
  • the data processing device is respectively connected to the N data control devices; the data processing device receives N intermediate models uploaded by the N data control devices; wherein, the N intermediate models are treated by the N data control devices using local training data to train the basic model It is obtained by training; a comprehensive model is generated according to N intermediate models.
  • This embodiment can be widely used in a distributed data storage architecture to ensure data privacy and security of a large amount of data; the distributed data storage architecture can be used to store patient information in a medical system, student information in an education system, traffic Predict vehicle information in the system, environmental data in the power system, etc.
  • the power grid architecture as an example, different regions have independent power IoT systems.
  • the power IoT system is based on IoT devices.
  • the power Internet of Things system acquires characteristic data of each region, and uses the acquired characteristic data of each region to train a model; however, there is a risk of data leakage in the transmission of characteristic data.
  • the data processing device may be another device independent of the data control device, or may be a data control device selected from a plurality of data control devices, the selected data control device performing the functions of the data processing device.
  • the power Internet of Things system in each region can be used as a data control device, and the electronic equipment in the power dispatching department responsible for executing the power dispatching plan can be a device completely independent from the power Internet of Things system in each region.
  • a power Internet of Things system can also be selected from the power Internet of Things systems in various regions as the power dispatching department, that is, to perform the function of data processing equipment.
  • a suitable data control device can be selected as the data processing device through preset rules, for example, the data control device with the best hardware performance can be selected as the data processing device .
  • the execution body of the data processing method in this embodiment is a data processing device, and the specific process is shown in FIG. 1 , including:
  • Step 101 Receive N intermediate models uploaded by N data control devices through a network.
  • the N intermediate models received by the data processing device are respectively trained by the N data control devices.
  • the data for training is distributed among N data control devices, and each data control device uses the locally stored training data to train the basic model to be trained, and then uploads the intermediate model to the data processing device after obtaining the intermediate model.
  • the basic model to be trained may be carried by the data processing device in advance through a model training task and delivered to the data control device, and the model training task is used to instruct the data control device to perform model training.
  • Step 102 based on the fusion algorithm, fuse the N intermediate models to generate a comprehensive model.
  • the training data stored by each data control device is different, the accuracy and generalization index of the intermediate model obtained by training are also different.
  • the data processing device The N intermediate models are fused to generate a comprehensive model that is finally used for data analysis and processing.
  • the training data used for model training in this embodiment is stored in different data control devices in a decentralized manner according to the main body of the data, and the model training process is set to be performed by the data control device, and the data The processing device only performs the fusion generation of the comprehensive model and cannot directly access the training data managed by the data control device, so that the training data is physically isolated and the purpose of data privacy protection is effectively achieved.
  • the total amount of training data used for model training has not changed, and the accuracy and generalization indicators of the model obtained by the data processing equipment can also be guaranteed.
  • the second embodiment of the present application relates to a data processing method, and the execution subject of this embodiment is a data processing device.
  • the data processing device will evaluate indicators such as the accuracy of the comprehensive model after generating the comprehensive model.
  • Step 201 delivering a model training task to N data control devices through a network.
  • the data processing device issues a model training task for instructing the data control device to perform model training to the data control device, and the model training task includes at least one of the following: policy guidance information and a basic model to be trained.
  • the strategy guidance information is used to instruct the data processing device to determine the training strategy used for training the basic model to be trained.
  • the policy guidance information in the model training task consists of parsed operation instructions.
  • the data processing device determines the data dictionary corresponding to the N data control devices according to the mapping relationship between the pre-stored data dictionary and the data control devices. Then, the preset operation instructions are parsed according to the data dictionary corresponding to the data control device to obtain policy guidance information.
  • parsed operation instructions are operation primitives, for example:
  • Primitives for redundant data removal operation data table name: UserName; redundant data removal ratio: adaptive redundant data removal parameters: value1, value2; expected completion time: 1 second; Primitives: Execute data table: UserInfo; Compression ratio setting: adaptive; Expected completion time: 1 second;
  • the primitives for executing data training operations of a data table are: Data table name: UserInfoDemo Field list: F1, F2, F3; Label field: L; Algorithm: logistic regression; Expected completion time: 10 seconds.
  • each data control device Since the data subjects of each data control device are different in practical applications, the types of operation primitives that can be recognized by each data control device are also different. Therefore, before the data processing device parses the preset operation instructions, it needs to obtain the data dictionary of each data control device, and then save the mapping relationship between the data control device and the data dictionary. When parsing, the corresponding data dictionary is searched in the mapping relationship between the data control device and the data dictionary, and then the operation instructions are converted into operation primitives that the data control device can recognize, so that the data processing device can be compatible with all different data control devices. equipment.
  • Step 202 Receive N intermediate models uploaded by N data control devices through the network.
  • step 101 is similar to step 101 in the first embodiment of the present application, and the relevant implementation details have been specifically described in the first embodiment, which will not be repeated here.
  • Step 203 based on the fusion algorithm, fuse the N intermediate models to generate a comprehensive model.
  • the data processing device merges each intermediate model uploaded by the data control device according to a preset fusion algorithm, and the specific merging process is affected by the fusion parameters.
  • the data processing device receives the training strategies of N intermediate models uploaded by the data control device before merging the intermediate models, and the training strategies of the intermediate models refer to the training strategies adopted for training the basic model to be trained.
  • the fusion parameters of the N intermediate models in the preset fusion algorithm are determined according to the training strategy of the N intermediate models.
  • the N intermediate models are fused according to the fusion parameters of the N intermediate models and the fusion algorithm to obtain a comprehensive model.
  • the training strategy of the intermediate models of the data processing equipment is taken into consideration, and the fusion parameters of each intermediate model in the fusion algorithm are determined according to the training strategy of the intermediate model, so that the final accuracy and generalization index of the model are affected by the data. Controlling the influence of the actual training strategy of the device can improve the accuracy of the integrated model obtained by fusion.
  • the data control device may upload the training strategy while uploading the intermediate model to the data processing device, that is, the data processing device simultaneously receives the intermediate model and the training strategy of the intermediate model.
  • the fusion algorithm may use a weighted algorithm, and the fusion parameter is the weight value of each intermediate model in the weighting algorithm, and the weight value of each intermediate model is determined according to the training strategy.
  • the training strategy adopted by the data control device is a single strategy, the weight value of each intermediate model in the weighting algorithm is determined only according to a single training strategy; if multiple strategies are used for model training, each training strategy is comprehensively considered to determine the weight value of each intermediate model in the weighting algorithm.
  • the weighted algorithm can make the integrated model obtained by fusion have better accuracy.
  • the training strategy of the intermediate model may include: a privacy protection strategy, a data training strategy, a compression point generation strategy, a redundant data removal strategy, a performance index ratio calculation strategy, and an adaptive resource allocation strategy.
  • the data control device may adopt the performance index ratio calculation strategy to obtain the current available hardware resource index of each data control device.
  • This strategy uses the system API to monitor various available resources of the current data control device, such as network bandwidth, hard disk I/O, CPU usage, etc., to obtain the performance indicators of the current data control device.
  • the proxy module will periodically send the local hardware resource status to the data processing device. After the data processing device summarizes all resource indicators, it calculates the relative resource indicators of each proxy module, that is, the performance indicator ratio, and finally sends it back to the data control device.
  • a weighted algorithm is used for model fusion, if the performance index of the data control device is relatively high, the intermediate model trained by the data control device will have a higher weight value in the fusion algorithm.
  • a compression point generation strategy for compressing data.
  • compression points There are two types of compression points: coarse-grained compression points and fine-grained compression points.
  • One of the coarse-grained compression points corresponds to multiple fine-grained compression points.
  • the fine-grained compression point corresponding to the coarse-grained compression point is calculated to obtain a more Precise fusion parameters.
  • the adopted fusion algorithm is a weighted algorithm
  • the weight value of the intermediate model is set based on the training strategy of the intermediate model, which may include the following methods: For the intermediate model trained by using the compression point generation strategy, the weight value can be It is set relatively low; or, for two intermediate models, if both of them are trained with the compression point generation strategy, the intermediate model trained with the fine-grained compression point method is compared to the intermediate model trained with the coarse-grained compression point method. For example, the weight value should be set smaller.
  • Data training strategy You can specify the algorithm used by the agent module for data training, and execute the algorithm. There are various specific choices for this strategy, such as choosing convolutional neural networks, logistic regression, reinforcement learning and other algorithms.
  • Data controllers can set their own according to their needs, and also accept the policy guidance information sent by the data processing equipment. First, the strategy needs to determine the current training strategy and the data training algorithm that needs to be used currently. Then allocate resources according to the settings and perform training operations. An intermediate model that can eventually be generated by training the policy on the data.
  • Redundant data removal strategy Use the coarse-grained compression points and fine-grained compression points generated by the compression point generation strategy to carry out redundant data removal strategy. First, calculate the influence value of all coarse-grained compression points on the update of data training parameters, and judge whether the data of this compression point is redundant according to this influence value. Traverse all coarse-grained compression points, and retain compression points with larger impact values. At the same time, fine-grained compression is performed on coarse-grained compression points with small influence values to generate multiple fine-grained compression points, and then the influence value of the current fine-grained compression point on parameter update is judged, redundant data is removed, and valuable data is retained.
  • Adaptive resource allocation strategy Use the performance index ratio of the current data control device to calculate the resources and the amount of training data that should be used by the current agent module for data training. This strategy can solve the problem of asynchronous data training progress between agent modules, and improve the overall training efficiency.
  • Step 204 Evaluate the comprehensive model to determine whether the accuracy of the comprehensive model satisfies the preset threshold; if the accuracy of the comprehensive model meets the preset threshold, retain the comprehensive model; if the accuracy of the comprehensive model does not meet the preset threshold, execute step 201 .
  • the comprehensive model is evaluated to determine whether the accuracy or generalization index of the model meets a preset threshold.
  • the integrated model is retained, and the integrated model can be delivered to each data control device, and the data control device can use the integrated model to perform data analysis and processing.
  • the comprehensive model is added to the model training task as the basic model to be trained and sent to the data control device for the data control device to retrain the model to further improve the accuracy of the model Or generalization metrics, etc.
  • the model accuracy evaluation methods are not necessarily the same. If it is an indicator prediction scenario, RMSE (root mean square error), MAE (mean absolute error), R square and other evaluation indicators; if it is a classification scenario, you can use evaluation indicators such as precision rate, recall rate, accuracy rate, error rate, and F1; if it is clustering, you can use Davies-Bouldin Index (Davidson Bouldin Index), Dunn Validity Index ( Dunn index) and other evaluation indicators.
  • the above indicators are purely mathematical. In practical applications, there are many variants of the accuracy evaluation indicators. For example, in the application scenario of the power Internet of Things system mentioned below, the fluctuation rate of the invalid power generation over a period of time can be used as the power scheme. Evaluation Metrics. From the characteristics of the power system, that is, when the ineffective power generation of the power system in a period of time is small and there is no large fluctuation, it means that the power dispatching scheme as a comprehensive model has better accuracy.
  • the data processing device can continue to use the strategy guidance information in the previous model training task while adding the previously generated comprehensive model to the model training task as the basic model to be trained ;
  • the policy guidance information can also be modified according to the requirements of data analysis and processing, so as to better improve the efficiency of model training and the accuracy or generalization index of the intermediate model obtained from the training.
  • the present embodiment is further explained below by taking the power Internet of Things system as an application scenario.
  • the data processing equipment is the electronic equipment that executes the power dispatching scheme in the power Internet of Things system; the data control equipment is the power system in different regions.
  • the data control device obtains information such as the definition of the supported data operation primitives, the access identifier of the data processor, and the policy descriptor from the data processing device; the data control device sets the level of the data processing device and the execution action flow corresponding to the operation primitive. and data acquisition masking rules; data processing equipment acquires data dictionary information from multiple data control equipment.
  • the data processing device takes the current power dispatching scheme as the basic model to be trained, and sends the basic model to be trained in the model training task to each data control device.
  • the power dispatching needs to be reasonably distributed to meet the daily electricity consumption of ordinary people and various production departments (such as factories, etc. ) production electricity. Therefore, we take the default power scheduling scheme as the basic model to be trained, and send the basic model to be trained and task initiation information to each data control device.
  • the data control device After the data control device receives the basic model to be trained from the data processing device, it first deploys the power dispatching scheme in the region according to the power dispatching method in the model, which ensures that each time the data control device obtains a new model, it can Deploy power dispatching plans in a timely manner. Then, data training is performed according to the policy guidance information in the model training task and the local rules.
  • the data set used is the information collected by the perception layer of the local power Internet of Things system; the data training strategy is adaptively selected by the data control equipment, and the available strategies are as follows: compression point generation strategy, redundant data removal strategy, Performance index ratio calculation strategy, etc.; data methods are uniformly specified by data processing equipment, such as classification algorithms, clustering algorithms, multi-layer neural network algorithms, etc.
  • the data control device After the data control device performs a data training, it obtains the currently trained intermediate model and the strategy used for training. First, the data control device uses an adaptive privacy protection policy to perform privacy protection operations on the intermediate model. The purpose of this is to prevent the leakage of local power usage and power scheduling schemes during the transmission process, resulting in insecure privacy. Then each data control device transmits the local intermediate model to the data processing device, so that the data processing device merges the various intermediate models and ensures the generalization of the finally obtained comprehensive model.
  • an adaptive privacy protection policy to perform privacy protection operations on the intermediate model. The purpose of this is to prevent the leakage of local power usage and power scheduling schemes during the transmission process, resulting in insecure privacy. Then each data control device transmits the local intermediate model to the data processing device, so that the data processing device merges the various intermediate models and ensures the generalization of the finally obtained comprehensive model.
  • the data processing device merges the models and evaluates the models. After the data processing device obtains the intermediate models of all current data control devices, the intermediate models are merged to obtain a comprehensive model.
  • the comprehensive model ie, the power dispatching scheme
  • the comprehensive model is evaluated using the test data of the data processing equipment. If the comprehensive model meets the requirements, the comprehensive model will be issued; otherwise, the model training task will be issued again, and the data control device will retrain a new intermediate model according to the new policy guidance information.
  • the training data used for model training in this embodiment is stored in different data control devices in a decentralized manner according to the main body of the data, and the model training process is set to be performed by the data control device, and the data The processing device only performs fusion generation of the comprehensive model and cannot directly access the training data managed by the data control device.
  • the training strategy of the intermediate model is taken into consideration, and the fusion parameters of each intermediate model in the fusion algorithm are determined according to the training strategy of the intermediate model, so that the final accuracy and generalization index of the model are controlled by the data control device. , which further improves the privacy and security of the data stored in the data control device.
  • the third embodiment of the present application relates to a data processing method, which is applied to a data control device.
  • the data control device is a terminal device used for model training and management of stored data;
  • the model to carry out data analysis and processing of terminal equipment.
  • the training data mentioned in this embodiment are respectively distributed in N data control devices, where N is a natural number greater than or equal to 1.
  • the data control device is connected in communication with the data processing device; the method includes: using the local training data to train the basic model to be trained to obtain an intermediate model; reporting the intermediate model to the data control device for the data control device to generate a comprehensive model.
  • the execution body of the data processing method in this embodiment is a data control device, and the specific process is shown in FIG. 3 , including:
  • Step 301 using the local training data to train the basic model to be trained to obtain an intermediate model.
  • the data control device locally stores a large amount of training data for model training.
  • the basic model to be trained is an initial model pre-stored by the data control device, and an intermediate model is obtained after inputting training data to the basic model to be trained for training.
  • the basic model to be trained is generated and delivered by a data processing device, and the data control device trains the basic model to be trained to obtain an intermediate model.
  • Step 302 reporting the intermediate model to the data processing device for the data processing device to generate a comprehensive model.
  • each data control equipment uploads the intermediate model obtained from the training to the data processing equipment for the data processing equipment to combine the intermediate models to generate a comprehensive Model.
  • this embodiment is a data processing method applied to a data control device implemented in cooperation with the first embodiment and the second embodiment of the present application.
  • the technical details mentioned in the first embodiment of the present application are described in this document. It can also be implemented in the embodiment, and in order to reduce repetition, details are not repeated here.
  • the training data used for model training in this embodiment is stored in different data control devices in a decentralized manner according to the main body of the data, and the model training process is set to be performed by the data control device, and the data
  • the processing device only performs the fusion generation of the comprehensive model and cannot directly access the training data managed by the data control device, so that the training data is physically isolated and the purpose of data privacy protection is effectively achieved.
  • the total amount of training data used for model training has not changed, and the accuracy and generalization indicators of the model obtained by the data processing equipment can also be guaranteed.
  • the fourth embodiment of the present application relates to a data processing method.
  • the main difference between this embodiment and the third embodiment of the present application is that when the data control device uses the training data to train the basic model to be trained, it will execute the method according to the local strategy.
  • the training strategy of the intermediate model is determined by the rules. After the training is completed, the training strategy of the intermediate model obtained from the training will be uploaded to the data processing module.
  • the execution body of the data processing method in this embodiment is a data control device, and the specific process is shown in FIG. 4 , including:
  • Step 401 Receive a model training task issued by a data processing device.
  • the data control device before training the basic model to be trained, receives a model training task issued by the data processing device, and the model training task may carry policy guidance information and the basic model to be trained.
  • the model training task may carry policy guidance information and the basic model to be trained.
  • Step 402 using the local training data to train the basic model to be trained to obtain an intermediate model.
  • the training strategy of the basic model to be trained is determined according to the strategy guidance information and the local preset policy execution rules.
  • the basic model to be trained is the model preset locally by the data control device.
  • the data control device determines the training strategy of the basic model to be trained according to the local preset strategy execution rules, and then performs the training strategy for the basic model to be trained in the model training task according to the training strategy.
  • the model is trained.
  • the data control device determines the training strategy of the basic model to be trained according to the policy guidance information and the local preset policy execution rules at the same time, and then according to The training strategy trains the base model to be trained in the model training task.
  • Step 403 reporting the intermediate model to the data processing device for the data processing device to generate a comprehensive model.
  • the data control device reports both the intermediate model and the training strategy of the intermediate model to the data processing device.
  • the data control device may report the training strategy of the intermediate model to the data processing device in advance, or simultaneously report the intermediate model and the training strategy of the intermediate model to the data processing device.
  • the data processing device determines the fusion parameters of the intermediate model in the fusion algorithm according to the training strategy of the intermediate model. That is, when the data processing equipment merges the intermediate models, the training strategy of the intermediate models of the data processing equipment is taken into consideration, and the fusion parameters of each intermediate model in the fusion algorithm are determined according to the training strategy of the intermediate models, so that the final accuracy and generalization index of the model can be achieved. Affected by the actual training strategy of the data control device, the privacy security of the data stored by the data control device is further improved.
  • the fifth embodiment of the present application relates to a data processing method, and the execution subject of this embodiment is a data control device.
  • the difference between this embodiment and the third embodiment of the present application is that after generating the intermediate model, the data control device will evaluate the accuracy of the intermediate model to determine whether to continue to train a higher-precision intermediate model for local data control equipment. Data analysis and processing.
  • Step 501 using the local training data to train the basic model to be trained to obtain an intermediate model.
  • the initial basic model to be trained in this embodiment is locally preset by the data control device, and the data control device first trains the locally preset basic model to be trained according to the requirements of edge intelligence to obtain an intermediate model.
  • the training strategy for model training includes: a performance index ratio calculation strategy, a compression point generation strategy, a redundant data removal strategy, a privacy protection strategy, and the like.
  • the use of performance indicators can better improve the efficiency of model training for the overall system than the calculation strategy; the training data can be compressed by using the compression point generation strategy.
  • the redundant data removal strategy is used to remove unnecessary data to improve the efficiency of model training. Specifically, it is first necessary to determine the impact value of all coarse-grained compression points on the update of fusion parameters, which can be fused through model training.
  • Parameter acquisition such as model parameter gradients, retains coarse-grained compression points with high influence values, and removes those with low influence values. For those compression points that are not easy to judge, consider the influence value of the fine-grained compression points on the update of fusion parameters, keep those with high influence values, and remove those with low influence values, and finally get the training data samples with redundant data removed; privacy protection is adopted.
  • the strategy is to perform privacy protection operations on the intermediate model when uploading the intermediate model, such as adding noise to the intermediate model.
  • the data processing module can carry the model training task in the issued model training task, thereby instructing the data processing device to determine the training strategy used for training the basic model to be trained.
  • the policy guidance information can only serve as a reference, and the actual execution of the training strategy may not fully meet the needs of the data processing device. Therefore, the local policy execution rules of the data control device It will have a certain impact on the accuracy of the intermediate model obtained by training.
  • Step 502 reporting the intermediate model to the data processing device for the data processing device to generate a comprehensive model.
  • the data processing device uploads the intermediate model to the data processing device, and the data processing device generates a comprehensive model according to the intermediate models uploaded by each data control device.
  • the training strategy used for training the basic model to be trained this time is also uploaded to the data processing device, so that the data processing device can perform the training according to the training strategy.
  • the performance index ratio determined according to the performance index ratio calculation strategy will affect the weight value of the intermediate model in the weighting algorithm.
  • Step 503 Perform an accuracy assessment on the intermediate model to determine whether the accuracy of the intermediate model meets a preset threshold; if the accuracy of the intermediate model meets the threshold, save the intermediate model for data analysis and processing; if the accuracy of the intermediate model does not meet the threshold, execute the step 504. Pull the comprehensive model from the data processing device. Then, model training is performed using the pulled comprehensive model as the base model to be trained.
  • the edge intelligent nodes all need to assume the responsibility of data processing. Therefore, the data control device in this embodiment not only uses the local training data to train the basic model to be trained to obtain the intermediate model, but also needs to Use the intermediate model obtained by training to perform data analysis and processing. Therefore, each data control device has certain requirements on the accuracy or generalization index of the local intermediate model.
  • the accuracy of the intermediate model will be evaluated. When the accuracy of the intermediate model meets the threshold, the intermediate model will be saved for data analysis and processing; when the accuracy of the intermediate model does not meet the threshold, data processing equipment will be pulled Take the comprehensive model, use the pulled comprehensive model as the basic model to be trained, and then use the local training data for model training.
  • the data processing method in this embodiment is shown in FIG. 6 , including:
  • Step 601 using the local training data to train the basic model to be trained to obtain an intermediate model.
  • step 601 This step is similar to step 601 in this embodiment, and the relevant implementation details have been described, and are not repeated here.
  • Step 602 when the accuracy of the intermediate model meets a preset threshold, perform low-precision processing on the intermediate model to obtain a low-precision intermediate model.
  • Step 603 reporting the low-precision intermediate model to the data processing device.
  • the main function of the low-precision model is to submit the trained model to the central module without leaking local data to ensure the generalization and accuracy of the final training model.
  • the high-precision model is edge intelligence used for local prediction, classification and other practical purposes.
  • the training data used for model training in this embodiment is stored in different data control devices in a decentralized manner according to the main body of the data, and the model training process is set to be performed by the data control device, and the data The processing device only performs fusion generation of the comprehensive model and cannot directly access the training data managed by the data control device.
  • the data processing device will upload the training strategy of the intermediate model when uploading the intermediate model.
  • the training strategy of the intermediate model is taken into consideration, and the fusion of each intermediate model in the fusion algorithm is determined according to the training strategy of the intermediate model. parameters, so that the final accuracy and generalization index of the model are controlled by the data control device, which further improves the privacy and security of the data stored by the data control device.
  • the sixth embodiment of the present application relates to a data processing system, as shown in FIG. 7 , including:
  • a data processing device 701 a data processing device 701, and a central module 7011 provided in the data processing device;
  • the N proxy modules 7021 are respectively set in the N data control devices 702, wherein the central module 7011 is respectively connected in communication with the N proxy modules.
  • the central module 7011 is used to receive the N intermediate models uploaded by the N proxy modules; wherein, the N intermediate models are obtained by the N proxy modules 7021 using the local training data to train the basic model to be trained; Model.
  • the agent module 7021 is used for using the local training data to train the basic model to be trained to obtain an intermediate model; the intermediate model is reported to the central module 7011 for the central module 7011 to generate a comprehensive model.
  • the seventh embodiment of the present application relates to a data processing device, as shown in FIG. 8 , comprising at least one processor 801 ; and at least one memory 802 ; wherein, the memory 802 stores instructions executable by the at least one processor 801 , the instructions are executed by the at least one processor 801 to enable the at least one processor 801 to execute the data processing method in the first or second embodiment.
  • the memory 802 and the processor 801 are connected by a bus, and the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors 801 and various circuits of the memory 802 together.
  • the bus may also connect together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein.
  • the bus interface provides the interface between the bus and the transceiver.
  • a transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium.
  • the data processed by the processor 801 is transmitted on the wireless medium through the antenna, and further, the antenna also receives the data and transmits the data to the processor 801 .
  • the processor 801 is responsible for managing the bus and general processing, and may also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions.
  • the memory 802 may be used to store data used by the processor 801 when performing operations.
  • the eighth embodiment of the present application relates to a data control device, also as shown in FIG. 8 , comprising at least one processor 801 ; and at least one memory 802 ; wherein, the memory 802 stores data that can be executed by the at least one processor 801 .
  • the instructions are executed by the at least one processor 801 to enable the at least one processor 801 to execute the data processing method in the first, second, third, fourth or fifth embodiment.
  • the memory 802 and the processor 801 are connected by a bus, and the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors 801 and various circuits of the memory 802 together.
  • the bus may also connect together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein.
  • the bus interface provides the interface between the bus and the transceiver.
  • a transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium.
  • the data processed by the processor 801 is transmitted on the wireless medium through the antenna, and further, the antenna also receives the data and transmits the data to the processor 801 .
  • the processor 801 is responsible for managing the bus and general processing, and may also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions.
  • the memory 802 may be used to store data used by the processor 801 when performing operations.
  • the ninth embodiment of the present application relates to a computer-readable storage medium storing a computer program.
  • the above method embodiments are implemented when the computer program is executed by the processor.
  • the aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Evolutionary Computation (AREA)
  • General Physics & Mathematics (AREA)
  • Human Resources & Organizations (AREA)
  • General Engineering & Computer Science (AREA)
  • Strategic Management (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computer Hardware Design (AREA)
  • Geometry (AREA)
  • Economics (AREA)
  • Educational Administration (AREA)
  • Development Economics (AREA)
  • Data Mining & Analysis (AREA)
  • Medical Informatics (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Game Theory and Decision Science (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

一种数据处理方法、系统、设备及存储介质，涉及通信技术领域。该数据处理方法应用于模型训练系统中的数据处理设备和N个数据控制设备，数据处理设备分别与N个数据控制设备通信连接，包括：接收N个数据控制设备上传的N个中间模型；其中，N个中间模型由N个数据处理设备训练得到；根据N个中间模型生成综合模型。

Description

数据处理方法、系统、设备及存储介质
交叉引用
本申请基于申请号为“202010606801.8”、申请日为2020年06月29日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以引入方式并入本申请。
技术领域
本申请涉及通信技术领域，特别涉及一种数据处理方法、系统、设备及存储介质。
背景技术
由于数据处理设备能够接触到由数据控制设备上传的大量的原始采集数据，可能导致一系列问题，譬如个人隐私信息或者地区的机密信息被泄露等。
发明内容
本申请的实施例提供了一种数据处理方法，应用于数据处理设备，数据处理设备分别与N个数据控制设备通信连接；方法包括：接收N个数据控制设备上传的N个中间模型；其中，N个中间模型由N个数据处理设备利用本地的训练数据对待训练基础模型进行训练得到；根据N个中间模型生成综合模型。
本申请的实施例还提供了一种数据处理方法，应用于数据控制设备，数据控制设备与数据处理设备通信连接；方法包括：利用本地的训练数据训练待训练基础模型，得到中间模型；将中间模型上报至数据控制设备，供数据控制设备生成综合模型。
本申请的实施例还提供了一种模型训练系统，包括：设置在数据处理设备中的中心模块以及分别设置在N个数据控制设备的N个代理模块；其中，中心模块与N个代理模块分别通信连接；中心模块，用于接收N个数据控制设备上传的N个中间模型；其中，N个中间模型由N个数据处理设备利用本地的训练数据对待训练基础模型进行训练得到；根据N个中间模型生成综合模型；代理模块，用于利用本地的训练数据训练待训练基础模型，得到中间模型；将中间模型上报至数据控制设备，供数据控制设备生成综合模型。
本申请的实施例还提供了一种设备，包括：至少一个处理器，以及与至少一个处理器通信连接的存储器；其中，存储器存储有可被至少一个处理器执行的指令，指令被至少一个处理器执行，以使至少一个处理器能够执行如上述的数据处理方法。
本申请的实施例还提供了一种计算机可读存储介质，存储有计算机程序，计算机程序被处理器执行时实现如上述的数据处理方法。
附图说明
一个或多个实施例通过与之对应的附图中的图片进行示例性说明，这些示例性说明并不构成对实施例的限定。
图1是根据本申请第一实施例中应用于数据处理设备的数据处理方法的流程图；
图2是根据本申请第二实施例中应用于数据处理设备的数据处理方法的流程图；
图3是根据本申请第三实施例中应用于数据控制设备的数据处理方法的流程图；
图4是根据本申请第四实施例中应用于数据控制设备的数据处理方法的流程图；
图5是根据本申请第五实施例中应用于数据控制设备的数据处理方法的流程图；
图6是根据本申请第五实施例中应用于数据控制设备的数据处理方法的流程图；
图7是根据本申请第六实施例中模型训练系统的结构示意图；
图8是根据本申请第七、八实施例中数据控制设备的结构示意图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，下面将结合附图对本申请的各实施方式进行详细的阐述。然而，本领域的普通技术人员可以理解，在本申请各实施例中，为了使读者更好地理解本申请而提出了许多技术细节。但是，即使没有这些技术细节和基于以下各实施方式的种种变化和修改，也可以实现本申请所要求保护的技术方案。以下各个实施例的划分是为了描述方便，不应对本申请的具体实现方式构成任何限定，各个是合理在不矛盾的前提下可以相互结合相互引用。
需要说明的是，虽然在装置示意图中进行了功能模块划分，在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于装置中的模块划分，或流程图中的顺序执行所示出或描述的步骤。说明书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。
随着通信技术的发展，数据的收集变得越来越容易，网络用户能够分析处理的数据越来越多。在数据处理设备模型训练质量的角度而言，用于模型训练的数据量越大，则模型的精度和泛化指标越良好。目前，数据处理设备会根据需要，通过网络从不同的数据控制设备获取原始采集数据以训练模型，其中，不同的数据控制设备可以遍布在不同地区且用于采集并存储其所在区域的数据。比如，不同地区的电力物联网系统可以理解为不同的数据控制设备，而电力物联网系统用于实际执行电力调度方案的电子设备可以理解为数据处理设备；又比如，不同地区的医疗系统可以理解为不同的数据控制设备，医疗系统的调度部门可以理解为数据处理设备。
由于数据处理设备能够接触到由数据控制设备上传的大量的原始采集数据，可能导致一系列问题，譬如个人隐私信息或者地区的机密信息被泄露等。
本申请实施例的目的在于提供一种数据处理方法、系统、设备及存储介质，避免数据处理设备直接接触所有数据控制设备的数据，保证模型训练精度的同时，提高模型训练数据的隐私性。
本申请的第一实施例涉及一种数据处理方法，应用于数据处理设备，数据处理设备是用于利用经过海量数据训练后的模型来进行数据分析处理的终端设备；数据控制设备是用于模型训练以及对存储数据进行管理的终端设备。本实施例中所提到的训练数据分别分布在N个数据控制设备中，N为大于等于1的自然数。数据处理设备分别与N个数据控制设备通信连接；数据处理设备接收N个数据控制设备上传的N个中间模型；其中，N个中间模型由N个 数据控制设备利用本地的训练数据对待训练基础模型进行训练得到；根据N个中间模型生成综合模型。
下面结合附图对本实施例中数据处理方法的实施细节进行具体的说明，以下内容仅为方便理解提供的实施细节，并非实施本方案的必须。本实施例可以广泛应用在分布式数据存储的架构中，用来保证大量数据的数据隐私安全；分布式数据存储结构可以用来存储医疗系统中的病患信息、教育系统中的学生信息、交通预测系统中的车辆信息、电力系统中的环境数据等。以电网架构为例，不同地区具有独立的电力物联网系统，电力物联网系统是基于物联网设备构成的，通过读取并保存摄像头、声音传感器、气体传感器、温度湿度传感器、压力传感器、光传感器、磁场传感器、运动传感器、RFRD射频识别系统以及GPS终端等物联网设备所采集的信息，来实现该地区内的电力调度。由于电力系统调度需要合理的调度方案，因此采用大量数据来训练出一个电力调度模型作为调度方案能够大大提升电网的使用效率。在相关技术中，电力物联网系统获取各地区的特征数据，并利用获取到的各地区的特征数据训练模型；但特征数据传输中存在数据泄密风险。
数据处理设备既可以是独立于数据控制设备的另一设备，也可以是从多个数据控制设备中选出的一个数据控制设备，选出的该数据控制设备执行数据处理设备的功能。同样以电力物联网系统为例，可以将各个地区的电力物联网系统作为数据控制设备，电力调度部门中负责执行电力调度方案的电子设备可以是与各个地区的电力物联网系统完全独立的一个设备；或者，也可以从各个地区的电力物联网系统中选定一个电力物联网系统作为电力调度部门，即执行数据处理设备的功能。如果数据处理设备是从多个数据控制设备中选择出来的，可以通过预设的规则来挑选出合适的数据控制设备作为数据处理设备，例如选硬件性能最佳的数据控制设备来作为数据处理设备。
本实施例中的数据处理方法的执行主体为数据处理设备，具体流程如图1所示，包括：
步骤101，通过网络接收N个数据控制设备上传的N个中间模型。
具体地说，数据处理设备接收的N个中间模型分别由N个数据控制设备训练得到。训练用的数据分布在N个数据控制设备中，各数据控制设备分别利用本地存储的训练数据对待训练基础模型进行训练，得到中间模型后将中间模型上传至数据处理设备。其中，待训练基础模型可以由数据处理设备预先通过模型训练任务携带并下发至数据控制设备，模型训练任务用于指示数据控制设备进行模型训练。
步骤102，基于融合算法，将N个中间模型融合生成综合模型。
具体地说，由于各个数据控制设备所存储的训练数据不同，因此训练所得到的中间模型的精度与泛化指标也有差异，为了得到一个精度与泛化指标均能够满足需求的模型，数据处理设备将N个中间模型进行融合，来生成得到最终用于进行数据分析处理的综合模型。与相关技术相比，本实施例中用于模型训练的训练数据以去中心化的方式根据数据的主体分别存储在不同的数据控制设备中，将模型训练的过程设置由数据控制设备进行，数据处理设备仅进行综合模型的融合生成而无法直接接触到数据控制设备所管理的训练数据，从而使得训练数据实现了物理隔离，有效地达到了数据隐私保护的目的。同时，用于模型训练的训练数据总量没有发生改变，最终用于数据处理设备所获得的模型的精度和泛化指标也能够得到保证。
本申请的第二实施例涉及一种数据处理方法，本实施例的执行主体为数据处理设备。在本实施例中，数据处理设备会在生成综合模型后对综合模型的精度等指标进行评估。
下面结合附图对本实施例中数据处理方法的实施细节进行具体的说明，具体流程如图2所示，包括：
步骤201，通过网络向N个数据控制设备下发模型训练任务。
具体的说，数据处理设备向数据控制设备下发用于指示数据控制设备进行模型训练的模型训练任务，模型训练任务中包含以下至少其中之一：策略指导信息、待训练基础模型。策略指导信息用于指导数据处理设备确定训练待训练基础模型所采用的训练策略。
在一个例子中，模型训练任务中的策略指导信息由解析后的操作指令构成。数据处理设备在向N个数据控制设备下发模型训练任务之前，会根据预存的数据字典与数据控制设备的映射关系，确定N个数据控制设备对应的数据字典。然后根据数据控制设备对应的数据字典解析预设的操作指令，得到策略指导信息。
进一步地说，解析后的操作指令为操作原语，例如：
进行冗余数据移除操作的原语：数据表名：UserName；冗余数据移除比例：自适应冗余数据移除参数：value1,value2；期望完成时间：1秒；执行压缩点生成操作的原语：执行数据表：UserInfo；压缩比率设置：自适应；期望完成时间：1秒；执行某数据表的数据训练操作的原语为：数据表名：UserInfoDemo字段列表：F1、F2、F3；标签字段：L；算法：逻辑回归；期望完成时间：10秒。
由于在实际应用中各个数据控制设备的数据主体不同，所以各个数据控制设备所能够识别的操作原语的类型也不相同。因此数据处理设备在解析预设的操作指令之前，需要获取各个数据控制设备的数据字典，然后并将数据控制设备与数据字典之间的映射关系进行保存，当数据处理设备对预设的操作指令进行解析时，会在数据控制设备与数据字典之间的映射关系查找对应的数据字典，然后将操作指令转换为数据控制设备能够识别的操作原语，使得数据处理设备能够兼容所有不同的数据控制设备。
步骤202，通过网络接收N个数据控制设备上传的N个中间模型。
该步骤与本申请第一实施例中的步骤101类似，相关的实施细节已在第一实施例中具体说明，在此不再赘述。
步骤203，基于融合算法，将N个中间模型融合生成综合模型。
具体地说，本实施例中，数据处理设备根据预设的融合算法来合并数据控制设备上传的各中间模型，具体的合并过程收到融合参数的影响。
在一个例子中，数据处理设备在合并中间模型前接收到数据控制设备上传的N个中间模型的训练策略，中间模型的训练策略是指对待训练基础模型进行训练所采用的训练策略。在中间模型合并的过程中，首先根据N个中间模型的训练策略确定N个中间模型在预设的融合算法中的融合参数。然后根据N个中间模型的融合参数以及融合算法将N个中间模型进行融合，得到综合模型。数据处理设备在合并中间模型时，将数据处理设备中间模型的训练策略纳入考量，根据中间模型的训练策略确定各中间模型在融合算法中的融合参数，使得模型最终的精度以及泛化指标受到数据控制设备的实际训练策略的影响，可以提高融合得到的综合模型的准确度。
在另一个例子中，数据控制设备可以在向数据处理设备上传中间模型的同时上传训练策略，即，数据处理设备同时接收到中间模型以及中间模型的训练策略。
在一个例子中，融合算法可以采用加权算法，此时融合参数为各个中间模型在加权算法中的权重值，各个中间模型的权重值根据训练策略来确定。当数据控制设备所采用的训练策略为单一策略时，则仅根据单一的训练策略来确定各个中间模型在加权算法中的权重值；若采用多种策略来进行模型训练，则综合考虑各个训练策略来确定各个中间模型在加权算法中的权重值。采用加权算法能够使得融合所得的综合模型具有更好的精度。
进一步说，中间模型的训练策略可以包括：隐私保护策略、数据训练策略、压缩点生成策略、冗余数据移除策略、性能指标比计算策略以及自适应资源分配策略。
数据控制设备可以采用性能指标比计算策略来获取各个数据控制设备当前的可用硬件资源指标。该策略通过使用系统API监控当前数据控制设备的各种可用资源，例如网络带宽，硬盘I/O情况，cpu使用率等，来获得当前数据控制设备的性能指标。为了获得当前数据控制设备相对于其他数据控制设备的性能状况，更好的提高整体系统的效率。代理模块会将本地的硬件资源状况定时发送给数据处理设备，数据处理设备汇总所有的资源指标之后，计算出每个代理模块的相对资源指标，即性能指标比，最后发送回数据控制设备。当使用加权算法时进行模型融合时，若数据控制设备的性能指标比较高，则会使数据控制设备训练所得的中间模型在融合算法中的权重值较高。
此外，若数据控制设备在进行中间模型的训练时还采用了其他的训练策略，例如用于对数据进行压缩的压缩点生成策略。压缩点包括两种，分别是粗粒度压缩点和细粒度压缩点。其中一个粗粒度压缩点对应着多个细粒度压缩点，当粗粒度压缩点不足以判断当前压缩点对于融合参数更新的影响时，则计算该粗粒度压缩点对应的细粒度压缩点，得到更加精确的融合参数。
在一个例子中，采用的融合算法为加权算法，基于中间模型的训练策略设置中间模型的权重值，可以包括如下方式：对于采用了压缩点生成策略训练得到的中间模型而言，其权重值可以设置得相对低一些；或者，对于两个中间模型，如果都采用了压缩点生成策略训练得到，采用细粒度压缩点方式训练得到的中间模型，相对于采用粗粒度压缩点方式训练得到的中间模型而言，权重值要设置得更小些。
下面对本实施例中所提到的其他训练策略进行说明：
数据训练策略：可以指定代理模块进行数据训练时所用的算法，并且执行该算法。该策略的具体选择多种多样，比如选择卷积神经网络，逻辑回归，强化学习等算法。数据控制者可以依据需要自行设定，同时也接受数据处理设备发送的策略指导信息。首先，该策略需要确定当前的训练策略，确定当前需要使用的数据训练算法。然后按照设定调配资源，进行训练操作。通过数据训练策略最终可以生成的中间模型。
冗余数据移除策略：利用压缩点生成策略生成的粗粒度压缩点和细粒度压缩点进行冗余数据移除策略。首先计算出所有粗粒度压缩点对数据训练参数更新的影响值，依据这个影响值来判断这个压缩点的数据是否冗余。遍历所有的粗粒度压缩点，保留影响值较大的压缩点。同时对于影响值较小的粗粒度压缩点进行细粒度压缩，生成多个细粒度压缩点，然后判断当前细粒度压缩点的对于参数更新的影响值，去除冗余数据，保留有价值的数据。
自适应资源分配策略：利用当前数据控制设备的性能指标比来计算当前代理模块进行数据训练时所应当使用的资源以及训练数据量。本策略可以解决代理模块之间数据训练进度不同步的问题，实现了总体的训练效率的提升。
步骤204，对综合模型进行评估，判断综合模型的精度是否满足预设阈值；若综合模型的精度满足预设阈值，则保留综合模型；若综合模型的精度不满足预设阈值，则执行步骤201。
具体地说，为了使得最终用于数据分析处理的综合模型能够满足需求。在数据处理设备合并中间模型得到综合模型后，会对综合模型进行评估，判断模型的精度或泛化指标等是否满足预设的阈值。当综合模型的精度满足预设阈值时，则保留综合模型，可以将该综合模型下发给各数据控制设备，数据控制设备可以采用该综合模型来进行数据分析处理。当综合模型的精度不满足预设阈值时，则将该综合模型作为待训练基础模型加入到模型训练任务中下发至数据控制设备，供数据控制设备再次进行模型训练，以进一步提高模型的精度或泛化指标等。
在具体的实现中，当本实施例应用在不同业务场景下时，模型精度评估方式不定相同，如果是指标预测场景，可采用RMSE(均方根误差)、MAE(平均绝对误差)、R方等评估指标；如果是分类场景，可采用精确率、召回率、准确率、错误率、F1等评估指标；如果是聚类，可采用Davies-Bouldin Index(戴维森堡丁指数)、Dunn Validity Index(邓恩指数)等评估指标。上述指标只是纯数学方面的，实际应用中，精度评估指标有很多变种，例如下文提到的关于电力物联网系统的应用场景中，可以采用一段时间内无效发电量的波动率来作为电力方案的评估指标。从电力系统的特点上来说，即，当电力系统在一段时间内的无效发电量较小，且没有大的起伏的情况下，则表示作为综合模型的电力调度方案的精度较好。
在一个例子中，若模型训练任务中包含策略指导信息，数据处理设备在模型训练任务中加入之前生成的综合模型作为待训练基础模型的同时，可以继续沿用上一次模型训练任务中的策略指导信息；也可以根据数据分析处理的需求对策略指导信息进行修改，以更好地提高模型训练的效率以及训练所得的中间模型的精度或泛化指标。
下面以电力物联网系统为应用场景进一步对本实施例进行解释。
本例子中，数据处理设备是电力物联网系统中执行电力调度方案的电子设备；数据控制设备是不同地区的电力系统。
首先数据控制设备从数据处理设备处获取支持数据操作原语定义、数据处理者的接入识别符、策略描述符等信息；数据控制设备设置数据处理设备的等级、操作原语对应的执行动作流和数据获取屏蔽规则等策略；数据处理设备从多个数据控制设备获取数据字典信息。
数据处理设备以当前的电力调度方案作为待训练基础模型，以模型训练任务携带待训练基础模型发送至各个数据控制设备。具体的说，对于多个地区而言，由于其发电量是由本地区的电厂决定的，故需要对电力调度进行合理的分配，以适应普通民众的生活用电以及各种生产部门(例如工厂等)的生产用电。因此我们将默认的电力调度方案作为待训练基础模型，把待训练基础模型和任务启动信息发送至各个数据控制设备。
数据控制设备在接收到数据处理设备下发的待训练基础模型后，首先按照该模型中电力调度方法部署本地区的电力调度方案，这保证了每次数据控制设备获得了新的模型之后都能及时部署电力调度方案。然后根据模型训练任务中的策略指导信息以及本地的规则进行数据 训练。其中使用的数据集为本地区电力物联网系统感知层收集到的信息；数据训练策略由数据控制设备自适应选择，可用的策略有以下几种：压缩点生成策略，冗余数据移除策略，性能指标比计算策略等；数据方法由数据处理设备统一指定，例如分类算法，聚类算法，多层的神经网络算法等。
数据控制设备在进行了一次数据训练之后，得到了当前训练的中间模型以及训练使用的策略。首先数据控制设备对中间模型使用自适应隐私保护策略进行隐私保护操作，这样做的目的时为了防止本地的电力使用情况、电力调度方案在传输过程中泄露出去，导致隐私的不安全。然后每一个数据控制设备将本地的中间模型传输到数据处理设备，以便数据处理设备合并各个中间模型，保证最后获得的综合模型的泛化性。
接着数据处理设备合并模型并评测模型，当数据处理设备获得了当前所有的数据控制设备的中间模型之后，将中间模型合并，得到综合模型。利用数据处理设备的测试数据对综合模型(即电力调度方案)进行评测。如果综合模型符合要求，则将综合模型下发；否则再次下发模型训练任务，由数据控制设备根据新的策略指导信息重新训练出新的中间模型。
需要说明的是，本实施例中的上述各示例均为方便理解进行的举例说明，并不对本申请的技术方案构成限定。
与相关技术相比，本实施例中用于模型训练的训练数据以去中心化的方式根据数据的主体分别存储在不同的数据控制设备中，将模型训练的过程设置由数据控制设备进行，数据处理设备仅进行综合模型的融合生成而无法直接接触到数据控制设备所管理的训练数据。此外，在合并中间模型时，将中间模型的训练策略纳入考量，根据中间模型的训练策略确定各中间模型在融合算法中的融合参数，使得模型最终的精度以及泛化指标受到数据控制设备的控制，进一步提高了数据控制设备所存储数据的隐私安全性。
本申请的第三实施例涉及一种数据处理方法，应用于数据控制设备，数据控制设备是用于模型训练以及对存储数据进行管理的终端设备；数据处理设备是用于利用经过海量数据训练后的模型来进行数据分析处理的终端设备。本实施例中所提到的训练数据分别分布在N个数据控制设备中，N为大于等于1的自然数。数据控制设备与数据处理设备通信连接；方法包括：利用本地的训练数据训练待训练基础模型，得到中间模型；将中间模型上报至数据控制设备，供数据控制设备生成综合模型。
下面结合附图对本实施例中数据处理方法的实施细节进行具体的说明，以下内容仅为方便理解提供的实施细节，并非实施本方案的必须。
本实施例中的数据处理方法的执行主体为数据控制设备，具体流程如图3所示，包括：
步骤301，利用本地的训练数据训练待训练基础模型，得到中间模型。
具体地说，数据控制设备在本地存储着大量用于进行模型训练的训练数据。待训练基础模型是数据控制设备预存的初始模型，在向待训练基础模型输入训练数据进行训练后，得到中间模型。
在一个例子中，待训练基础模型由数据处理设备生成并下发，数据控制设备对待训练基础模型进行训练得到中间模型。
步骤302，将中间模型上报至数据处理设备，供数据处理设备生成综合模型。
具体地说，由于多个数据控制设备的数据主体不同，训练数据的样本也不同，因此不同的数据控制设备训练所得的中间模型的精度和泛化指标均有所差异。为了使数据处理设备最终采用的模型的精度以及泛化指标数据分析处理的需求，各数据控制设备将训练所得的中间模型上传至数据处理设备，供数据处理设备将各中间模型进行合并来生成综合模型。
可以发现，本实施例是与本申请第一实施例以及第二实施例相互配合实施的、应用于数据控制设备的数据处理方法，在本申请第一实施例中所提到的技术细节在本实施例中也同样可以实现，为了减少重复，在这里不再赘述。
与相关技术相比，本实施例中用于模型训练的训练数据以去中心化的方式根据数据的主体分别存储在不同的数据控制设备中，将模型训练的过程设置由数据控制设备进行，数据处理设备仅进行综合模型的融合生成而无法直接接触到数据控制设备所管理的训练数据，从而使得训练数据实现了物理隔离，有效地达到了数据隐私保护的目的。同时，用于模型训练的训练数据总量没有发生改变，最终用于数据处理设备所获得的模型的精度和泛化指标也能够得到保证。
本申请的第四实施例涉及一种数据处理方法，本实施例与本申请第三实施例的主要区别在于，数据控制设备在利用训练数据对待训练基础模型进行训练时，会根据本地的策略执行规则来确定中间模型的训练策略，训练完成后，还会将训练所得的中间模型的训练策略上传至数据处理模块。
本实施例中的数据处理方法的执行主体为数据控制设备，具体流程如图4所示，包括：
步骤401，接收数据处理设备下发的模型训练任务。
具体地说，本实施例中，数据控制设备在训练待训练基础模型之前，接收数据处理设备下发的模型训练任务，模型训练任务中可以携带策略指导信息和待训练基础模型。当数据控制设备所训练的待训练基础模型由数据处理设备生成时，待训练基础模型由模型训练任务携带下发。
步骤402，利用本地的训练数据训练待训练基础模型，得到中间模型。
具体的说，当数据控制设备接收到模型训练任务后，若模型训练任务中仅包含策略指导信息，则根据策略指导信息以及本地预设的策略执行规则来确定出待训练基础模型的训练策略，此时待训练基础模型为数据控制设备本地预设的模型。
在一个例子中，若模型训练任务仅包含待训练基础模型，则数据控制设备根据本地预设的策略执行规则确定待训练基础模型的训练策略，然后根据训练策略对模型训练任务中的待训练基础模型进行训练。
在一个例子中，若模型训练任务中同时包含策略指导信息以及待训练基础模型，则数据控制设备同时根据策略指导信息以及本地预设的策略执行规则确定出待训练基础模型的训练策略，然后根据训练策略对模型训练任务中的待训练基础模型进行训练。
步骤403，将中间模型上报至数据处理设备，供数据处理设备生成综合模型。
具体的说，在本实施例中，数据控制设备将中间模型和中间模型的训练策略均上报至数据处理设备。数据控制设备可以预先将中间模型的训练策略上报至数据处理设备，也可以同时将中间模型以及中间模型的训练策略上报至数据处理设备。数据处理设备根据中间模型的 训练策略确定出中间模型在融合算法中的融合参数。即，数据处理设备在合并中间模型时，将数据处理设备中间模型的训练策略纳入考量，根据中间模型的训练策略确定各中间模型在融合算法中的融合参数，使得模型最终的精度以及泛化指标受到数据控制设备的实际训练策略的影响，进一步提高了数据控制设备所存储数据的隐私安全性。
本申请的第五实施例涉及一种数据处理方法，本实施例的执行主体为数据控制设备。本实施例与本申请第三实施例的区别在于，数据控制设备在生成中间模型后会对中间模型的精度进行评估，来判断是否继续训练出精度更高的中间模型来进行数据控制设备本地的数据分析处理。
下面结合附图对本实施例中数据处理方法的实施细节进行具体的说明，具体流程如图5所示，包括：
步骤501，利用本地的训练数据训练待训练基础模型，得到中间模型。
具体地说，本实施例中的初始的待训练基础模型是数据控制设备本地预设的，数据控制设备首先根据边缘智能的需求对本地预设的待训练基础模型进行训练，得到中间模型。
在一个实施例中，用于模型训练的训练策略包括：性能指标比计算策略、压缩点生成策略、冗余数据移除策略以及隐私保护策略等。其中，采用性能指标比计算策略能够更好地提高整体系统进行模型训练的效率；采用压缩点生成策略能够对训练数据进行压缩，压缩点有两种，包括：粗粒度压缩点和细粒度压缩点；冗余数据移除策略用来去除不必要的数据以提高模型训练的效率，具体的，首先需要判断所有的粗粒度压缩点对于融合参数更新的影响值，该影响值可以通过模型训练时融合参数获得，例如模型参数梯度，保留影响值高的粗粒度压缩点，去除影响值低的。对于那些不容易判断的压缩点，考虑其细粒度压缩点对于融合参数更新的影响值，保留影响值高的，去除影响值低的，最后得到了去除冗余数据的训练数据样本；采用隐私保护策略则在上传中间模型时，对中间模型进行隐私保护操作，例如进行中间模型的噪声添加操作。
数据处理模块可以在下发的模型训练任务中携带模型训练任务，从而实现指导数据处理设备确定训练待训练基础模型所采用的训练策略。但由于数据的具体训练过程仅受到数据控制设备的控制，因此策略指导信息只能起到参考作用，最终实际执行训练策略不一定完全满足数据处理设备的需求，因此数据控制设备本地的策略执行规则会对训练所得的中间模型的精度产生一定影响。
步骤502，将中间模型上报至数据处理设备，供数据处理设备生成综合模型。
具体地说，数据处理设备在每一次训练出中间模型后，均会将中间模型上传至数据处理设备由数据处理设备根据各个数据控制设备上传的中间模型来生成综合模型。
在一个例子中，在数据控制设备将中间模型上报至数据处理设备时，还会将本次对待训练基础模型进行训练所采用的训练策略也上传到数据处理设备，供数据处理设备根据训练策略来确定出中间模型在融合算法中的融合参数。以加权算法为例，融合参数为加权算法的权重值，在其他训练策略相同的前提下，根据性能指标比计算策略所确定的性能指标比会影响中间模型在加权算法中的权重值。
步骤503，对中间模型进行精度评估，判断中间模型的精度是否满足预设阈值；若中间 模型的精度满足阈值，则保存中间模型进行数据分析处理；若中间模型的精度不满足阈值，则执行步骤504，向数据处理设备拉取综合模型。然后以拉取的综合模型作为待训练基础模型进行模型训练。
具体地说，在边缘智能的场景下，边缘智能节点均需要承担数据处理的责任，因此本实施例中的数据控制设备除了利用本地的训练数据对待训练基础模型进行训练得到中间模型以外，还需要利用训练所得的中间模型来进行数据的分析处理。因此，各个数据控制设备均对本地的中间模型的精度或泛化指标有一定需求。在训练得到中间模型后，会对中间模型的精度等指标进行评估，当中间模型的精度满足阈值，则保存中间模型进行数据分析处理；当中间模型的精度不满足阈值，则向数据处理设备拉取综合模型，以拉取的综合模型作为待训练基础模型再次利用本地的训练数据进行模型训练。
在另一个例子中，当对中间模型进行精度评估后，中间模型的精度满足预设阈值时，本实施例中的数据处理方法如图6所示，包括：
步骤601，利用本地的训练数据训练待训练基础模型，得到中间模型。
该步骤与本实施例中的步骤601类似，相关的实施细节已经说明，在此不再赘述。
步骤602，当中间模型的精度满足预设阈值，对中间模型进行低精度处理，得到低精度的中间模型。
步骤603，将低精度的中间模型上报至数据处理设备。
具体的说，低精度模型的主要作用是在不泄露本地数据前提下，将训练出来的模型提交到中心模块，以保证最后训练模型的泛化性和准确性。而高精度模型则是边缘智能用于本地的预测，分类等实际用途中。
需要说明的是，本实施例中的上述各示例均为方便理解进行的举例说明，并不对本申请的技术方案构成限定。
与相关技术相比，本实施例中用于模型训练的训练数据以去中心化的方式根据数据的主体分别存储在不同的数据控制设备中，将模型训练的过程设置由数据控制设备进行，数据处理设备仅进行综合模型的融合生成而无法直接接触到数据控制设备所管理的训练数据。此外，数据处理设备在上传中间模型的同时会上传中间模型的训练策略，在合并中间模型时，将中间模型的训练策略纳入考量，根据中间模型的训练策略确定各中间模型在融合算法中的融合参数，使得模型最终的精度以及泛化指标受到数据控制设备的控制，进一步提高了数据控制设备所存储数据的隐私安全性。
本申请的第六实施例涉及一种数据处理系统，如图7所示，包括：
数据处理设备701，以及设置在数据处理设备中的中心模块7011；
分别设置在N个数据控制设备702的N个代理模块7021，其中，中心模块7011与N个代理模块分别通信连接。
中心模块7011，用于接收N个代理模块上传的N个中间模型；其中，N个中间模型由N个代理模块7021利用本地的训练数据对待训练基础模型进行训练得到；根据N个中间模型生成综合模型。
代理模块7021，用于利用本地的训练数据训练待训练基础模型，得到中间模型；将中间 模型上报至中心模块7011，供中心模块7011生成综合模型。
本申请的第七实施例涉及一种数据处理设备，如图8所示，包括至少一个处理器801；以及，至少一个存储器802；其中，存储器802存储有可被至少一个处理器801执行的指令，指令被至少一个处理器801执行，以使至少一个处理器801能够执行第一或第二实施例中的数据处理方法。
其中，存储器802和处理器801采用总线方式连接，总线可以包括任意数量的互联的总线和桥，总线将一个或多个处理器801和存储器802的各种电路连接在一起。总线还可以将诸如外围设备、稳压器和功率管理电路等之类的各种其他电路连接在一起，这些都是本领域所公知的，因此，本文不再对其进行进一步描述。总线接口在总线和收发机之间提供接口。收发机可以是一个元件，也可以是多个元件，比如多个接收器和发送器，提供用于在传输介质上与各种其他装置通信的单元。经处理器801处理的数据通过天线在无线介质上进行传输，进一步，天线还接收数据并将数据传送给处理器801。处理器801负责管理总线和通常的处理，还可以提供各种功能，包括定时，外围接口，电压调节、电源管理以及其他控制功能。而存储器802可以被用于存储处理器801在执行操作时所使用的数据。
本申请的第八实施例涉及一种数据控制设备，同样如图8所示，包括至少一个处理器801；以及，至少一个存储器802；其中，存储器802存储有可被至少一个处理器801执行的指令，指令被至少一个处理器801执行，以使至少一个处理器801能够执行第一、二、第三、第四或第五实施例中的数据处理方法。
其中，存储器802和处理器801采用总线方式连接，总线可以包括任意数量的互联的总线和桥，总线将一个或多个处理器801和存储器802的各种电路连接在一起。总线还可以将诸如外围设备、稳压器和功率管理电路等之类的各种其他电路连接在一起，这些都是本领域所公知的，因此，本文不再对其进行进一步描述。总线接口在总线和收发机之间提供接口。收发机可以是一个元件，也可以是多个元件，比如多个接收器和发送器，提供用于在传输介质上与各种其他装置通信的单元。经处理器801处理的数据通过天线在无线介质上进行传输，进一步，天线还接收数据并将数据传送给处理器801。处理器801负责管理总线和通常的处理，还可以提供各种功能，包括定时，外围接口，电压调节、电源管理以及其他控制功能。而存储器802可以被用于存储处理器801在执行操作时所使用的数据。
本申请的第九实施例涉及一种计算机可读存储介质，存储有计算机程序。计算机程序被处理器执行时实现上述方法实施例。
即，本领域技术人员可以理解，实现上述方法实施例中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，该程序存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
本领域的普通技术人员可以理解，上述是实现本申请的具体实施例，而在实际应用中，可以在形式上和细节上对其作各种改变，而不偏离本申请的精神和范围。

Claims (17)

  1. 一种数据处理方法，应用于数据处理设备，所述数据处理设备分别与N个数据控制设备通信连接；所述方法包括：
    通过网络接收所述N个数据控制设备上传的N个中间模型；其中，所述N个中间模型由所述N个数据控制设备利用本地的训练数据对待训练基础模型进行训练得到；
    基于融合算法，将所述N个中间模型融合生成综合模型。
  2. 根据权利要求1所述的数据处理方法，其中，在所述接收所述N个数据控制设备上传的N个中间模型之前，还包括：
    向所述N个数据控制设备下发模型训练任务，所述模型训练任务包括以下至少一个：策略指导信息、所述待训练基础模型；
    其中，所述策略指导信息用于指导所述数据控制设备选取训练所述待训练基础模型采用的训练策略。
  3. 根据权利要求1至2中任一项所述的数据处理方法，其中，在所述基于融合算法，将所述N个中间模型融合生成综合模型之后，还包括：
    对所述综合模型进行精度评估；
    当所述综合模型的精度指标不满足预设阈值，将所述综合模型作为所述待训练基础模型，并向所述N个数据控制设备下发所述待训练基础模型和新的策略指导信息，并接收所述N个数据控制设备利用本地的训练数据对所述待训练基础模型进行训练得到的N个新的中间模型；根据所述N个新的中间模型生成综合模型。
  4. 根据权利要求1至3中任一项所述的数据处理方法，其中，在所述基于融合算法，将所述N个中间模型融合生成综合模型之前，还包括：
    接收所述N个数据控制设备上传的所述N个中间模型的训练策略；所述中间模型的训练策略是指对所述待训练基础模型进行训练所采用的策略；
    所述基于融合算法，将所述N个中间模型融合生成综合模型，包括：
    根据所述N个中间模型的训练策略，确定所述N个中间模型在预设的融合算法中的融合参数；
    根据所述N个中间模型的融合参数以及所述融合算法，将所述N个中间模型进行融合，得到所述综合模型。
  5. 根据权利要求4所述的数据处理方法，其中，所述融合算法为加权算法，所述融合参数为权重值。
  6. 根据权利要求2所述的数据处理方法，其中，所述模型训练任务包括所述策略指导信息；
    在所述向所述N个数据控制设备下发模型训练任务之前，还包括：
    根据预存的数据字典与所述数据控制设备的映射关系，确定各所述数据控制设备对应的数据字典；
    根据各所述数据控制设备对应的数据字典，将向各所述数据控制设备下发的模型训练任务中的策略指导信息解析为能够被各所述数据控制设备识别的形式。
  7. 一种数据处理方法，应用于数据控制设备，所述数据控制设备与数据处理设备通信连接；所述方法包括：
    利用本地的训练数据训练待训练基础模型，得到中间模型；
    将所述中间模型上报至所述数据处理设备，供所述数据处理设备生成综合模型。
  8. 根据权利要求7所述的数据处理方法，其中，在所述利用本地的训练数据训练待训练基础模型之前，还包括：
    接收所述数据处理设备下发的策略指导信息；
    所述利用本地的训练数据训练待训练基础模型，得到中间模型，包括：
    根据所述策略指导信息以及本地的策略执行规则确定训练所述待训练基础模型的训练策略；
    根据所述训练策略，利用本地的训练数据训练所述待训练基础模型，得到所述中间模型。
  9. 根据权利要求7至8中任一项所述的数据处理方法，其中，所述利用本地的训练数据训练待训练基础模型，得到中间模型，包括：
    根据本地的策略执行规则确定训练所述待训练基础模型的训练策略；
    根据所述训练策略确定数据训练操作；
    对所述训练数据执行所述数据训练操作，得到所述中间模型。
  10. 根据权利要求9所述的数据处理方法，其中，所述待训练基础模型的训练策略包括：性能指标比计算策略、用于预处理所述训练数据的策略；
    所述根据本地的策略执行规则确定训练所述待训练基础模型的训练策略，包括：
    根据所述性能指标比计算策略确定本地的性能指标比；
    根据所述性能指标比确定出用于预处理所述训练数据的策略。
  11. 根据权利要求10所述的数据处理方法，其中，所述根据所述性能指标比计算策略获取本地的性能指标比，包括：
    获取本地的硬件资源使用情况，将所述硬件资源使用情况上传至所述数据处理设备；其中，所述数据处理设备根据N个数据控制设备上传的所述硬件资源使用情况生成所述性能指标比；
    获取所述数据处理设备发送的所述性能指标比。
  12. 根据权利要求8至9中任一项所述的数据处理方法，其中，在所述利用本地的训练 数据训练待训练基础模型，得到中间模型之后，还包括：
    当所述中间模型的精度不满足预设阈值，则向所述数据处理设备拉取所述综合模型，并将拉取到的所述综合模型作为所述待训练基础模型，执行所述利用本地的训练数据训练待训练基础模型，得到中间模型。
  13. 根据权利要求12所述的数据处理方法，其中，所述将所述中间模型上报至所述数据处理设备，包括：
    当所述中间模型的精度满足预设阈值，对所述中间模型进行低精度处理，得到低精度的所述中间模型；
    将低精度的所述中间模型上报至所述数据处理设备。
  14. 根据权利要求8至13任一项所述的数据处理方法，其中，在所述利用本地的训练数据训练待训练基础模型，得到中间模型之后，还包括：
    向所述数据处理设备上传所述中间模型的训练策略；其中，所述训练策略是指对所述待训练基础模型进行训练所采用的策略。
  15. 一种数据处理系统，包括：设置在数据处理设备中的中心模块以及分别设置在N个数据控制设备的N个代理模块；其中，所述中心模块与N个所述代理模块分别通信连接；
    所述中心模块，用于接收所述N个代理模块上传的N个中间模型；根据所述N个中间模型生成综合模型；
    所述代理模块，用于利用本地的训练数据训练待训练基础模型，得到中间模型；将所述中间模型上报至所述中心模块。
  16. 一种设备，包括：
    至少一个处理器；以及，
    与所述至少一个处理器通信连接的存储器；其中，
    所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行如权利要求1至6中所述的数据处理方法，和/或，权利要求7至14中任一项所述的数据处理方法。
  17. 一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1至6中任一项所述的数据处理方法，和/或，如权利要求7至14中任一项所述的数据处理方法。
PCT/CN2021/103193 2020-06-29 2021-06-29 数据处理方法、系统、设备及存储介质 WO2022002068A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010606801.8A CN113935390A (zh) 2020-06-29 2020-06-29 数据处理方法、系统、设备及存储介质
CN202010606801.8 2020-06-29

Publications (1)

Publication Number Publication Date
WO2022002068A1 true WO2022002068A1 (zh) 2022-01-06

Family

ID=79273048

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/103193 WO2022002068A1 (zh) 2020-06-29 2021-06-29 数据处理方法、系统、设备及存储介质

Country Status (2)

Country Link
CN (1) CN113935390A (zh)
WO (1) WO2022002068A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115611393A (zh) * 2022-11-07 2023-01-17 中节能晶和智慧城市科技(浙江)有限公司 一种多端协同的多水厂混凝剂投放方法和系统
CN117194991A (zh) * 2023-11-03 2023-12-08 四川并济科技有限公司 基于gpu集群的高维数据推荐系统及方法
CN117272688A (zh) * 2023-11-20 2023-12-22 四川省交通勘察设计研究院有限公司 一种结构力学仿真数据的压缩与解压方法、装置及系统

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104143327A (zh) * 2013-07-10 2014-11-12 腾讯科技（深圳）有限公司 一种声学模型训练方法和装置
CN105575389A (zh) * 2015-12-07 2016-05-11 百度在线网络技术（北京）有限公司 模型训练方法、系统和装置
CN108431794A (zh) * 2016-03-18 2018-08-21 微软技术许可有限责任公司 用于训练学习机的方法和装置
CN109600255A (zh) * 2018-12-04 2019-04-09 中山大学 一种去中心化的参数服务器优化算法
US20200027009A1 (en) * 2018-07-23 2020-01-23 Kabushiki Kaisha Toshiba Device and method for optimising model performance
CN110795477A (zh) * 2019-09-20 2020-02-14 平安科技（深圳）有限公司 数据的训练方法及装置、系统
CN111081337A (zh) * 2020-03-23 2020-04-28 腾讯科技（深圳）有限公司 一种协同任务预测方法及计算机可读存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104143327A (zh) * 2013-07-10 2014-11-12 腾讯科技（深圳）有限公司 一种声学模型训练方法和装置
CN105575389A (zh) * 2015-12-07 2016-05-11 百度在线网络技术（北京）有限公司 模型训练方法、系统和装置
CN108431794A (zh) * 2016-03-18 2018-08-21 微软技术许可有限责任公司 用于训练学习机的方法和装置
US20200027009A1 (en) * 2018-07-23 2020-01-23 Kabushiki Kaisha Toshiba Device and method for optimising model performance
CN109600255A (zh) * 2018-12-04 2019-04-09 中山大学 一种去中心化的参数服务器优化算法
CN110795477A (zh) * 2019-09-20 2020-02-14 平安科技（深圳）有限公司 数据的训练方法及装置、系统
CN111081337A (zh) * 2020-03-23 2020-04-28 腾讯科技（深圳）有限公司 一种协同任务预测方法及计算机可读存储介质

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115611393A (zh) * 2022-11-07 2023-01-17 中节能晶和智慧城市科技(浙江)有限公司 一种多端协同的多水厂混凝剂投放方法和系统
CN117194991A (zh) * 2023-11-03 2023-12-08 四川并济科技有限公司 基于gpu集群的高维数据推荐系统及方法
CN117194991B (zh) * 2023-11-03 2024-02-13 四川并济科技有限公司 基于gpu集群的高维数据推荐系统及方法
CN117272688A (zh) * 2023-11-20 2023-12-22 四川省交通勘察设计研究院有限公司 一种结构力学仿真数据的压缩与解压方法、装置及系统
CN117272688B (zh) * 2023-11-20 2024-02-13 四川省交通勘察设计研究院有限公司 一种结构力学仿真数据的压缩与解压方法、装置及系统

Also Published As

Publication number Publication date
CN113935390A (zh) 2022-01-14

Similar Documents

Publication Publication Date Title
WO2022002068A1 (zh) 数据处理方法、系统、设备及存储介质
Kato et al. Ten challenges in advancing machine learning technologies toward 6G
Tak et al. Federated edge learning: Design issues and challenges
WO2020168761A1 (zh) 训练模型的方法和装置
Salau et al. Recent advances in artificial intelligence for wireless internet of things and cyber–physical systems: A comprehensive survey
Wang Migration strategy of cloud collaborative computing for delay-sensitive industrial IoT applications in the context of intelligent manufacturing
CN113608882B (zh) 基于人工智能和大数据的信息处理方法、系统及云平台
CN112187859B (zh) 物联网业务与边缘网络能力动态映射的方法及电子设备
CN112100928B (zh) 一种基于温度改善锂离子电池性能的方法和装置
CN112235389A (zh) 基于物联网的智能终端数据处理方法及装置
CN111065999A (zh) 移动设备的功率状态控制
Chen et al. Deep-broad learning system for traffic flow prediction toward 5G cellular wireless network
Dahiya et al. Efficient green solution for a balanced energy consumption and delay in the iot-fog-cloud computing
EP3783840A1 (en) Deep learning-based bandwidth filtering method and device, server, and storage medium
Zhang et al. Deep reinforcement learning algorithm for latency-oriented IIoT resource orchestration
Zhang Storage optimization algorithm design of cloud computing edge node based on artificial intelligence technology
Tripathi et al. Data-driven optimizations in IoT: A new frontier of challenges and opportunities
CN112232985B (zh) 用于泛在电力物联网的配用电数据监测方法及装置
CN111291883A (zh) 数据处理方法及数据处理装置
KR101960688B1 (ko) 사물지능통신 기기의 전원 관리 방법，및 사물지능통신 기기의 전원 관리를 위한 네트워크 서버
DE102022124386A1 (de) Verfahren und vorrichtungen, um den netzwerkverkehr auszurichten und den leistungsverbrauch zu verbessern
CN115827232A (zh) 一种为业务模型确定配置的方法、装置、系统及设备
US20210014088A1 (en) Method and apparatus for bandwidth filtering based on deep learning, server and storage medium
ur Rehman et al. LTE downlink throughput modeling using neural networks
CN111092755B (zh) 一种基于资源占用的边缘服务迁移仿真方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21832653

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 16/05/2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21832653

Country of ref document: EP

Kind code of ref document: A1