CN113222169B - Federated Machine Composition Service Method and System Combined with Big Data Analysis Feedback - Google Patents

Abstract

The invention discloses a federal machine combination service system combining big data analysis feedback, which comprises a plurality of devices (1) distributed at different addresses and a big data analysis and scheduling module (2); each device (1) comprises a data preprocessing module (3), a data acquisition module (6) and a data reading module (11); the big data analysis and scheduling module (2) analyzes and schedules all the equipment, the working module and the data progress data which participate in the federal machine learning; the big data analysis and scheduling module (2) is in data communication connection with the data reading module (11), the data training fusion sub-module (4) and the federal data training module (5); the data acquisition module (6) is in data communication connection with the single machine storage module (7), and the single machine storage module (7) is also in data communication connection with the data preprocessing module (3) and the data reading module (11) respectively. According to the federal machine combination service system combining big data analysis feedback, before data recording training is carried out, data is cleaned on the data records, so that abnormal parts of the data records are removed, the accuracy of the data records is guaranteed, and the accuracy of a data model is guaranteed.

Description

Federated Machine Composition Service Method and System Combined with Big Data Analysis Feedback

technical field

The invention relates to the technical field of intelligent manufacturing, and specifically relates to a federated machine combination service method and system combined with big data analysis and feedback.

Background technique

The twentieth century is the end of intelligent production and intelligent manufacturing. In today's life, equipment is not only intelligent and automatic, but also from the separate operation of equipment in the past to the collaborative operation of today, which is inseparable from Cross-domain, cross-device operation and collaboration, which inevitably involves collaboration between different devices or fields. In traditional technologies, if the usual logical relationship is used to process, not only the processing cycle is long, but a large amount of data logic is required Therefore, in order to adapt to large-scale computing and computing processing capabilities, this will inevitably increase the demand for logical computing capabilities of processors, and large-scale integrated circuits or ultra-large-scale The computing and processing capabilities of integrated circuits also directly affect their production costs. On the one hand, it is necessary to improve the processing capability of the processor itself. On the other hand, it is hoped to reduce the demand for computing power, that is, to optimize the computing and processing requirements of artificial intelligence. Improve the calculation model to reduce the demand for calculation. On the other hand, for all kinds of smart devices, in order to achieve the technical effects of various intelligent processing, it is necessary to combine the data of each sensor device or each institution, perform comprehensive logic calculations, and make a comprehensive judgment after summarizing. There are huge difficulties and economic costs in integrating scattered data, and there are currently technologies for joint access and processing of distributed data, such as federated machine learning, also known as federated learning, federated learning, federated learning, Federated machine learning is a machine learning framework that can effectively help multiple agencies conduct data usage and machine learning modeling while meeting the requirements of user privacy protection, data security, and government regulations;

For example, patent CN 110263936A discloses a horizontal federated learning method, which includes: the community coordinator obtains the global model parameters sent by the central coordinator, and sends the global model parameters to each participant; The model parameter update obtained by model training with parameters, the update of each model parameter is fused to obtain the update of the community model parameter, and it is determined whether the community model parameter update needs to be sent to the central coordinator; if so, the community model parameter update is sent to the central coordinator The coordinator obtains the global model parameter update returned by the central coordinator, and sends the global model parameter update to each participant, so that each participant can perform model training based on the global model parameter update. The invention also discloses a horizontal federated learning device, equipment and a computer storage medium. The invention improves the learning efficiency of horizontal federated learning.

Patent CN111324440A discloses an execution method, device, equipment and readable storage medium of an automated process, and relates to the field of financial technology. The modal data is used as the input of the deep learning model to obtain the intention analysis model; the behavior intention corresponding to the automation process in the terminal is determined through the intention analysis model; the target operation instruction corresponding to the automation process is executed according to the behavior intention to execute The automated process. The invention realizes the analysis of the behavior intention corresponding to the automation process through the intention analysis model, executes the automation process according to the behavior intention, and avoids the failure of the execution of the automation process caused by the change of the execution environment corresponding to the automation process, thereby improving the automation process. The adaptability of the execution environment means that the adaptability of the automation process is improved, and the execution success rate of the automation process is improved.

Patent CN111882308A discloses a blockchain security transaction method, including: collecting transaction request information of each node participating in the security transaction; recording the transaction request information, and verifying the legality; generating a transaction block, and packaging the transaction block to the block of the block chain; obtain and verify the bookkeeping right transferred by the bookkeeping right node; broadcast the verified ownership of the bookkeeping right to each node, so that each node follows the preset The consensus setting algorithm reaches a consensus; the transaction request information is verified through the bookkeeping right, and when the verification is successful, each node is notified to record the transaction request information, and the account information of each node is updated synchronously. The blockchain security transaction method provided by the invention can solve the problems in the prior art, such as prolonged transaction data confirmation, illegal information cannot be tampered with, poor security, and poor privacy.

Patent CN112257876A discloses a federated learning method, device, computer equipment and media, belonging to the field of computer technology. The method includes: the first computer device acquires the sample label information corresponding to the sample identification, and obtains the first fusion information corresponding to the sample identification; the second computer equipment obtains the second fusion information corresponding to the sample identification, and sends the second fusion information to the first computer equipment information; the first computer device obtains the gradient operator corresponding to the sample identification based on the first fusion information, the second fusion information and the sample label information, and sends the gradient operator to the second computer device; the first computer device and the second computer device based on The gradient operator adjusts the model parameters of the first sub-model and the model parameters of the second sub-model in the machine learning model respectively. While ensuring user privacy, this method improves the training speed of the model, enriches the amount of information of the characteristics of the sample, and improves the accuracy of the model.

Patent CN112217706A discloses a data processing method, device, and equipment. On the one hand, the equipment in the data processing system is connected in a ring structure, and each equipment has two communication links with other equipment. Even if one of the communication links is temporarily interrupted, The device can also communicate with other devices through other communication links, and the data processing system has good stability and robustness. On the other hand, when the data processing system performs data processing, the model parameters determined by each device are sequentially transmitted according to the above-mentioned communication link, and each device fuses the received model parameters with the model parameters determined by itself before transmitting, The amount of data transmitted between devices is small, and there is no need to send model parameters to one device collectively, which can effectively avoid the problems of overload and communication congestion, effectively improve the speed and efficiency of data processing, and ensure the stability of data processing.

Patent CN112330048A discloses a scoring card model training method, device, storage medium and electronic device. The method includes: binning the continuous variables in the wide data table to obtain discrete variables; inputting the variables into a constrained logistic regression model In , convert the logistic regression model into a scorecard model, and calculate the compensation and scale of the scorecard model, where the constraint condition of the logistic regression model is to limit the lower bound of the variable coefficient to be non-negative. Through this invention, since the lower bound of the variable coefficient in the logistic regression model is restricted to be non-negative, it solves the problem that the scorecard model in the related art has a multi-collinear relationship between independent variables when the logistic regression algorithm is used to train the model, which causes the individual variable coefficient to be negative. , which leads to the problem that the model loses its original explanatory power, and thus achieves the effect of avoiding multiple model iterations and reducing the time cost and training overhead of model training.

It can be seen that at present, the knowledge transfer technology based on federated learning on the market still has the following defects:

1. In the existing technology, the application of big data is mainly used for pure management of massive data, for economic forecasting or commercial applications, and there are still few industrial applications or industrial application guidance.

2. On the other hand, there is neither the corresponding technology nor the corresponding technical inspiration for grouping data, so when grouping data, I don’t know how to group it. If it is based on experience, it is obviously unscientific, and unscientific grouping will obviously The model obtained by data training is inaccurate.

3. In the prior art, when training data records, the size and quantity of data are not considered, and when all data are directly trained in order to obtain a model, it is easy to cause the amount of data to be too large, so that on the one hand, the amount of data calculation Large data calculation is difficult; at the same time, the large amount of data can easily lead to inaccurate data training models.

4. In the prior art, in the data records, the abnormal data records that may exist have not been preliminarily cleaned up, and it is easy to generate abnormal data and cause the model obtained by data training to be abnormal.

Faced with the above technical problems, people hope to provide a technical means of federated machine learning service method that can quickly perform data training while reducing the demand on the capabilities of the data processing system, so as to obtain a technical solution for quickly processing data and obtaining a data model. But so far, there is no effective way to solve the above technical problems in the prior art.

Facing the above technical problems, it is hoped to provide a federated machine combination service method and system combined with big data analysis feedback to solve the above technical problems.

Contents of the invention

In view of the above technical problems, the purpose of the present invention is to provide a federated machine combination service method and system combined with big data analysis feedback to solve the problems raised in the above background technology.

To achieve the above object, the present invention provides the following technical solutions:

A federal machine combination service system combined with big data analysis feedback, including multiple devices distributed at different addresses, big data analysis and scheduling modules; each device includes a data preprocessing module, a data acquisition module and a data reading module;

The data training fusion sub-module, the data training fusion sub-module is set on some of the devices; the federated data training module, the federated data training module is set on one of the devices; all devices include a stand-alone storage module, set A local data storage module is set on the device with the data training fusion sub-module, and a global data storage module is set on the device with the federated data training module;

The big data analysis and scheduling module analyzes and schedules all equipment, work modules and data progress data participating in federated machine learning; the big data analysis and scheduling module connects the data reading module and the data training fusion The sub-module is connected to the federated data training module; the data acquisition module is connected to the stand-alone storage module in data communication, and the stand-alone storage module is also connected to the data preprocessing module and the data reading module in data communication respectively;

When the device is running, the data acquisition module acquires the operating data and status data installed on the device, forms a data record, and stores the data record in the stand-alone storage module of the device, and the data The preprocessing module reads the data records stored in the stand-alone storage module, and uses mathematical statistics methods and set requirements to analyze each data record, and when a certain data record is found to be obviously unreasonable, delete the data record;

The big data analysis and scheduling module uses the past operating characteristics of the equipment and the characteristics of the generated data records to formulate grouping rules for the equipment, so that the big data analysis and scheduling module groups all the equipment and divides all The device is divided into several groups according to certain rules, and at least one of the data training fusion sub-modules exists in each group, and the information of the groups is sent to the data reading module, the data training fusion sub-module and the federated data training module, and modify the read permission of the data records of the data reading module, the data training fusion sub-module and the federated data training module;

The data training fusion sub-module establishes a data communication connection with the data reading module corresponding to the group according to the read authority assigned by the big data analysis and scheduling module, so that the data training fusion sub-module passes through the The data reading module reads the data records stored in the stand-alone storage module for data learning and training, obtains the data federation sub-model, and combines the data federation sub-model and the data randomly obtained from the data federation sub-model A certain amount of data records are extracted from the records and sent to the local data storage module;

The federated data training module reads the data federated sub-models and the data records stored in the local data storage module, and adopts a parameter weighted mode for all the federated sub-models to obtain the total data Federated model, and use the read and extracted data records for data training to obtain corresponding parameters, so as to obtain the overall data federated model, and send it to the global data storage module for storage;

The big data analysis and scheduling module arbitrarily extracts a certain number of data records from the stand-alone storage modules on all devices to verify the overall data federation model. During the verification of the data federation model, if the data in the data output and data records meet the model accuracy requirements, then the establishment of the overall data federation model is completed; otherwise, the data stored in the local data storage module Randomly selected data records are randomly selected again, and the process of establishing the overall data federation model is performed by using the federated data training module again.

Preferably, in the process of obtaining the overall data federation model in a parameter-weighted mode for all the federated sub-models, the initial values of the parameters are based on the previous total of the data federation models by the big data analysis and scheduling module. The parameters adopted by the above-mentioned data federation model or the characteristics of the data record volume analyzed by using big data are used as the initial value of the parameters, and on this basis, the read and extracted data records are used for data record training, and the final corresponding parameters are obtained. Generate the overall data federation model to increase the convergence speed of data record training.

Preferably, when each group uses the data training fusion sub-module to perform data training to generate the data federation sub-model, in order to increase the dimension of the training data, for the group, some of the groups adopt longitudinal data federated learning, and the rest of the groups adopt Federated transfer learning; or some groups use vertical federated data learning, some groups use horizontal federated learning, and the rest use federated transfer learning.

Preferably, when the data preprocessing module performs data cleaning of data records, the big data analysis and scheduling module uses existing historical data records or data federation sub-models to perform preliminary data cleaning on the data, and for each data record Carry out analysis, and when the data record deviates to a certain extent, the data record is eliminated to make the data record cleaning more accurate.

Preferably, the big data analysis and scheduling module, when cleaning the data records, combines the unreasonable data records eliminated with the previous equipment operation characteristics to analyze the reasons for the abnormal data records, so as to modify the data records Or add parameters for data logging.

Preferably, when using the big data analysis and scheduling module to group all the devices, use the big data estimation method to estimate the size of the data record volume of each device in advance, and when grouping, record the data The large amount of data is in the same group, and the small amount of data records is a group, so as to prevent the data records with a large amount of data records from flooding the data with a small amount of records during data training, so that the overall data federation model can be constructed accurately sex.

Preferably, when using the big data analysis and scheduling module to group all the devices, use the big data estimation method to estimate the size of the data record volume of each device in advance, and when grouping, the data records Groups with a large number of devices have a small number of devices, while groups with a small amount of data records have a large number of devices, so as to ensure that the number of data records in each group is moderate, so that all the data training fusion The amount of calculation of the submodule is appropriate.

Preferably, for the data training fusion sub-module and the federated data training module on multiple devices in the same group, the big data analysis and scheduling module designates one of the federated data training modules as the service migration In the federated data training module of the system, one of the data training fusion sub-modules in the same group is a data training fusion sub-module to realize the data training of the same group, and one or several data training fusion sub-modules can be executed The data training task is handed over to the big data analysis and scheduling module for execution.

On the other hand, the present application also provides a federated machine combination service method combined with big data analysis feedback, including a federated machine combination service system combined with big data analysis feedback, which is characterized in that it includes the following steps:

Step S1, initialize the federated machine combination service system combined with big data analysis feedback, the big data analysis and scheduling module uses the past operating characteristics of the equipment and the characteristics of the generated data records to formulate the grouping rules of the equipment,

Thus, the big data analysis and scheduling module groups all the devices, and divides all the devices into several groups according to certain rules. Specifically, the big data analysis and scheduling module uses the estimated The method estimates the size of the data recording volume of each of the devices. When using the grouping of all the devices, the data recording volume is large in the same group, and the data recording volume is small into one group, so as to prevent the data recording volume from Large data records submerge the data with a small amount of records during data training, the number of devices for groups with a large amount of data records is small, and the number of devices for groups with a small amount of data records is large, so as to ensure The quantity of the described data record of each grouping is moderate; And guarantee that there is at least one described data training fusion sub-module in each described grouping, and the information of described grouping is sent to described data reading module, described data training the fusion submodule and the federated data training module, and modifying the reading authority of the data records of the data reading module, the data training fusion submodule and the federated data training module;

Step S2, when the device is running, the data acquisition module acquires the running data and status data installed on the device, forms a data record, and stores the data record in the stand-alone storage module of the device;

Step S3, the data preprocessing module reads the data records stored in the stand-alone storage module, and uses mathematical statistics methods and set requirements to analyze each of the data records, and when a certain data record is found When it is obviously unreasonable, delete the data record;

Step S4, the data training fusion sub-module establishes a data communication connection with the data reading module of its corresponding group according to the read authority assigned by the big data analysis and scheduling module, so that the data training fusion sub-module Reading the data records stored in the stand-alone storage module through the data reading module to perform data learning and training to obtain a data federation sub-model;

Step S5, sending the data federation sub-model and a certain amount of data records randomly extracted from the data records used in obtaining the data federation sub-model to the local data storage module;

Step S6, the federated data training module reads the data federated sub-models and the data records stored in the local data storage module, adopts a parameter weighted mode for all the federated sub-models to obtain the total The data federation model, and use the read and extracted data records to perform data training, so as to obtain the overall data federation model;

Step S7, the big data analysis and scheduling module arbitrarily extracts a certain number of data records from the stand-alone storage modules on all devices to verify the overall data federation model, and in the data records When the overall data federation model is used for verification, when the data in the data output and data records meet the model accuracy requirements, the establishment of the overall data federation model is completed.

Step S8, otherwise, the randomly selected data records stored in the local data storage module are randomly selected again, and the federated data training module is used again to establish the overall data federated model.

Preferably, for the data training fusion sub-module and the federated data training module on multiple devices in the same group, the big data analysis and scheduling module designates one of the federated data training modules as the service migration In the federated data training module of the system, one of the data training fusion sub-modules in the same group is a data training fusion sub-module to realize the data training of the same group, and one or several data training fusion sub-modules can be executed The data training task is handed over to the big data analysis and scheduling module for execution.

Compared with prior art, the beneficial effect of the present invention is:

1. The federal machine combination service system combined with big data analysis feedback of the present invention breaks through the traditional global overall data training in order to form a data federation model, and adopts a distributed data federation training method, so that on the one hand, the scale of data samples can be increased, so that The training is more accurate, and the other can reduce the overall cost of data.

2. The federal machine combination service system combined with big data analysis feedback of the present invention uses big data analysis methods to analyze the data records obtained by each device using big data to estimate each device, thereby obtaining the data of each device. Estimate the size of the data, and then send it to the federated machine learning scheduling module to provide data grouping, so that the data grouping is more accurate and effective, and effectively solve the technical problems of grouping.

3. The federal machine combination service system combined with big data analysis feedback of the present invention, when grouping devices for federated training, group the devices with a large amount of data records into the same group, and group the devices with a small amount of data records into one group, so as to prevent data The data records with a large amount of records overwhelm the data with a small amount of records during data training, so as to construct the overall accuracy of the data federation model; at the same time, the number of devices grouped with a large amount of data records is small, and For groups with a small amount of data records, the number of devices is large, so as to ensure that the number of data records in each group is moderate, so that the calculation amount of all the data training and fusion sub-modules is appropriate.

4. The federal machine combination service system combined with big data analysis and feedback of the present invention cleans up the data records to remove abnormal parts of the data records, and at the same time, conducts reasonable analysis on the abnormal parts of the data to find out the cause of the abnormalities.

Description of drawings

Fig. 1 is the overall structure schematic diagram of the present invention;

Fig. 2 is a schematic diagram of the data flow structure of each module provided in the device of the present invention.

In the figure: 1. Equipment; 2. Big data analysis and scheduling module; 3. Data preprocessing module; 4. Data fusion sub-module; 5. Federation data training module; 6. Data sensing module; 7. Stand-alone storage module; 8. Local data storage device; 9. Global data storage module; 10. Grouping; 11. Data reading module.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Specific embodiment one:

A federal machine combination service system combined with big data analysis feedback, including multiple devices 1 distributed at different addresses, big data analysis and scheduling modules 2; each device 1 includes a data preprocessing module 3, a data acquisition module 6 and a data read module 11;

The data training fusion sub-module 4, the data training fusion sub-module 4 is set on some of the devices; the federated data training module 5, the federated data training module 5 is set on one of the devices 1; all the devices 1 Including a stand-alone storage module 7, a local data storage module 8 is set on the device provided with the data training fusion sub-module 4, and a global data storage module 9 is provided on the device 1 provided with the federated data training module 5;

The big data analysis and scheduling module 2 analyzes and schedules all equipment, work modules and data progress data participating in federated machine learning; the big data analysis and scheduling module 2 data communication connects the data reading module 11, the Data training fusion sub-module 4 and the federated data training module 5; the data acquisition module 6 and the stand-alone storage module 7 are connected in data communication, and the stand-alone storage module 7 is also respectively connected with the data preprocessing module 3 and the data Read module 11 data communication connection;

When the device 1 is running, the data acquisition module 6 obtains the running data and status data installed on the device 1 to form a data record, and stores the data record in the stand-alone storage module of the device 1 7. The data preprocessing module 3 reads the data records stored in the stand-alone storage module 7, and uses mathematical statistics methods and set requirements to analyze each of the data records, and when a certain data is found When the record is obviously unreasonable, delete the data record;

The big data analysis and scheduling module 2 utilizes the past operating characteristics of the equipment and the generated data record volume characteristics to formulate grouping rules for the equipment, so that the big data analysis and scheduling module 2 groups all the equipment 1 Divide all the devices 1 into several groups 10 according to certain rules, and ensure that there is at least one data training fusion sub-module 4 in each group 10, and send the group information to the Data reading module 11, the data training fusion sub-module 4 and the federated data training module 5, and modify the data reading module 11, the data training fusion sub-module 4 and the federated data training module 5 read access to said data records;

The data training fusion sub-module 4 establishes a data communication connection with the data reading module 11 of its corresponding group according to the read authority assigned by the big data analysis and scheduling module 2, so that the data training fusion sub-module 4 Read the data records stored in the stand-alone storage module 7 through the data reading module 11 to perform data learning and training to obtain a data federation sub-model, and combine the data federation sub-model and the data federation obtained randomly A certain amount of data records are extracted from the data records applied in the sub-model and sent to the local data storage module 8;

The federated data training module 5 reads the data federated sub-models and the data records stored in the local data storage module 8, and adopts a parameter weighted mode for all the federated sub-models to obtain the total The data federation model is described, and the extracted data records are used for data training to obtain corresponding parameters, so as to obtain the overall data federation model, and send to the global data storage module 9 for storage;

The big data analysis and scheduling module 2 arbitrarily extracts a certain number of data records from the stand-alone storage modules 7 on all devices 1 to verify the overall data federation model. In the data records When the overall data federation model is used for verification, when the data in the data output and data records meet the model accuracy requirements, the establishment of the overall data federation model is completed; otherwise, in the local data storage module The randomly extracted data records stored in step 8 are re-selected randomly, and the federated data training module 5 is used again to establish the overall data federated model.

Preferably, in the process of obtaining the overall data federation model in a parameter-weighted mode for all the federated sub-models, the initial value of the parameters is based on the previous overall data of the big data analysis and scheduling module 1 The parameters used by the data federation model or the characteristics of the data record volume analyzed by big data are used as the initial value of the parameters, and on this basis, the extracted data records are used for data record training to obtain the final corresponding parameters, so that The overall data federation model is obtained to increase the convergence speed of data recording training.

Preferably, when each group uses the data training fusion sub-module 4 to perform data training to generate the data federation sub-model, in order to increase the dimension of the training data, for the group, some of the groups adopt longitudinal data federated learning, and the rest of the groups are grouped Adopt federated transfer learning; or use vertical federated data learning for some groups, horizontal federated learning for some groups, and federated transfer learning for the rest.

Preferably, when the data preprocessing module 3 performs data cleaning of data records, the big data analysis and scheduling module 2 uses existing historical data records or data federation sub-models to perform preliminary data cleaning on the data, and for each The data records are analyzed, and when the data records deviate to a certain extent, the data records are eliminated to make the data record cleaning more accurate.

Preferably, the big data analysis and scheduling module 2, when cleaning up data records, combines the unreasonable data records eliminated with the previous equipment operation characteristics to analyze the reasons for abnormal data records, so that the data records can be processed. Modify or add parameters of data records.

Preferably, when using the big data analysis and scheduling module 2 to group all the devices 1, use the big data estimation method to estimate the size of the data records of each device 1 in advance, and when grouping, Group the data with a large amount of records into the same group, and group the data with a small amount of data records into one group, so as to prevent the data records with a large amount of data records from submerging the data with a small amount of records during data training, so as to construct the total data Federated Model Accuracy.

Preferably, when using the big data analysis and scheduling module 2 to group all the devices 1, use the big data estimation method to estimate the size of the data records of each device 1 in advance, and when grouping, For groups with a large amount of data records, the number of devices 1 is small, and for groups with a small amount of data records, the number of devices 1 is large, so as to ensure that the number of data records in each group is moderate, so that all The calculation amount of the data training fusion sub-module 4 is appropriate.

Preferably, for the data training fusion sub-module 4 and the federated data training module 5 on multiple devices in the same group, the big data analysis and scheduling module 2 specifies one of the federated data training modules 5 For the federated data training module 5 of the service migration system, one of the data training fusion sub-modules 4 of the same group is a data training fusion sub-module to realize the data training of the same group, and one or several of them can be The data training task performed by the data training fusion sub-module 4 is handed over to the big data analysis and scheduling module 2 for execution.

Specific embodiment two:

A federated machine combination service method combined with big data analysis feedback, including a federated machine combination service system combined with big data analysis feedback, including the following steps:

Step S1, initialize the federated machine combination service system combined with big data analysis feedback, the big data analysis and scheduling module 2 uses the past operating characteristics of the equipment and the characteristics of the generated data records to formulate the grouping rules of the equipment ,

Thus, the big data analysis and scheduling module 2 groups all the devices 1, and divides all the devices 1 into several groups 10 according to certain rules. Specifically, the big data analysis and scheduling module 2 pre-uses The big data estimation method estimates the size of the data record volume of each of the devices 1 acquired. When grouping all the devices 1 by using Grouping, in order to prevent the data records with a large amount of data records from flooding the data with a small amount of records during data training, the number of devices 1 in the group with a large amount of data records is small, and the grouping with a small amount of data records, so The number of said devices 1 is large, so as to ensure that the number of said data records of each group is moderate; and ensure that there is at least one said data training fusion sub-module 4 in each said group 10, and send the information of said group Give the data reading module 11, the data training fusion sub-module 4 and the federated data training module 5, and modify the data reading module 11, the data training fusion sub-module 4 and the federated data training Read access to said data records of module 5;

Step S2, when the device 1 is running, the data acquisition module 6 obtains the running data and status data installed on the device 1, forms a data record, and stores the data record in the Stand-alone storage module 7;

Step S3, the data preprocessing module 3 reads the data records stored in the stand-alone storage module 7, and uses mathematical statistics methods and set requirements to analyze each of the data records, and when a certain data record is found When the data record is obviously unreasonable, delete the data record;

Step S4, the data training fusion sub-module 4 establishes a data communication connection with the data reading module 11 of its corresponding group according to the reading authority assigned by the big data analysis and scheduling module 2, so that the data training The fusion sub-module 4 reads the data records stored in the stand-alone storage module 7 through the data reading module 11 and carries out data learning and training to obtain the data federation sub-model;

Step S5, sending the data federation sub-model and a certain amount of data records randomly extracted from the data records used in obtaining the data federation sub-model to the local data storage module 8;

Step S6, the federated data training module 5 reads the data federated sub-models and the data records stored in the local data storage module 8, and adopts a parameter weighted mode for all the federated sub-models to obtain Summarize the data federation model, and use the read and extracted data records to perform data training, so as to obtain the overall data federation model;

Step S7, the big data analysis and scheduling module 2 arbitrarily extracts a certain number of data records from the stand-alone storage modules 7 on all devices 1, and uses them to verify the overall data federation model. When the data records are checked using the overall data federation model, when the data in the data output and data records meet the model accuracy requirements, the overall data federation model is established.

Step S8, otherwise, the randomly selected data records stored in the local data storage module 8 are randomly selected again, and the federated data training module 5 is used again to establish the overall data federated model.

Preferably, for the data training fusion sub-module 4 and the federated data training module 5 on multiple devices in the same group, the big data analysis and scheduling module 2 specifies one of the federated data training modules 5 For the federated data training module 5 of the service migration system, one of the data training fusion sub-modules 4 of the same group is a data training fusion sub-module to realize the data training of the same group, and one or several of them can be The data training task performed by the data training fusion sub-module 4 is handed over to the big data analysis and scheduling module 2 for execution.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications and substitutions can be made to these embodiments without departing from the principle and spirit of the present invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims (8)

1. A federal machine combination service system combined with big data analysis feedback, including multiple devices (1) distributed at different addresses, big data analysis and scheduling modules (2); each device (1) includes a data preprocessing module (3), data acquisition module (6) and data reading module (11); The data training fusion sub-module (4), the data training fusion sub-module (4) is set on some of the devices; the federated data training module (5), the federated data training module (5) is set in one of the On the device (1); all devices (1) include a stand-alone storage module (7), and a local data storage module (8) is set on the device provided with the data training fusion sub-module (4), provided with the federation The device (1) of the data training module (5) is provided with a global data storage module (9); The big data analysis and scheduling module (2) analyzes and schedules all equipment, work modules and data progress data participating in federated machine learning; the big data analysis and scheduling module (2) data communication connects the data reading module (11), the data training fusion sub-module (4) and the federated data training module (5); the data acquisition module (6) is connected to the stand-alone storage module (7) data communication, and the stand-alone storage The module (7) is also connected with the data communication of the data preprocessing module (3) and the data reading module (11) respectively; It is characterized by: When the device (1) is running, the data acquisition module (6) obtains the running data and status data installed on the device (1), forms a data record, and stores the data record in the device ( 1) the stand-alone storage module (7), the data preprocessing module (3) reads the data records stored in the stand-alone storage module (7), and uses mathematical statistics methods and set requirements To analyze each of the above data records, and delete the data record when it is found that a certain data record is obviously unreasonable; The big data analysis and scheduling module (2) utilizes the past operating characteristics of the equipment and the characteristics of the amount of data records generated to formulate grouping rules for the equipment, so that the big data analysis and scheduling module (2) performs all The device (1) is grouped, and all the devices (1) are divided into several groups (10) according to certain rules, and it is guaranteed that there is at least one of the data training fusion submodules in each group (10) ( 4), and send the grouped information to the data reading module (11), the data training fusion sub-module (4) and the federated data training module (5), and modify the data reading module (11), the data training fusion sub-module (4) and the read authority of the data records of the federated data training module (5); The data training fusion sub-module (4) establishes a data communication connection with the data reading module (11) of its corresponding grouping according to the reading authority assigned by the big data analysis and scheduling module (2), so that the The data training fusion sub-module (4) reads the data records stored in the stand-alone storage module (7) through the data reading module (11) and performs data learning and training to obtain a data federation sub-model, and The data federation sub-model and a certain amount of data records randomly extracted from the data records applied in the data federation sub-model are sent to the local data storage module (8); The federated data training module (5) reads the data federated sub-model and the data records stored in the local data storage module (8), and adopts a parameter weighted mode for all the federated sub-models to obtain Go out the total data federation model, and utilize the extracted data record of reading to carry out data training, obtain corresponding parameter, thereby draw the total data federation model, and send to described global data storage module (9) to store; The big data analysis and scheduling module (2) arbitrarily extracts a certain number of data records from the stand-alone storage modules (7) on all devices (1), and is used to verify the overall data federation model. When the data records are checked using the general data federation model, when the data in the data output and data records meet the model accuracy requirements, the establishment of the general data federation model is completed; otherwise, the local data storage module The randomly extracted data records stored in (8) are randomly extracted again, and the process of establishing a total data federation model is carried out by using the federated data training module (5) again; Wherein, the manner of dividing all the devices (1) into several groups (10) according to certain rules is: When using the big data analysis and scheduling module (2) to group all the devices (1), use a big data estimation method in advance to estimate the size of the data records acquired by each of the devices (1), and When grouping, the data records with a large amount of data are grouped together, and the data records with a small amount of data are grouped into one group, so as to prevent the data records with a large amount of data records from submerging the data with a small amount of records during data training; When the big data analysis and scheduling module (2) groups all the devices (1), it uses the estimation method of big data to estimate the size of the data records of each device (1), and when grouping, the A group with a large amount of data recording has a small number of devices ( 1 ), while a group with a small amount of data recording has a large number of devices ( 1 ). 2. A federated machine combination service system combined with big data analysis feedback according to claim 1, characterized in that: in the process of obtaining the overall data federated model by adopting a parameter weighted mode for all the federated sub-models, The initial value of the parameter adopts the parameters adopted by the big data analysis and scheduling module (2) according to the previous total data federation model or utilizes the characteristics of the data record volume analyzed by the big data as the initial value of the parameter, on this basis Use the read and extracted data records to perform data record training, and obtain the final corresponding parameters, so as to obtain the overall data federation model, so as to increase the convergence speed of data record training. 3. A kind of federal machine combination service system combined with big data analysis feedback according to claim 1, characterized in that: use the data training fusion sub-module (4) in each group to carry out data training to generate the data federation sub-module For the model, in order to increase the dimension of the training data, some of the groups adopt longitudinal data federated learning, and the rest of the groups adopt federated transfer learning; or some of the groups adopt vertical federated data learning, some groups adopt horizontal federated learning, and the rest adopt Federated Transfer Learning. 4. A kind of federal machine combination service system combined with big data analysis feedback according to claim 1, characterized in that: when the data preprocessing module (3) performs data cleaning of data records, the big data analysis and scheduling module (2) use the existing historical data records or data federation sub-model to perform preliminary data cleaning on the data, analyze each data record, and remove the data record when the data record deviates to a certain extent, so that the data Record cleaning is more accurate. 5. A federated machine combination service system combined with big data analysis feedback according to claim 4, characterized in that: said big data analysis and scheduling module (2), when cleaning up data records, removes unidentified Reasonable data records are combined with the past equipment operation characteristics to analyze the reasons for abnormal data records, so as to modify the data records or add data record parameters. 6. A kind of federated machine combination service system combined with big data analysis feedback according to claim 1, characterized in that: the data training fusion sub-module (4), the The federated data training module (5), the big data analysis and scheduling module (2) designates one of the federated data training modules (5) as the federated data training module (5) of the federated machine combination service system, One of the data training fusion submodules (4) of the same group is a data training fusion submodule, which realizes the data training of the same group, and the data training performed by one or several data training fusion submodules (4) can be carried out. The task is handed over to the big data analysis and scheduling module (2) for execution. 7. A federated machine combination service method combined with big data analysis feedback, comprising the federated machine combination service system combined with big data analysis feedback as described in any one of claims 1-6, characterized in that it comprises the following steps: Step S1, initialize the federated machine combination service system combined with big data analysis feedback, the big data analysis and scheduling module (2) utilizes the past operating characteristics of the equipment and the characteristics of the generated data records to formulate the grouping rules, Thus, the big data analysis and scheduling module (2) groups all the devices (1), divides all the devices (1) into several groups (10) according to certain rules, specifically, the big The data analysis and scheduling module (2) uses the estimation method of big data to estimate and acquire the size of the data recording volume of each said equipment (1) in advance, and when all equipments (1) are grouped, the data recording volume The large ones are in the same group, and the ones with a small amount of data records are grouped into one group, so as to prevent the data records with a large amount of data records from flooding the data with a small amount of records during data training, and the devices for groups with a large amount of data records The quantity of (1) is few, and the grouping of data record amount is little, the quantity of described equipment (1) is many, so that guarantee the quantity of the described data record of each grouping; And guarantee that in each described grouping (10) There is at least one said data training fusion sub-module (4), and the information of said grouping is sent to said data reading module (11), said data training fusion sub-module (4) and said federated data training module (5), and modify the reading authority of the data records of the data reading module (11), the data training fusion sub-module (4) and the federated data training module (5); Step S2, when the device (1) is running, the data acquisition module (6) obtains the running data and status data installed on the device (1), forms a data record, and stores the data record in the The stand-alone storage module (7) of the device (1); Step S3, the data preprocessing module (3) reads the data records stored in the stand-alone storage module (7), and uses mathematical statistics methods and set requirements to analyze each of the data records, When a data record is found to be obviously unreasonable, delete the data record; Step S4, the data training fusion sub-module (4) establishes a data communication connection with the data reading module (11) of its corresponding group according to the read permission assigned by the big data analysis and scheduling module (2) , so that the data training fusion sub-module (4) reads and stores the data records stored in the stand-alone storage module (7) through the data reading module (11) to carry out data learning and training, and obtains the data federation sub-model ; Step S5, sending the data federation sub-model and a certain amount of data records randomly extracted from the data records used in obtaining the data federation sub-model to the local data storage module (8); Step S6, the federated data training module (5) reads the data federated sub-models and the data records stored in the local data storage module (8), and adopts parameter weighting for all the federated sub-models The overall data federation model is obtained from the model, and the data training is performed using the read and extracted data records, so as to obtain the overall data federation model; Step S7, the big data analysis and scheduling module (2) arbitrarily extracts a certain number of data records from the stand-alone storage modules (7) on all devices (1) for calibrating the overall data federation model In the verification of the data records using the general data federation model, when the data in the data output and data records meet the model accuracy requirements, the establishment of the general data federation model is completed. Step S8, otherwise, the randomly extracted data records stored in the local data storage module (8) are rerandomly extracted, and the federated data training module (5) is used again to establish a general data federated model. 8. A kind of federal machine combination service method combined with big data analysis feedback according to claim 7, characterized in that: the data training fusion sub-module (4), the The federated data training module (5), the big data analysis and scheduling module (2) designates one of the federated data training modules (5) as the federated data training module (5) of the federated machine combination service system, One of the data training fusion submodules (4) of the same group is a data training fusion submodule, which realizes the data training of the same group, and the data training performed by one or several data training fusion submodules (4) can be carried out. The task is handed over to the big data analysis and scheduling module (2) for execution.

Images