CN114707606A - Data processing method and device based on federal learning, equipment and storage medium - Google Patents

Data processing method and device based on federal learning, equipment and storage medium Download PDF

Info

Publication number
CN114707606A
CN114707606A CN202210377079.4A CN202210377079A CN114707606A CN 114707606 A CN114707606 A CN 114707606A CN 202210377079 A CN202210377079 A CN 202210377079A CN 114707606 A CN114707606 A CN 114707606A
Authority
CN
China
Prior art keywords
training sample
fault detection
training
importance
ciphertext
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210377079.4A
Other languages
Chinese (zh)
Other versions
CN114707606B (en
Inventor
宋雅奇
丁鹏
沈云
时晓厚
郭璐
刘晨
曹振强
王向明
刘心
张丽伟
郭琦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Telecom Corp Ltd
Original Assignee
China Telecom Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Telecom Corp Ltd filed Critical China Telecom Corp Ltd
Priority to CN202210377079.4A priority Critical patent/CN114707606B/en
Publication of CN114707606A publication Critical patent/CN114707606A/en
Application granted granted Critical
Publication of CN114707606B publication Critical patent/CN114707606B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/602—Providing cryptographic facilities or services
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Medical Informatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the application discloses a data processing method and device based on federal learning, electronic equipment, storage medium and program product, wherein the method comprises the following steps: each edge node in the plurality of edge nodes can obtain a training sample, the importance of the training sample is calculated, if the importance of the training sample is larger than a preset importance threshold, the training sample is encrypted to obtain a ciphertext of the training sample, and the ciphertext of the training sample is sent to a cloud server; the cloud server trains the fault detection model based on the ciphertext of the training sample sent by each edge node, and sends the trained fault detection model to the edge nodes respectively; and each edge node carries out fault detection based on the received fault detection model. The technical scheme of the embodiment of the application can improve the fault detection precision.

Description

Data processing method and device based on federal learning, equipment and storage medium
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an electronic device, a storage medium, and a program product based on federal learning.
Background
In industrial production, it is generally necessary to detect failures of produced products or semi-finished products, for example, in the semiconductor industry, after soldering is completed on an electronic component, the electronic component needs to be detected to determine whether failures such as offset of a solder fillet, missing of a solder fillet, short circuit caused by solder connection of adjacent solder fillets, irregular bonding pad, overlong solder fillet, and the like occur. However, at present, the fault detection is usually performed on the product manually or according to data collected by corresponding sensors and corresponding threshold values, and therefore the fault detection efficiency and the precision are low.
Disclosure of Invention
In order to solve the above technical problem, embodiments of the present application provide a data processing method and apparatus, an electronic device, a storage medium, and a program product based on federal learning, which can improve failure detection efficiency and accuracy to at least some extent.
Other features and advantages of the present application will be apparent from the following detailed description, or may be learned by practice of the application.
According to an aspect of an embodiment of the present application, there is provided a data processing method based on federated learning, applied to a federated learning system including a cloud server and a plurality of edge nodes, where the method is performed by the plurality of edge nodes respectively, and the method includes:
acquiring a training sample, and calculating the importance of the training sample;
if the importance of the training sample is larger than a preset importance threshold, encrypting the training sample to obtain a ciphertext of the training sample;
sending the ciphertext of the training sample to the cloud server so that the cloud server trains a fault detection model according to the ciphertext of the training sample;
and receiving the trained fault detection model sent by the cloud server, and performing fault detection based on the received fault detection model.
According to an aspect of the embodiments of the present application, there is provided a data processing method based on federated learning, applied to a federated learning system including a cloud server and a plurality of edge nodes, the method being performed by the cloud server, the method including:
receiving a ciphertext of a training sample sent by each edge node, and training a fault detection model based on the ciphertext of the training sample sent by each edge node; the ciphertext of the training sample sent by each edge node is obtained by encrypting the training sample by each edge node under the condition that the importance degree of the training sample is determined to be greater than a preset importance degree threshold;
and respectively sending the trained fault detection models to the edge nodes so as to enable the edge nodes to carry out fault detection based on the received fault detection models.
According to an aspect of an embodiment of the present application, there is provided a data processing apparatus based on federated learning, where the data processing apparatus is applied to a federated learning system that includes a cloud server and a plurality of edge nodes, and the apparatus is configured at the plurality of edge nodes respectively, and the apparatus includes:
the importance evaluation module is configured to obtain a training sample and calculate the importance of the training sample;
the data encapsulation module is configured to encrypt the training sample to obtain a ciphertext of the training sample if the importance of the training sample is greater than a preset importance threshold;
the edge joint training module is configured to send the ciphertext of the training sample to the cloud server so that the cloud server trains a fault detection model according to the ciphertext of the training sample;
and the fault detection module is configured to receive the trained fault detection model sent by the cloud server, so as to perform fault detection based on the received fault detection model.
According to an aspect of an embodiment of the present application, there is provided a data processing apparatus based on federated learning, applied to a federated learning system including a cloud server and a plurality of edge nodes, where the apparatus is configured in the cloud server, the apparatus including:
the model aggregation module is configured to receive the ciphertext of the training sample sent by each edge node, and train the fault detection model based on the ciphertext of the training sample sent by each edge node; the ciphertext of the training sample sent by each edge node is obtained by encrypting the training sample by each edge node under the condition that the importance degree of the training sample is determined to be greater than a preset importance degree threshold;
and the model updating module is configured to send the trained fault detection models to the edge nodes respectively so that the edge nodes perform fault detection based on the received fault detection models.
According to an aspect of an embodiment of the present application, there is provided an electronic device including:
one or more processors;
a storage device for storing one or more computer programs that, when executed by the one or more processors, cause the electronic equipment to implement the federated learning-based data processing method as described above.
According to an aspect of embodiments of the present application, there is provided a computer-readable storage medium having stored thereon a computer program, which, when executed by a processor of an electronic device, causes the electronic device to execute the federal learning based data processing method as described above.
According to an aspect of an embodiment of the present application, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the federated learning-based data processing method as described above.
In the technical scheme provided by the embodiment of the application, each edge node in a plurality of edge nodes can obtain a training sample, the importance of the training sample is calculated, if the importance of the training sample is greater than a preset importance threshold, the training sample is encrypted to obtain a ciphertext of the training sample, and the ciphertext of the training sample is sent to a cloud server; the cloud server trains the fault detection model based on the ciphertext of the training sample sent by each edge node, and sends the trained fault detection model to the edge nodes respectively; and each edge node carries out fault detection based on the received fault detection model. Therefore, on one hand, the cloud server can train the fault detection model according to the ciphertext of the training sample sent by the edge nodes, so that the source of the training sample is increased, the data barrier among different devices is broken through, the generalization capability of the fault detection model is improved, the fault detection precision is improved, the data safety is guaranteed, and meanwhile, when the importance of the training sample is greater than the preset importance threshold, the ciphertext of the training sample is sent to the cloud server, so that the cloud server trains the fault detection model based on the ciphertext of the training sample, the computing resource is saved, and the convergence speed of the model is improved; on the other hand, the edge nodes are used for completing the collection of the training samples, and the cloud server is used for completing the training of the fault detection model, so that the advantages of the edge nodes and the cloud server are reasonably utilized, and the acquisition speed of the training samples and the training efficiency of the fault detection model are improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application. It is obvious that the drawings in the following description are only some embodiments of the application, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:
FIG. 1 is a schematic illustration of an implementation environment to which the present application relates;
FIG. 2 is a flow chart illustrating a federated learning-based data processing method according to an exemplary embodiment of the present application;
FIG. 3 is a flowchart of step S210 shown in FIG. 2 in an exemplary embodiment;
FIG. 4 is a flowchart of step S220 shown in FIG. 2 in an exemplary embodiment;
FIG. 5 is a flow chart illustrating a federated learning-based data processing method according to an exemplary embodiment of the present application;
FIG. 6 is a flow chart of a federated learning-based data processing method that is illustrated in another exemplary embodiment of the present application;
FIG. 7 is a block diagram of a federated learning-based data processing apparatus that is illustrated in an exemplary embodiment of the present application;
FIG. 8 is a block diagram of a federated learning-based data processing apparatus that is illustrated in an exemplary embodiment of the present application;
FIG. 9 is a schematic diagram of a federated learning system as shown in an exemplary embodiment of the present application;
FIG. 10 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
It should also be noted that: reference to "a plurality" in this application means two or more. "and/or" describe the association relationship of the associated objects, meaning that there may be three relationships, e.g., A and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
In industrial production, it is generally necessary to detect failures of produced products or semi-finished products, for example, in the semiconductor industry, after soldering is completed on an electronic component, the electronic component needs to be detected to determine whether failures such as offset of a solder fillet, missing of a solder fillet, short circuit caused by solder connection of adjacent solder fillets, irregular bonding pad, overlong solder fillet, and the like occur. However, at present, the fault detection is usually performed on the product manually or according to data collected by corresponding sensors and corresponding threshold values, and therefore the fault detection efficiency and the precision are low. Based on this, embodiments of the present application provide a data processing method and apparatus, an electronic device, a storage medium, and a program product based on federal learning, so that data processing accuracy based on federal learning can be improved.
Referring to fig. 1, fig. 1 is a schematic illustration of an implementation environment to which the present application relates. The implementation environment is a federated learning system, including a cloud server 100 and a plurality of edge nodes 200. Wherein, the cloud server 100 and each edge node 200 communicate with each other through a wired or wireless network.
The cloud server 100 may provide basic cloud computing services such as cloud services, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, middleware services, a domain name service, a security service, a CDN (Content Delivery Network), and a big data and artificial intelligence platform.
The edge node 200 is a service platform constructed on the network edge side close to the user, and is used for providing resources such as storage, calculation, network and the like, so that part of key service applications are sunk to the edge of the access network, and the width and delay loss caused by network transmission and multistage forwarding are reduced.
The cloud server 100 and the edge node 200 collectively maintain a fault detection model, wherein the edge node 200 is a provider of data and a user of the fault detection model, which may correspond to different product manufacturers. The cloud server 100 is a training party of the fault detection model, and can obtain ciphertexts of training samples sent by different edge nodes 200, and train the fault detection model based on the ciphertexts of the training samples, so that data barriers among different product manufacturers are broken, and generalization capability and accuracy of the fault detection model are improved.
In an exemplary embodiment, each edge node 200 acquires a training sample, calculates the importance of the training sample, encrypts the training sample if the importance of the training sample is greater than a preset importance threshold, obtains a ciphertext of the training sample, and sends the ciphertext of the training sample to the cloud server 100; the cloud server 100 receives ciphertexts of training samples sent by the edge nodes, trains the fault detection model based on the ciphertexts of the training samples sent by the edge nodes 200, and sends the trained fault detection model to the edge nodes 200 respectively; each edge node 200 performs fault detection based on the received fault detection model. On one hand, the cloud server can train the fault detection model according to the ciphertext of the training sample sent by the edge nodes, so that the source of the training sample is increased, the data barrier among different devices is broken through, the generalization capability of the fault detection model is improved, the precision of fault detection is improved, the data safety is guaranteed, and meanwhile, when the importance of the training sample is greater than a preset importance threshold, the ciphertext of the training sample is sent to the cloud server, so that the cloud server trains the fault detection model based on the ciphertext of the training sample, the computing resource is saved, and the convergence speed of the model is improved; on the other hand, the edge nodes are used for completing the collection of the training samples, and the cloud server is used for completing the training of the fault detection model, so that the advantages of the edge nodes and the cloud server are reasonably utilized, and the acquisition speed of the training samples and the training efficiency of the fault detection model are improved.
It should be noted that the number of edge nodes 200 in fig. 1 is merely illustrative, and there may be any number of edge nodes 200 according to actual needs.
Referring to fig. 2, fig. 2 is a flowchart illustrating a federated learning-based data processing method that may be applied to the implementation environment shown in fig. 1 and executed by each edge node 200 in the implementation environment shown in fig. 1 according to an exemplary embodiment of the present application.
As shown in fig. 2, in an exemplary embodiment, the data processing method based on federal learning may include steps S210 to S240, which are described in detail as follows:
step S210, the edge node obtains a training sample, and calculates the importance of the training sample.
It should be noted that, in the first place, it should be noted that the edge node is a party in the federal learning system for providing training data (i.e., training samples).
Secondly, the training samples are used for training the fault detection model so as to optimize the model parameters. The training sample comprises the associated data of the product to be detected and the label determined based on the associated data of the product to be detected. In an optional example, considering that the appearance of a faulty product is different from that of a qualified product, for example, it can be seen from the appearance of the product whether fillet weld offset occurs in the product, therefore, the associated data of the product to be detected can be a picture of the product to be detected, wherein the picture of the product to be detected can be obtained by shooting the product to be detected through image acquisition equipment, and the image acquisition equipment includes but is not limited to a camera; or, in another optional example, the associated data of the product to be detected may further include the weight of the product to be detected, and the like. The label is used for representing whether the product to be detected breaks down or not, and can also represent the type of the product breaking down.
In addition, the importance of the training samples is used to characterize the importance of the training samples to model optimization.
In order to save computing resources and improve optimization efficiency of the model, in this embodiment, the edge node may first obtain a training sample, and then calculate importance of the training sample.
The specific mode of the edge node for acquiring the training sample can be flexibly set according to actual needs.
In an optional implementation manner, if the associated data of the product to be detected contained in the training sample is a picture, the edge node may obtain multiple pictures of the product to be detected from the image acquisition device, identify the multiple pictures to screen a first target picture representing that the product to be detected has a fault from the multiple pictures, obtain an audit result of the first target picture, determine a label of the first target picture according to the audit result, and use the first target picture and the label of the first target picture as the training sample. That is, the image acquisition device can acquire the pictures of the product to be detected to obtain a plurality of pictures, and upload the plurality of pictures to the edge node; after the edge node receives the multiple pictures, the multiple pictures can be identified, if a certain picture represents that a product to be detected breaks down, the picture is taken as a first target picture, a manual auditing result of the identification results of the first target picture and the first target picture is obtained, the tags of the first target picture and the first target picture are determined according to the manual auditing result, and the tags of the first target picture and the first target picture are selected.
Optionally, the image acquisition device may continuously acquire the picture of the product to be detected to obtain a video stream, and send the video stream to the edge gateway; after receiving the video stream, the edge gateway performs deframing on the video stream to obtain a plurality of pictures, and transmits the plurality of pictures to the edge node. The edge node can identify the multiple pictures based on the fault detection model stored at the local terminal so as to screen out the first target picture.
In another optional implementation manner, if the associated data of the product to be detected contained in the training sample is a picture, the edge node may obtain multiple pictures of the product to be detected from the image acquisition device, identify the multiple pictures to screen a second target picture representing that the product to be detected does not have a fault from the multiple pictures, obtain an audit result of the second target picture, determine a label of the second target picture according to the audit result, and use the second target picture and the label of the second target picture as the training sample. The determining of the second target picture is similar to the determining of the first target picture, and details are not repeated here. In another optional embodiment, the edge node may further use the first target picture, the label of the first target picture, the second target picture, and the label of the second target picture as training samples.
The specific mode of calculating the importance of the training sample by the edge node can also be flexibly set according to actual needs.
In an optional embodiment, in order to further improve the security, after the edge node acquires the training sample, the edge node may detect the training sample to determine whether the training sample contains sensitive data, and if not, calculate the importance of the training sample; and if so, filtering the sensitive data in the training sample, and calculating the importance of the filtered training sample. For example, in an optional implementation, if the training sample includes data of a specified type, it may be determined that the training sample includes the sensitive data.
And S220, if the importance of the training sample is greater than a preset importance threshold, encrypting the training sample by the edge node to obtain the ciphertext of the training sample.
The preset importance threshold is a judgment basis for judging whether the training sample is encrypted and sending the ciphertext of the training sample to the cloud server, and the specific value can be flexibly set according to actual needs.
In this embodiment, after calculating the importance of the training sample, the edge node may determine whether the importance of the training sample is greater than a preset importance threshold, and if so, encrypt the training sample to obtain a ciphertext of the training sample.
The specific encryption mode can be flexibly set according to actual needs. For example. Encryption methods include, but are not limited to: homomorphic encryption, ciphertext sharing, differential privacy, and the like. The homomorphic encryption is a cryptography technology based on a mathematical problem amount computer complexity theory, data which is subjected to homomorphic encryption is processed to obtain an output, the output is decrypted, and the result of the output is the same as the output result obtained by processing unencrypted original data by the same method; differential privacy is a means in cryptography that aims to provide a way to maximize the accuracy of data queries while minimizing the chances of identifying their records when querying from a statistical database.
In step S230, the edge node sends the ciphertext of the training sample to the cloud server, so that the cloud server trains the fault detection model according to the ciphertext of the training sample.
The fault detection model is a machine learning model and is used for carrying out fault detection based on data to be identified. The method can be applied to different scenes, for example, can be applied to detecting faults of the electronic element, for example, detecting whether faults of short circuit, irregular bonding pad, overlong welding leg and the like caused by welding leg offset, welding leg missing, and adjacent welding leg soldering tin connection of the element occur or not.
After the edge node encrypts the training sample to obtain the ciphertext of the training sample, the ciphertext of the training sample is transmitted to the cloud server. The cloud server trains the fault detection model based on the ciphertext of the training samples received from different edge nodes to optimize the fault detection model.
In step S240, the edge node receives the trained fault detection model sent by the cloud server, so as to perform fault detection based on the received fault detection model.
After the cloud server trains the fault detection model based on the ciphertext of the training sample, the trained fault detection model can be respectively sent to the edge nodes. After each edge node receives the fault detection model, the received fault detection model can be stored, and when fault detection is needed, data to be identified is input into the locally stored fault detection model to obtain a detection result.
In this embodiment, on one hand, the cloud server may train the fault detection model according to the ciphertexts of the training samples sent by the plurality of edge nodes, so that the sources of the training samples are increased, data barriers between different devices are broken, the generalization capability of the fault detection model is improved, the precision of fault detection is improved, and data safety is ensured; on the other hand, the edge nodes are used for completing the collection and fault detection of the training samples, and the cloud server is used for completing the training of the fault detection model, so that the advantages of the edge nodes and the cloud server are reasonably utilized, the resources of the edge nodes and the cloud server are reasonably utilized, and the acquisition speed of the training samples and the training efficiency of the fault detection model are improved.
In an exemplary embodiment, referring to FIG. 3, FIG. 3 is a flow chart of step S210 in the embodiment of FIG. 2 in an exemplary embodiment. As shown in fig. 3, the process of calculating the importance of the training samples may include steps S211 to S212, which are described in detail as follows:
step S211, the edge node acquires the associated parameters of the training sample; the associated parameters of the training samples comprise at least one of the number of the training samples, the data dimension of the training samples and the number of the features contained in the training samples.
The correlation parameters of the training samples refer to parameters related to the training samples, and considering that the number of the training samples, the data dimensions, the number of included features, and the like may affect the accuracy of the model, therefore, the correlation parameters of the training samples may include, but are not limited to, at least one of the number of the training samples, the data dimensions of the training samples, the number of included features of the training samples, and the like. It should be noted that the number of the training samples is used to represent the number of the training samples, that is, the number of the training samples obtained by the edge node; the data dimension of the training sample is used for representing the dimension number of the training sample, and the data dimension is an important basic concept for forming a specific relation among data and expressing multiple data meanings; when fault detection is performed, data to be recognized needs to be processed to extract features from the data to be recognized, and the number of features of the training samples represents the number of features extracted from the training samples, for example, if the type of the training samples is pictures, the number of features extracted from the pictures is small for blank pictures and pictures with low definition, and the number of features extracted from the pictures containing complete products to be detected is large.
In this embodiment, the edge node may obtain the correlation parameter of the training sample, so as to calculate the importance of the training sample based on the correlation parameter of the training sample.
In step S212, the edge node determines the importance of the training sample according to the associated parameters of the training sample.
After the edge node obtains the correlation parameters of the training samples, the importance of the training samples can be determined according to the obtained correlation parameters of the training samples. Considering that the training samples are too few, the improvement on the model performance is small, and therefore, the number of the training samples and the importance of the training samples can be in a positive correlation relationship. The more the data dimensionality of the training sample is, the more dimensionality the model can perform data processing from the more dimensionalities, so that the data detection precision is performed, the model performance is improved, and therefore the data dimensionality of the training sample and the importance of the training sample can be in a positive correlation relation. The more the training samples contain the more the number of features, the higher the accuracy of the model for data analysis, and therefore, the number of features contained in the training samples can be in positive correlation with the importance of the training samples.
In an optional embodiment, the process that the edge node determines the importance of the training sample according to the associated parameter of the training sample may include: and determining the importance of the training samples according to the associated parameters of the training samples and the weights corresponding to different associated parameters. That is, corresponding weights are set for different associated parameters of the training samples, and the importance of the training samples can be calculated based on the associated parameters of the training samples and the weights corresponding to the different associated parameters.
In an alternative embodiment, considering that the performance of the edge node itself may also affect the training of the model, for example, if the edge node has less computing resources and communication resources, the iteration time of the model may be longer, and the update speed of the model may be affected, so the process of determining the importance of the training sample by the edge node according to the associated parameters of the training sample may include: the edge node obtains the performance parameters of the edge node, and the importance of the training sample is determined according to the performance parameters and the correlation parameters of the training sample. The performance parameters include, but are not limited to, communication performance parameters and the like, and the more excellent the performance parameters are, the higher the importance of the training samples is. Based on the importance degree of the training samples determined in the mode, the updating speed of the fault detection model can be improved.
In an alternative embodiment, considering that the models themselves have different accuracies and different requirements for training samples, the lower the accuracy of the models, the more training samples that are required, and therefore, the process of determining the importance of the training samples according to the associated parameters of the training samples and the weights corresponding to the different associated parameters may include: and the edge node acquires the precision of the fault detection model stored at the local terminal, and determines the importance of the training sample according to the precision of the fault detection model and the associated parameters of the training sample. The precision of the fault detection model and the importance of the training sample can be in a negative correlation relationship, or can be in a positive correlation relationship. Because the fault detection model is continuously updated, the edge node can determine the importance of the training sample according to the precision of the latest fault detection model stored at the local terminal and the associated parameters of the training sample.
It should be noted that, in this embodiment, the importance of the training sample may also be determined by combining the performance parameter of the edge node itself, the accuracy of the fault detection model, and the correlation parameter of the training sample, and under this condition, the process of determining the importance of the training sample by the edge node according to the correlation parameter of the training sample and the weights corresponding to different correlation parameters may include: and the edge node determines the importance of the training sample according to the performance parameters of the edge node, the precision of the latest fault detection model stored at the local terminal and the associated parameters of the training sample.
Considering that different parameters have different influence degrees on model training, corresponding weights may be set for the different parameters, and under this condition, the process of determining the importance of the training samples by the edge node according to the performance parameters of the edge node, the accuracy of the fault detection model, and the associated parameters of the training samples may include: and the edge node determines the importance of the training sample according to the performance parameters of the edge node, the precision of the fault detection model stored at the local terminal, the associated parameters of the training sample and the weights corresponding to different parameters. In one example, the importance of the training samples may be calculated as follows:
Di＝λ1,iFi+λ2,iNi+λ3,iSi+λ4,iTi+λ5,iPi
wherein D isiIs the importance of the training sample for the ith edge node.
FiThe feature number of the training sample obtained by the ith edge node is shown. Extracting features from the training samples by the latest fault detection model locally stored by the ith edge node; if the number of the training samples acquired by the ith edge node is multiple, FiThe average value of the feature quantities contained in the training samples may be, for example, if the ith edge node includes 3 training samples, and the feature quantities extracted from the 3 training samples are 3, 4 and 5 respectively, FiIs 4.
NiIs the number of training samples acquired by the ith edge node.
SiThe data dimension of the training sample obtained by the ith edge node is obtained. SiThe dimension of data analysis on the training samples by the latest fault detection model locally stored by the ith edge node can be used. Or, SiOr the data dimension included in the training sample obtained by the ith edge node, where if the number of the training samples is multiple, S isiThe average value of the data dimensions included in the multiple training samples may be, for example, if the ith edge result obtains 2 training samples, the training sample 1 is 4-dimensional data, and the training sample 2 is 3-dimensional data, then SiIs 3.5.
TiIs the time, T, required for the ith edge node to transmit the acquired training sampleiThe specific calculation method of (a) can be flexibly set according to actual needs, and in one example, can be obtained according to the following method:
Figure BDA0003590051800000111
wherein Mq isiIs the size of a single packet transmitted by the ith edge node.
PiIs an absolute value of the precision of the latest fault detection model stored by the ith edge node itself and the target precision of the fault detection model, and may be an absolute value of a quotient of the precision of the latest fault detection model stored by the ith edge node itself and the target precision of the fault detection model, and the quotient isIn the above description, the target accuracy refers to the accuracy that the fault detection model needs to achieve after being trained.
λ1,i-λ5,iIs the weight of the corresponding parameter, where1,i-λ3,iIs a positive number, λ4,i-λ5,iIs a negative number.
In the embodiment, the edge node acquires the associated parameters of the training sample; the associated parameters of the training samples comprise at least one of the number of the training samples, the data dimensionality of the training samples and the number of features contained in the training samples; the edge nodes determine the importance of the training samples according to the associated parameters of the training samples, so that the precision and the updating speed of the fault detection model are improved.
In an exemplary embodiment, referring to fig. 4, fig. 4 is a flowchart of step S220 in the embodiment shown in fig. 2 in an exemplary embodiment. As shown in fig. 4, the process of encrypting the training sample to obtain the ciphertext of the training sample may include steps S221 to S222, which are described in detail as follows:
step S221, the edge node selects an encryption scheme of the training sample from the multiple encryption schemes.
In this embodiment, the edge node is preset with a plurality of encryption modes, and the encryption mode of the training sample can be selected from the plurality of encryption modes.
The specific process of selecting the encryption mode of the training sample can be flexibly set according to actual needs.
In an optional embodiment, the edge node may select an encryption scheme of the training sample from multiple encryption schemes according to its performance parameter. It should be understood that, the computation costs corresponding to different encryption manners are different, and the amount of resources required to be occupied is different, so that the edge node may select the encryption manner of the training sample from the multiple encryption manners according to its performance parameter, where the performance of the edge node and the computation cost corresponding to the encryption manner of the training sample may be in a positive correlation, that is, the better the performance of the edge node is, the higher the computation cost corresponding to the encryption manner of the training sample may be.
In another alternative embodiment, the edge node may select the encryption scheme of the training sample from a plurality of encryption schemes according to the importance of the training sample. It should be understood that, the loss and the precision of the different encryption methods are different, and therefore, the encryption method of the training sample may be selected from multiple encryption methods according to the importance of the training sample, where the importance of the training sample and the precision of the encryption method of the training sample may be in a positive correlation, that is, the higher the importance of the training sample is, the higher the precision of the encryption method of the training sample may be.
In step S222, the edge node encrypts the training sample based on the encryption mode of the training sample to obtain the ciphertext of the training sample.
After determining the encryption mode of the training sample, the edge node can encrypt the training sample according to the encryption mode of the training sample to obtain the ciphertext of the training sample.
In this embodiment, the edge node selects an encryption mode of the training sample from a plurality of encryption modes, encrypts the training sample based on the encryption mode of the training sample, and obtains a ciphertext of the training sample, so that the training sample can be encrypted based on different encryption modes to meet different requirements.
Referring to fig. 5, fig. 5 is a flowchart illustrating a federated learning-based data processing method according to an exemplary embodiment of the present application, which may be applied to the implementation environment shown in fig. 1 and executed by the cloud server 100 in the implementation environment shown in fig. 1.
As shown in fig. 5, in an exemplary embodiment, the data processing method based on federal learning may include steps S510 to S520, which are described in detail as follows:
step S510, the cloud server receives the ciphertext of the training sample sent by each edge node, and trains the fault detection model based on the ciphertext of the training sample sent by each edge node; and the ciphertext of the training sample sent by each edge node is obtained by encrypting the training sample by each edge node under the condition that the importance of the training sample is determined to be greater than a preset importance threshold.
The cloud server can receive ciphertext of the training sample sent by each edge node in the federated learning system, and train the fault detection model based on the received ciphertext of the training sample. The ciphertext of the training sample sent by each edge node is obtained by encrypting the training sample by the edge node under the condition that the importance degree of the training sample is determined to be greater than a preset importance degree threshold value. The specific way for each edge node to obtain the ciphertext of the training sample may be referred to the aforementioned description, and is not described herein again.
Herein, the training may refer to training in a model training phase or training in a model updating phase. The model training stage is that the initial fault detection model is trained to obtain a fault detection model meeting certain conditions, so that the obtained fault detection model can be used for fault detection, and the certain conditions can be flexibly set according to actual needs, for example, the precision of the fault detection model can reach target precision, or the iteration times of the fault detection model reach certain times; the model updating stage is a stage of optimizing the fault detection model based on the training sample after the fault detection model meeting certain conditions is obtained.
In some embodiments, the cloud server may further send a training instruction to each edge node, so that after each edge node receives the training instruction, the importance of the training sample is calculated, and under the condition that the importance of the training sample is greater than a preset importance threshold, the training sample is encrypted to obtain a ciphertext of the training sample, and the ciphertext of the training sample is sent to the cloud server.
In step S520, the cloud server sends the trained fault detection model to the plurality of edge nodes, respectively, so that the plurality of edge nodes perform fault detection based on the received fault detection model.
And the cloud server sends the trained fault detection model to each edge node in the federal learning system respectively. And after each edge node receives the fault detection model, fault detection is carried out based on the fault detection model.
In order to avoid the situation that the performance of the trained fault detection model is poor and cannot meet the requirements, in some embodiments, the process of sending the trained fault detection model to the plurality of edge nodes by the cloud server may include: and if the trained fault detection model meets the preset conditions, the cloud server respectively sends the trained fault detection model to the edge nodes.
That is to say, after the cloud server trains the fault detection model stored at the local terminal based on the ciphertext of the training sample, it may be determined whether the trained fault detection model meets the preset condition, and if so, the cloud server sends the trained fault detection model to each edge node respectively.
The preset conditions can be flexibly set according to actual needs. For example, if the precision of the fault detection model before training does not reach the target precision, the preset condition may be that the precision of the fault detection model after training reaches the target precision, that is, in the model training stage, the preset condition is that the precision of the fault detection model after training reaches the target precision; if the accuracy of the fault detection model before training reaches the target accuracy, the preset condition may be that the performance of the fault detection model after training exceeds the performance of the fault detection model before training, that is, in the model updating stage, the preset condition is that the performance of the fault detection model after updating exceeds the performance of the fault detection model before updating, where the performance may refer to at least one of generalization capability, accuracy, and the like of the fault detection model.
In some embodiments, the cloud server may also send the trained fault detection model and the gradient of the fault detection model to each edge node. The cloud server can also store the trained fault detection model and the gradient of the fault detection model to the local.
In this embodiment, on one hand, the cloud server may train the fault detection model according to the ciphertexts of the training samples sent by the plurality of edge nodes, so that the sources of the training samples are increased, data barriers between different devices are broken, the generalization capability of the fault detection model is improved, the precision of fault detection is improved, and data safety is ensured; on the other hand, the edge nodes are used for completing the collection of the training samples, and the cloud server is used for completing the training of the fault detection model, so that the advantages of the edge nodes and the cloud server are reasonably utilized, and the acquisition speed of the training samples and the training efficiency of the fault detection model are improved.
Referring to fig. 6, fig. 6 is a flowchart illustrating a data processing method based on federal learning according to an exemplary embodiment of the present application, where the data processing method based on federal learning may be applied to a federal learning system, and in this embodiment, the data processing method based on federal learning is described by taking a case where the federal learning system includes an edge node a, an edge node B, and a cloud server as an example, as shown in fig. 6, in an exemplary embodiment, the data processing method based on federal learning may include steps S601 to S610, which are described in detail as follows:
step S601, the edge node a obtains a first training sample, and calculates the importance of the first training sample.
The training sample obtained by the edge node A is recorded as a first training sample.
Step S602, if the importance of the first training sample is greater than the preset importance threshold, the edge node a encrypts the first training sample to obtain a ciphertext of the first training sample.
In step S603, the edge node a sends the ciphertext of the first training sample to the cloud server.
In step S604, the edge node B obtains a second training sample, and calculates the importance of the second training sample.
And recording the training sample acquired by the edge node B as a second training sample.
In step S605, if the importance of the second training sample is greater than the preset importance threshold, the edge node B encrypts the second training sample to obtain the ciphertext of the second training sample.
Step S606, the edge node B sends the ciphertext of the second training sample to the cloud server.
In step S607, the cloud server trains the fault detection model based on the ciphertext of the first training sample and the ciphertext of the second training sample.
In step S608, if the trained fault detection model meets the preset condition, the cloud server sends the trained fault detection model to the edge node a and the edge node B, respectively.
In step S609, the edge node a performs fault detection based on the received fault detection model.
After receiving the fault detection model, the edge node a may return to step S601 to perform iteration so as to continuously optimize the fault detection model.
In step S610, the edge node B performs fault detection based on the received fault detection model.
After receiving the fault detection model, the edge node B may return to step S604 to perform iteration to continuously optimize the fault detection model.
It should be noted that details of the implementation of steps S601 to S610 have been described in detail in the foregoing embodiments, and are not described herein again.
Referring to fig. 7, fig. 7 is a block diagram of a data processing apparatus based on federal learning according to an exemplary embodiment of the present application. The apparatus may be applied to a federated learning system including a cloud server and a plurality of edge nodes, and is configured at the plurality of edge nodes, respectively, as shown in fig. 7, the apparatus includes:
an importance evaluation module 701 configured to obtain a training sample and calculate an importance of the training sample;
the data encapsulation module 702 is configured to encrypt the training sample to obtain a ciphertext of the training sample if the importance of the training sample is greater than a preset importance threshold;
the edge joint training module 703 is configured to send the ciphertext of the training sample to the cloud server, so that the cloud server trains the fault detection model according to the ciphertext of the training sample;
a fault detection module 704 configured to receive the trained fault detection model sent by the cloud server, so as to perform fault detection based on the received fault detection model.
In another exemplary embodiment, the importance evaluation module 701 is further configured to obtain an associated parameter of the training sample; the associated parameters of the training samples comprise at least one of the number of the training samples, the data dimensionality of the training samples and the number of features contained in the training samples; and determining the importance of the training samples according to the associated parameters of the training samples.
In another exemplary embodiment, the importance evaluation module 701 is further configured to obtain a performance parameter of itself; and determining the importance of the training samples according to the performance parameters and the associated parameters of the training samples.
In another exemplary embodiment, the data encapsulation module 702 is further configured to select an encryption scheme of the training sample from a plurality of encryption schemes; and encrypting the training sample based on the encryption mode of the training sample to obtain the ciphertext of the training sample.
Referring to fig. 8, fig. 8 is a block diagram of a data processing apparatus based on federal learning according to an exemplary embodiment of the present application. The apparatus may be applied to a federated learning system including a cloud server and a plurality of edge nodes, and is configured in the cloud server, as shown in fig. 8, the apparatus includes:
the model aggregation module 801 is configured to receive the ciphertext of the training sample sent by each edge node, and train the fault detection model based on the ciphertext of the training sample sent by each edge node; the ciphertext of the training sample sent by each edge node is obtained by encrypting the training sample by each edge node under the condition that the importance of the training sample is determined to be greater than a preset importance threshold;
the model updating module 802 is configured to send the trained fault detection model to the plurality of edge nodes, respectively, so that the plurality of edge nodes perform fault detection based on the received fault detection model.
In another exemplary embodiment, the model updating module 802 is further configured to send the trained fault detection models to the plurality of edge nodes respectively if the trained fault detection models satisfy the preset condition.
Referring to fig. 9, fig. 9 is a schematic diagram of a federated learning system provided in an exemplary embodiment, and as shown in fig. 9, the federated learning system includes a plurality of edge nodes, edge gateways to which the plurality of edge nodes are respectively connected, and a cloud server, and the cloud server includes a model training module, a model aggregation module, a model updating module, and a data storage module.
Each edge node comprises a data pool storage module, an importance evaluation module, a data encapsulation module, an edge joint training module and a fault detection module.
Each edge gateway comprises a video stream input module and a data uploading module.
The video stream input module of each edge gateway can acquire a video stream acquired by a camera, the video stream is deframed to obtain a plurality of pictures, and the plurality of pictures are transmitted to the data pool storage module corresponding to the edge result through the data uploading module.
And the model training module of the cloud server is used for sending a training instruction to the importance evaluation module in each edge node.
After the importance degree evaluation module of each edge node receives the training instruction, acquiring a training sample from a data pool storage module of the local end, and calculating the importance degree of the training sample; if the data encapsulation module of the local end determines that the importance of the training sample is greater than a preset importance threshold, encrypting the training sample to obtain a ciphertext of the training sample; and the edge joint training module at the local end transmits the ciphertext of the training sample to the cloud server.
After a model aggregation module in the cloud server receives the ciphertext of the training sample sent by each edge node, the fault detection model is trained based on the ciphertext of the training sample, and if the model updating module of the cloud server determines that the trained fault detection model meets the preset conditions, the model updating module sends the fault detection model to each edge node and stores the fault detection model and the gradient of the fault detection model to a data storage module.
And the fault detection module of each edge node performs fault detection based on the received fault detection model.
It should be noted that the data processing apparatus based on federal learning provided in the foregoing embodiment and the data processing method based on federal learning provided in the foregoing embodiment belong to the same concept, and specific ways in which the modules and units execute operations have been described in detail in the method embodiment, and are not described herein again.
An embodiment of the present application further provides an electronic device, including: one or more processors; the storage device is configured to store one or more computer programs that, when executed by one or more processors, enable the electronic device to implement the federated learning-based data processing method provided in the above-described embodiments.
FIG. 10 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.
It should be noted that the computer system 1000 of the electronic device shown in fig. 10 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 10, the computer system 1000 includes a Central Processing Unit (CPU)1001, which can perform various appropriate actions and processes, such as executing the method in the above-described embodiment, according to a computer program stored in a Read-Only Memory (ROM) 1002 or a computer program loaded from a storage portion 1008 into a Random Access Memory (RAM) 1003. In the RAM 1003, various computer programs and data necessary for system operation are also stored. The CPU 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. An Input/Output (I/O) interface 1005 is also connected to the bus 1004.
The following components are connected to the I/O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage portion 1008 including a hard disk and the like; and a communication section 1009 including a Network interface card such as a LAN (Local Area Network) card, a modem, or the like. The communication section 1009 performs communication processing via a network such as the internet. The driver 1010 is also connected to the I/O interface 1005 as necessary. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1010 as necessary, so that a computer program read out therefrom is mounted into the storage section 1008 as necessary.
In particular, according to an embodiment of the present application, the processes described above with reference to the flowcharts may be implemented as a computer program. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising a computer program for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication part 1009 and/or installed from the removable medium 1011. When the computer program is executed by a Central Processing Unit (CPU)1001, various functions defined in the system of the present application are executed.
It should be noted that the computer readable medium shown in the embodiments of the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer-readable signal medium may include a propagated data signal with a computer program embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer program embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present application may be implemented by software or hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.
Another aspect of the present application also provides a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor of an electronic device, cause the electronic device to implement the foregoing method. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist alone without being assembled into the electronic device.
Another aspect of the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the method provided in the various embodiments described above. Wherein the computer program may be stored in a computer readable storage medium; the processor of the electronic device may read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device performs the methods provided in the above-described embodiments.
The above description is only a preferred exemplary embodiment of the present application, and is not intended to limit the embodiments of the present application, and those skilled in the art can easily make various changes and modifications according to the main concept and spirit of the present application, so that the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. A data processing method based on federated learning is applied to a federated learning system comprising a cloud server and a plurality of edge nodes, wherein the method is respectively executed by the plurality of edge nodes, and the method comprises the following steps:
acquiring a training sample, and calculating the importance of the training sample;
if the importance of the training sample is larger than a preset importance threshold, encrypting the training sample to obtain a ciphertext of the training sample;
sending the ciphertext of the training sample to the cloud server so that the cloud server trains a fault detection model according to the ciphertext of the training sample;
and receiving the trained fault detection model sent by the cloud server, and performing fault detection based on the received fault detection model.
2. The method of claim 1, wherein said calculating the importance of the training samples comprises:
acquiring the correlation parameters of the training samples; the correlation parameters of the training samples comprise at least one of the number of the training samples, the data dimensions of the training samples and the number of features contained in the training samples;
and determining the importance of the training sample according to the associated parameters of the training sample.
3. The method of claim 2, wherein determining the importance of the training samples according to the associated parameters of the training samples comprises:
acquiring performance parameters of the device;
and determining the importance of the training samples according to the performance parameters and the associated parameters of the training samples.
4. The method of claim 1, wherein the encrypting the training samples to obtain the ciphertext of the training samples comprises:
selecting an encryption mode of the training sample from a plurality of encryption modes;
and encrypting the training sample based on the encryption mode of the training sample to obtain the ciphertext of the training sample.
5. A data processing method based on federated learning is applied to a federated learning system comprising a cloud server and a plurality of edge nodes, and the method is executed by the cloud server and comprises the following steps:
receiving a ciphertext of a training sample sent by each edge node, and training a fault detection model based on the ciphertext of the training sample sent by each edge node; the ciphertext of the training sample sent by each edge node is obtained by encrypting the training sample by each edge node under the condition that the importance degree of the training sample is determined to be greater than a preset importance degree threshold;
and respectively sending the trained fault detection models to the edge nodes so as to enable the edge nodes to carry out fault detection based on the received fault detection models.
6. The method of claim 5, wherein the sending the trained fault detection models to the plurality of edge nodes, respectively, comprises:
and if the trained fault detection model meets the preset conditions, respectively sending the trained fault detection model to the edge nodes.
7. A data processing device based on federal learning is applied to a federal learning system comprising a cloud server and a plurality of edge nodes, wherein the device is respectively configured on the plurality of edge nodes, and the device comprises:
the importance evaluation module is configured to obtain a training sample and calculate the importance of the training sample;
the data encapsulation module is configured to encrypt the training sample to obtain a ciphertext of the training sample if the importance of the training sample is greater than a preset importance threshold;
the edge joint training module is configured to send the ciphertext of the training sample to the cloud server so that the cloud server trains a fault detection model according to the ciphertext of the training sample;
and the fault detection module is configured to receive the trained fault detection model sent by the cloud server, so as to perform fault detection based on the received fault detection model.
8. A data processing device based on federated learning is applied to a federated learning system comprising a cloud server and a plurality of edge nodes, and the device is configured in the cloud server, and the device comprises:
the model aggregation module is configured to receive the ciphertext of the training sample sent by each edge node, and train the fault detection model based on the ciphertext of the training sample sent by each edge node; the ciphertext of the training sample sent by each edge node is obtained by encrypting the training sample by each edge node under the condition that the importance degree of the training sample is determined to be greater than a preset importance degree threshold;
and the model updating module is configured to send the trained fault detection models to the edge nodes respectively so that the edge nodes perform fault detection based on the received fault detection models.
9. An electronic device, comprising:
one or more processors;
storage means for storing one or more computer programs that, when executed by the one or more processors, cause the electronic device to implement the federated learning-based data processing method of any one of claims 1-6.
10. A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor of an electronic device, causes the electronic device to execute the federal learning based data processing method as claimed in any of claims 1 to 6.
CN202210377079.4A 2022-04-11 2022-04-11 Data processing method and device based on federal learning, equipment and storage medium Active CN114707606B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210377079.4A CN114707606B (en) 2022-04-11 2022-04-11 Data processing method and device based on federal learning, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210377079.4A CN114707606B (en) 2022-04-11 2022-04-11 Data processing method and device based on federal learning, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN114707606A true CN114707606A (en) 2022-07-05
CN114707606B CN114707606B (en) 2023-12-22

Family

ID=82172584

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210377079.4A Active CN114707606B (en) 2022-04-11 2022-04-11 Data processing method and device based on federal learning, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114707606B (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109444570A (en) * 2018-09-18 2019-03-08 中国人民解放军第五七九工厂 A kind of electronic product fault diagnosis module and method based on memory
CN111143308A (en) * 2019-12-26 2020-05-12 许昌中科森尼瑞技术有限公司 Federal learning-based high-low voltage motor data processing method, system and device
CN111537945A (en) * 2020-06-28 2020-08-14 南方电网科学研究院有限责任公司 Intelligent ammeter fault diagnosis method and equipment based on federal learning
CN112257873A (en) * 2020-11-11 2021-01-22 深圳前海微众银行股份有限公司 Training method, device, system, equipment and storage medium of machine learning model
WO2021228110A1 (en) * 2020-05-14 2021-11-18 深圳前海微众银行股份有限公司 Federated modeling method, device, equipment, and computer-readable storage medium
CN113901412A (en) * 2021-10-18 2022-01-07 国网山东省电力公司济宁供电公司 Power quality disturbance detection method and device, electronic equipment and storage medium
CN114006769A (en) * 2021-11-25 2022-02-01 中国银行股份有限公司 Model training method and device based on horizontal federal learning
CN114118442A (en) * 2021-11-24 2022-03-01 中国电信股份有限公司 Model training method, system, equipment and medium based on longitudinal federal learning
CN114169010A (en) * 2021-12-13 2022-03-11 安徽理工大学 Edge privacy protection method based on federal learning

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109444570A (en) * 2018-09-18 2019-03-08 中国人民解放军第五七九工厂 A kind of electronic product fault diagnosis module and method based on memory
CN111143308A (en) * 2019-12-26 2020-05-12 许昌中科森尼瑞技术有限公司 Federal learning-based high-low voltage motor data processing method, system and device
WO2021228110A1 (en) * 2020-05-14 2021-11-18 深圳前海微众银行股份有限公司 Federated modeling method, device, equipment, and computer-readable storage medium
CN111537945A (en) * 2020-06-28 2020-08-14 南方电网科学研究院有限责任公司 Intelligent ammeter fault diagnosis method and equipment based on federal learning
CN112257873A (en) * 2020-11-11 2021-01-22 深圳前海微众银行股份有限公司 Training method, device, system, equipment and storage medium of machine learning model
CN113901412A (en) * 2021-10-18 2022-01-07 国网山东省电力公司济宁供电公司 Power quality disturbance detection method and device, electronic equipment and storage medium
CN114118442A (en) * 2021-11-24 2022-03-01 中国电信股份有限公司 Model training method, system, equipment and medium based on longitudinal federal learning
CN114006769A (en) * 2021-11-25 2022-02-01 中国银行股份有限公司 Model training method and device based on horizontal federal learning
CN114169010A (en) * 2021-12-13 2022-03-11 安徽理工大学 Edge privacy protection method based on federal learning

Also Published As

Publication number Publication date
CN114707606B (en) 2023-12-22

Similar Documents

Publication Publication Date Title
US20220245472A1 (en) Data processing method and apparatus, and non-transitory computer readable storage medium
CN110929880A (en) Method and device for federated learning and computer readable storage medium
CN108491267B (en) Method and apparatus for generating information
CN113627085B (en) Transverse federal learning modeling optimization method, equipment and medium
US20150278523A1 (en) Method and system for testing cloud based applications and services in a production environment using segregated backend systems
CN111563267A (en) Method and device for processing federal characteristic engineering data
CN111915019A (en) Federal learning method, system, computer device, and storage medium
US20180240131A1 (en) Identifying deceptive social media content
CN113127307A (en) Method for processing tracing request, related device, system and storage medium
CN110569282A (en) Data processing method, data processing device, computing equipment and computer readable storage medium
CN112861939A (en) Feature selection method, device, readable storage medium and computer program product
CN115049070A (en) Screening method and device of federal characteristic engineering data, equipment and storage medium
US20130290476A1 (en) Identifying Business Transactions from Traffic in an Enterprise Content Management System
CN117176802B (en) Full-link monitoring method and device for service request, electronic equipment and medium
CN112437022B (en) Network traffic identification method, device and computer storage medium
CN112507265B (en) Method and device for abnormality detection based on tree structure and related products
CN114707606B (en) Data processing method and device based on federal learning, equipment and storage medium
CN112565404A (en) Data processing method, edge server, center server and medium
CN116578911A (en) Data processing method, device, electronic equipment and computer storage medium
CN113383328A (en) Cloud platform and method for efficiently processing collected data
CN110020166B (en) Data analysis method and related equipment
CN113780247B (en) Traffic light detection method and device, electronic equipment and computer readable medium
Priovolos et al. Using anomaly detection techniques for securing 5G infrastructure and applications
WO2016026309A1 (en) Comment message processing method and apparatus
US7930161B1 (en) Method and apparatus for horizontal and vertical modeled representation and analysis of distributed systems

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant