EP4350572A1 - Procédé, appareil et système pour générer un modèle de réseau neuronal, dispositifs, support et produit-programme - Google Patents

Procédé, appareil et système pour générer un modèle de réseau neuronal, dispositifs, support et produit-programme Download PDF

Info

Publication number
EP4350572A1
EP4350572A1 EP22827695.2A EP22827695A EP4350572A1 EP 4350572 A1 EP4350572 A1 EP 4350572A1 EP 22827695 A EP22827695 A EP 22827695A EP 4350572 A1 EP4350572 A1 EP 4350572A1
Authority
EP
European Patent Office
Prior art keywords
model
parameter
subnetwork
hypemetwork
devices
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP22827695.2A
Other languages
German (de)
English (en)
Inventor
Mi LUO
Fei Chen
Zhenguo Li
Jiashi FENG
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of EP4350572A1 publication Critical patent/EP4350572A1/fr
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/098—Distributed learning, e.g. federated learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Definitions

  • Embodiments of the present disclosure mainly relate to the field of artificial intelligence. More specifically, embodiments of the present disclosure relate to a method, an apparatus, and a system for generating a neural network model, a device, a computer-readable storage medium, and a computer program product.
  • Federated learning Federated Learning
  • Federated Learning can fully utilize data and computing capabilities at a client, allowing multiple parties to collaborate to build a general and more robust machine learning model without sharing data.
  • federated learning can resolve key problems such as data ownership, data privacy, and data access permission, and has great business value.
  • embodiments of the present disclosure provide a solution for generating a neural network model applicable to federated learning.
  • a method for generating a neural network model including: A first device sends an indication about a structure of a subnetwork model to a second device, where the subnetwork model is determined by adjusting a structure of a hypemetwork model. The first device receives a parameter of the subnetwork model from the second device, where the parameter of the subnetwork model is determined by the second device based on the indication and the subnetwork model. The first device trains the subnetwork model based on the received parameter of the subnetwork model.
  • an efficient federated learning scheme between a plurality of devices is provided, so that communication costs and device computing costs required in a federated learning process are further reduced while improving model precision.
  • the method may further include: The first device obtains a preconfigured parameter of the hypemetwork model.
  • the first device determines the structure of the subnetwork model based on the preconfigured parameter and by adjusting the structure of the hypemetwork model.
  • a personalized subnetwork model may be generated on a device.
  • the obtaining a preconfigured parameter of the hypemetwork model may include: The first device generates a local parameter by training the hypemetwork model. The first device sends the local parameter to the second device. The first device receives the preconfigured parameter from the second device, where the preconfigured parameter is determined by the second device based on at least the local parameter received from the first device.
  • an optimized preconfigured parameter can be generated for a hypemetwork model based on data distribution of a device, to facilitate determining of a structure of a subnetwork model at low computing costs.
  • the determining the structure of the subnetwork model may include: initializing the subnetwork model to a hypemetwork model with the preconfigured parameter; and iteratively updating the subnetwork model by performing the following operations at least once: adjusting a structure of a plurality of layers of the subnetwork model to obtain a plurality of candidate network models; selecting one candidate network model based on accuracy of the plurality of candidate network models to update the subnetwork model; and if the subnetwork model meets a constraint of the first device, determining the structure of the subnetwork model.
  • a device may simplify a hypemetwork model by using accuracy as a measurement indicator, to obtain a personalized neural network model that meets a resource limitation and data distribution of the device.
  • the adjusting a structure of a plurality of layers may include: deleting parameters related to some nodes of one layer in the plurality of layers to obtain one of the plurality of candidate network models.
  • the method further includes: determining the some nodes based on a predetermined quantity percentage. In the foregoing manner, a node and a parameter that have small impact on model accuracy can be removed, so that a model structure is simplified and model accuracy is ensured.
  • the constraint includes: a calculation amount of the subnetwork model is less than a first threshold, or a quantity of parameters of the subnetwork model is less than a second threshold. In the foregoing manner, a calculation amount and a quantity of parameters of a subnetwork model of a device may be reduced to meet a corresponding resource constraint.
  • the first threshold and the second threshold are both associated with performance of the first device. In the foregoing manner, a corresponding resource constraint is set based on device performance.
  • the indication is in a form of a mask, and the mask indicates whether the subnetwork model has a corresponding parameter of the hypemetwork model.
  • the method according to the first aspect may further include: The first device determines a change in the parameter by calculating a difference between the parameter of the trained subnetwork model and the parameter received from the second device.
  • a method for generating a neural network model includes: A second device receives an indication about a structure of a subnetwork model from a plurality of first devices, where the subnetwork model is determined by adjusting a structure of a hypemetwork model. The second device determines a parameter of the subnetwork model based on the indication and the hypemetwork model. The second device sends the parameter of the subnetwork model to the plurality of first devices for the plurality of first devices to separately train the subnetwork model.
  • an efficient federated learning scheme between a plurality of devices is provided, so that communication costs and device computing costs required in a federated learning process are further reduced while improving model precision.
  • the indication is in a form of a mask, and the mask indicates whether the subnetwork model has a corresponding parameter of the hypemetwork model.
  • the method according to the second aspect may further include: The second device receives, from the plurality of first devices, a change in the parameter of the trained subnetwork model.
  • the method according to the second aspect may further include: The second device updates the hypemetwork model by using the received change in the parameter.
  • the hypemetwork model at the second device may be iteratively updated, to generate a hypemetwork model that better meets data distribution of the first device of a device.
  • the updating the hypemetwork model may include: updating the hypemetwork model based on an update weight of the parameter of the subnetwork model, where the update weight depends on a quantity of subnetwork models having the parameter.
  • a parameter of a server device may be weighted and updated, to generate a hypemetwork model that better meets data distribution of a client.
  • the method according to the second aspect may further include: The second device determines a preconfigured parameter of the hypemetwork model for the plurality of first devices to determine respective subnetwork models from the hypemetwork model.
  • a plurality of first devices may use local data to determine a structure of a subnetwork model starting from hypemetwork models with a same preconfigured parameter.
  • the determining a preconfigured parameter of the hypemetwork model may include: determining the preconfigured parameter of the hypernetwork model based on a local parameter determined by locally training the hypemetwork model by the plurality of first devices. In the foregoing manner, an optimized preconfigured parameter can be generated for a hypemetwork model based on data distribution of a plurality of first devices, so that the first devices can generate a personalized neural network model at low computing costs.
  • an apparatus for generating a neural network model including: a sending unit, configured to send an indication about a structure of a subnetwork model to a second device, where the subnetwork model is determined by adjusting a structure of a hypemetwork model; a receiving unit, configured to receive a parameter of the subnetwork model from the second device, where the parameter of the subnetwork model is determined by the second device based on the indication and the hypemetwork model; and a training unit, configured to train the subnetwork model based on the received parameter of the subnetwork model.
  • the sending unit is further configured to send a parameter of the trained subnetwork model to the second device for the second device to update the hypemetwork model.
  • the receiving unit may be further configured to obtain a preconfigured parameter of the hypemetwork model.
  • the apparatus according to the third aspect further includes a model determining unit.
  • the model determining unit is configured to determine the structure of the subnetwork model based on the preconfigured parameter and by adjusting the structure of the hypemetwork model. In the foregoing manner, a personalized subnetwork model may be generated on a device.
  • the training unit is configured to locally train the hypemetwork model to determine a local parameter of the hypemetwork model.
  • the sending unit is further configured to send the local parameter to the second device.
  • the receiving unit is further configured to receive the preconfigured parameter from the second device, where the preconfigured parameter is determined by the second device based on at least the local parameter received from a first device.
  • an optimized preconfigured parameter can be generated for a hypemetwork model based on data distribution of a device, to facilitate determining of a structure of a subnetwork model by the device at low computing costs.
  • the model determining unit is further configured to initialize the subnetwork model to a hypemetwork model with the preconfigured parameter.
  • the model determining unit is further configured to iteratively update the subnetwork model by performing the following operations at least once: adjusting a structure of a plurality of layers of the subnetwork model to obtain a plurality of candidate network models; selecting one candidate network model based on accuracy of the plurality of candidate network models to update the subnetwork model; and if the subnetwork model meets a constraint of the first device, stopping the iterative update.
  • a device may simplify a hypemetwork model by using accuracy as a measurement indicator, to obtain a personalized neural network model that meets a resource limitation and data distribution of the device.
  • the model determining unit is further configured to delete parameters related to some nodes of one layer in the plurality of layers to obtain one of the plurality of candidate network models. In some embodiments, the model determining unit is configured to determine the some nodes based on a predetermined quantity percentage. In the foregoing manner, a node and a parameter that have small impact on model accuracy can be removed, so that a model structure is simplified and model accuracy is ensured.
  • a calculation amount of the subnetwork model is less than a first threshold, or a quantity of parameters of the subnetwork model is less than a second threshold. In the foregoing manner, a calculation amount and a quantity of parameters of a subnetwork model of a device may be reduced to meet a corresponding resource constraint.
  • the first threshold and the second threshold are both associated with performance of the first device. In the foregoing manner, a corresponding resource constraint is set based on device performance.
  • the indication is in a form of a mask, and the mask indicates whether the subnetwork model has a corresponding parameter of the hypemetwork model.
  • the sending unit may be further configured to send the change in the parameter of the trained subnetwork model to the second device for the second device to update the hypernetwork model, where it may be determined that the change in the parameter is a difference between the parameter of the trained subnetwork model and the parameter received from the second device.
  • an apparatus for generating a neural network model including: a receiving unit, configured to receive an indication about a structure of a subnetwork model from a plurality of first devices, where the subnetwork model is determined by adjusting a structure of a hypemetwork model; a unit for determining a parameter of a subnetwork model, configured to determine a parameter of the subnetwork model based on the indication and the hypemetwork model; a sending unit, configured to send the parameter of the subnetwork model to the plurality of first devices for the plurality of first devices to separately train the subnetwork model; and a hypemetwork update unit, where the receiving unit is further configured to receive a parameter of the trained subnetwork model from the plurality of first devices; and the hypemetwork update unit is configured to update the hypemetwork model by using the received parameter.
  • a receiving unit configured to receive an indication about a structure of a subnetwork model from a plurality of first devices, where the subnetwork model is determined by adjusting a structure of a hypemetwork model
  • the indication is in a form of a mask, and the mask indicates whether the subnetwork model has a corresponding parameter of the hypemetwork model.
  • the receiving unit may be further configured to receive, from the plurality of first devices, a change in the parameter of the trained subnetwork model. In the foregoing manner, in a federated learning process, only a parameter that changes after training needs to be transmitted, to reduce communication costs of transmission between a server device and a client device.
  • the hypernetwork update unit may be further configured to update the hypemetwork model by using the received change in the parameter.
  • the hypemetwork model at the second device may be iteratively updated, to generate a hypemetwork model that better meets data distribution of the first device of a device.
  • the hypemetwork update unit may be further configured to update the hypemetwork model based on an update weight of the parameter of the subnetwork model, where the update weight depends on a quantity of subnetwork models having the parameter.
  • a parameter of a server device may be weighted and updated, to generate a hypemetwork model that better meets data distribution of a client.
  • the hypemetwork update unit may be further configured to determine a preconfigured parameter of the hypemetwork model for the plurality of first devices to determine respective subnetwork models from the hypemetwork model.
  • a plurality of first devices may use local data to determine a structure of a subnetwork model starting from hypemetwork models with a same preconfigured parameter.
  • the hypemetwork update unit may be further configured to determine the preconfigured parameter based on a local parameter determined by locally training the hypemetwork model by the plurality of first devices.
  • an optimized preconfigured parameter can be generated for a hypemetwork model based on data distribution of a plurality of first devices, so that the first devices can generate a personalized neural network model at low computing costs.
  • a system for generating a neural network model including: a first device, configured to perform the method according to the first aspect of the present disclosure; and a second device, configured to perform the method according to the second aspect of the present disclosure.
  • an electronic device including: at least one computing unit and at least one memory.
  • the at least one memory is coupled to the at least one computing unit and stores instructions executed by the at least one computing unit.
  • the device is enabled to implement the method according to any one of the implementations of the first aspect or the second aspect.
  • a computer-readable storage medium storing a computer program.
  • the computer program is executed by a processor, the method according to any one of the implementations of the first aspect or the second aspect is implemented.
  • a computer program product including computer-executable instructions. When the instructions are executed by a processor, some or all steps of the method according to any one of the implementations of the first aspect or the second aspect are implemented.
  • the system in the fifth method, the electronic device in the sixth aspect, the computer storage medium in the seventh aspect, or the computer program product in the eighth aspect are all configured to perform the method provided in the first aspect and/or the second aspect. Therefore, explanations for or descriptions of the first aspect and/or the second aspect are also applicable to the fifth aspect, the sixth aspect, the seventh aspect, and the eighth aspect.
  • beneficial effects that can be achieved in the fifth aspect, the sixth aspect, the seventh aspect, and the eighth aspect refer to beneficial effects in corresponding methods. Details are not described herein again.
  • the term “include” and similar terms thereof should be understood as open inclusion, that is, “include but are not limited to”.
  • the term “based” should be understood as “at least partially based”.
  • the terms “one embodiment” or “the embodiment” should be understood as “at least one embodiment”.
  • the terms “first”, “second”, and the like may indicate different or same objects. Other explicit and implicit definitions may also be included below.
  • a “neural network” is capable of processing inputs and providing corresponding outputs, which usually includes an input layer and an output layer, and one or more hidden layers between the input layer and the output layer.
  • a neural network used in a deep learning application usually includes many hidden layers, to increase the depth of the network. The layers of the neural network are connected in sequence, so that an output of a former layer is provided as an input of a latter layer. The input layer receives an input of the neural network, and an output of the output layer is used as a final output of the neural network.
  • the terms “neural network”, “network”, “neural network model”, and “model” may be used interchangeably.
  • federated learning is a machine learning algorithm, and usually includes one server device and a plurality of client devices.
  • a machine learning model is trained by using data of a plurality of clients and by transmitting non-sensitive information such as a parameter, to achieve privacy protection.
  • a “hypemetwork” refers to a neural network structure shared by a server and a plurality of clients in a federated learning environment.
  • a “subnetwork” (also referred to as a subnetwork model or a personalized network) is a neural network model that is independently maintained, modified, trained, and inferred by each client in the federated learning environment.
  • the subnetwork model and the subnetwork may be used interchangeably, and the hypemetwork model and the hypemetwork may be used interchangeably.
  • the subnetwork model may be obtained by pruning the hypemetwork.
  • the pruning refers to deleting some parameters and corresponding calculation operations in a network model and keeping a calculation process of the remaining part unchanged.
  • Federated learning is a distributed machine learning technology.
  • the server device and the plurality of client devices separately perform model training locally without collecting data.
  • federated learning has attracted more and more attention.
  • neural network architecture search (Neural Architecture Search, NAS) methods are used to automatically search for a better model architecture.
  • these methods only search for a globally shared model without considering client-side personalization.
  • data distribution for example, pictures taken by different users
  • resource constraints for example, different client devices have different computing capabilities. Therefore, if a same model is deployed on all clients, performance of the model deteriorates, affecting user experience.
  • data is not evenly distributed on different clients. As a result, a global model cannot achieve optimal performance on a plurality of clients at the same time.
  • embodiments of the present disclosure provide a federated learning-based personalized network architecture search framework and a training method.
  • personalized model architectures are searched in a federated learning system through server-client interaction to provide personalized model architectures for clients with differentiated computer capabilities to meet resource constraints (for example, model sizes, floating-point operations FLOPS, and inference speeds).
  • resource constraints for example, model sizes, floating-point operations FLOPS, and inference speeds.
  • effective federated training is performed for different client network architectures.
  • communication costs and device computing costs required in a federated learning process are further reduced while improving model precision.
  • Embodiments of the present disclosure provide a personalized network structure search framework (Architecture of Personalization Federated Learning, APFL) for a resource constrained device.
  • the framework can customize model architectures for different devices based on specific resource requirements and local data distribution.
  • the framework provides a communication-friendly subnetwork federated training strategy to efficiently complete local machine learning tasks on devices.
  • FIG. 1 is an architectural diagram of a federated learning system 100 for implementing the foregoing framework according to some embodiments of the present disclosure.
  • the system 200 includes a plurality of first devices 110-1, 110-2, ..., 110-K (collectively referred to as a first device 110) and a second device 120.
  • the first device may be referred to as a client device or a client for short, and may be a terminal device having limited computing resources (for example, a processor computing capability, memory, and storage space), for example, a smartphone, a desktop computer, a notebook computer, or a tablet computer.
  • the second device 120 may be referred to as a distributed or centralized server device or server cluster implemented in a cloud computing environment, and generally has higher computing resources or performance than those or that of a terminal device.
  • the plurality of first devices 110 and the second device 120 may communicate with each other and transmit data to each other in various wired or wireless manners.
  • the first device 110 and the second device 120 may construct and train neural network models, for example, federated learning, for various applications such as image recognition, speech processing, and natural speech processing in a collaborative manner.
  • the first device 110 includes a subnetwork 111, local data 112, and a control unit 113.
  • the control unit 113 may be a virtual or physical processing unit, and can, for example, use the local data 112 to train the subnetwork 111, use the local data 112 to adjust a structure of the subnetwork 111, and may further enable the first devices 110 to communicate with the second device 120, for example, receive a parameter of the subnetwork from the second device or transmit a parameter of the subnetwork or other information to a server.
  • the second device 120 may include a hypemetwork 121 and a control unit 123.
  • control unit 123 may alternatively be a virtual or physical processing unit, and can maintain the hypemetwork 121, for example, update a parameter of the hypemetwork 121 through training or based on a parameter of the subnetwork 111, and may further enable the second device 120 to communicate with some or all devices in the first devices 110, for example, send the parameter of the hypemetwork 123 to the first devices 110, send parameters of the subnetwork 111 for different first devices 110, and receive the parameter of the subnetwork 111 from the first devices 110.
  • the subnetwork model and the subnetwork may be used interchangeably, and the hypemetwork model and the hypemetwork may be used interchangeably.
  • the hypemetwork 123 may be shared between the first device 110 and the second device 120, so that the first device 110 may prune a hypemetwork to determine the subnetwork 111 of the first device 110.
  • the determined subnetwork 111 may meet resource constraints and data distributions of different first devices 110.
  • a federated training method for a hypemetwork 121 and a subnetwork 111 implemented by a plurality of first devices 110 and a second device is provided.
  • the following provides detailed description with reference to FIG. 2 to FIG. 8 .
  • a plurality of first devices 110 and a second device 120 transmit parameters and other information of respective neural network models to each other, to implement federated training.
  • FIG. 2 is a schematic diagram of communication 200 for federated training according to an embodiment of the present disclosure.
  • the plurality of first devices 110 send (202) indications about structures of subnetwork models 111 of the first devices to the second device 120.
  • a structure of the subnetwork model 111 is determined based on a hypemetwork 123, and an indication of the structure may be in a form of a mask, for example, a 0-1 vector having the same dimension as that of a hypernetwork model. If a value of a component is 1, it indicates that a corresponding parameter of the subnetwork model is reserved. If a value of a component is 0, it indicates that a corresponding parameter is deleted from the hypemetwork 123.
  • the second device 120 calculates (204) a parameter of the subnetwork model 111 of each first device 110 based on the received indication. For example, the second device 120 may perform component multiplication on the parameter of the hypemetwork model 121 and the received mask, and extract a non-zero component to obtain the parameter of the subnetwork model 111. Then, the second device transmits (206) the obtained parameter of the subnetwork model 111 to the corresponding first device.
  • each first device 110 trains (208), based on the received parameter, various subnetwork models 111 by using local data 112, and transmits (210) the parameter of the trained subnetwork model 111 to the second device 120.
  • the transmitted parameter may be a value of the parameter or may be a change in the parameter, to be specific, an update amount.
  • the second device 120 updates the hypemetwork model 121 by using the received parameter of the subnetwork model 111 trained by the first device 110.
  • the parameter of the hypemetwork model 121 is updated by calculating an average value, a weighted average, or another manner of the parameters.
  • Such federated learning may be iterative, in other words, actions of 204, 206, 208, 210, 212 are repeated after the second device 120 updates the hypemetwork model 121.
  • the first device 110 that participates in the foregoing process each time may be variable, for example, may be any subset of a currently online first device 110.
  • the second device 120 maintains a hypemetwork ⁇ , a personalized network ⁇ k of each first device 110 is a subnetwork of ⁇ , and architectures of the subnetworks ⁇ 1 , ... , ⁇ K may be different.
  • FIG. 3 is a flowchart of a process 300 for generating a neural network model according to some embodiments of the present disclosure.
  • Box 310 Send an indication of a structure of a subnetwork model to a second device, where the subnetwork model is determined by adjusting a structure of a hypemetwork model.
  • the indication may be in a form of a mask (mask), indicating whether the subnetwork model 111 has a corresponding parameter of the hypemetwork model 121.
  • the first device u k sends the mask z k , to the second device 120.
  • z k is a 0-1 vector that has the same dimension as that of the hypemetwork 121 ⁇ , and indicates a subnetwork structure ⁇ k .
  • the federated training of a subnetwork may be iteratively performed, for example, is performed T' rounds.
  • K' ( K' ⁇ K ) first devices may be selected from the K first devices 110 to participate in the round of iteration.
  • actions in Blocks 720 to 740 are performed.
  • Block 320 Receive a parameter of the subnetwork model from the second device.
  • the parameter of the subnetwork model 111 is computed by the second device 120 by using a current parameter of the hypemetwork model 121 and a mask of the subnetwork model. For example, ⁇ ( t ) ⁇ z L k is calculated.
  • the second device 120 may extract a non-zero component and transmit the non-zero component to the first device u k .
  • ⁇ indicates multiplication of vector components. In this way, the first device 110 receives the parameter of the subnetwork model 111.
  • Block 330 Train the subnetwork model based on the received parameter.
  • the first device u k receives and assigns a value to the parameter of the subnetwork model 111, and updates the subnetwork parameter by using local training data 112 to obtain an updated parameter
  • S' -step gradient descent may be performed with local training data, as shown in the following Equation (1):
  • ⁇ is a learning rate.
  • L is a cross-entropy loss function.
  • D l k , s is a random sample of a training set D L K .
  • Block 340 Send a parameter of the trained subnetwork model to the second device.
  • the parameter that is of the subnetwork model 111 and that is sent by the first device 110 to the second device 120 may include a change in the calculated parameter, and transmit the calculated to the second device.
  • the second device 120 may also directly send the parameter of the trained subnetwork model to the second device 120 without considering the change in the parameter.
  • the second device 120 receives the parameter of the trained subnetwork model 111 from K' clients, and updates the hypemetwork 121 based on the received parameter.
  • the parameter received by the second device 120 may be a change in a parameter or an update amount, or may be an updated parameter of the subnetwork model 121.
  • FIG. 4 is a flowchart of a process 400 for generating a neural network model according to an embodiment of the present disclosure.
  • the process 400 may be implemented by the second device 120 shown in FIG. 2 , and more specifically, is performed by the control unit 123.
  • Block 410 Receive an indication about a structure of a subnetwork model 111 from a plurality of first devices 110, where the subnetwork model 111 is determined by adjusting a structure of a hypemetwork model 121.
  • the indication is in a form of a mask, and the mask indicates whether the subnetwork model has a corresponding parameter of the hypemetwork model.
  • the mask may be a 0-1 vector. If a component in the vector is 1, it indicates that a corresponding parameter in a hypemetwork is reserved in the subnetwork model. If a value of a component is 0, it indicates that a corresponding parameter is deleted.
  • Block 420 Determine a parameter of the subnetwork model based on the indication and the hypemetwork model.
  • the second device 120 performs component multiplication on an indication in a form of a mask and the parameter of the hypemetwork 121, and determines a non-zero vector in the indication as the parameter of the subnetwork model 111.
  • Block 430 Send the parameter of the subnetwork model to the plurality of first devices for the plurality of first devices to separately train the subnetwork model.
  • the first devices 110 train local personalized subnetwork models 111 by using local training data 112 through a gradient descent method. As described above, after training, the first devices 110 send parameters of the trained subnetwork models 111 to the second device 120.
  • Block 440 Receive parameters of trained subnetwork models from the plurality of first devices.
  • the parameter of the trained subnetwork model received by the second device 120 includes a change in the parameter, and the change in the parameter may be a difference between the parameter before the subnetwork model is trained and the parameter after the subnetwork model is trained.
  • Block 450 Update the hypemetwork model by using the received parameters.
  • the hypemetwork model 121 is updated based on an update weight of the parameter of the subnetwork model 111.
  • the update weight depends on a quantity of subnetwork models having the corresponding parameter in the subnetwork models of these first devices 110.
  • the update weight Z t may be set according to the following Equation (2):
  • Recip(x) is the reciprocal of each component in the vector x. If a value of the component is set to 0, the reciprocal is set to 0.
  • the second device 120 updates a hypemetwork parameter based on the update weight and the received parameter of the subnetwork model 111.
  • the hypemetwork parameter is updated according to the following Equation (3):
  • ⁇ ( t +1) is a hypemetwork parameter after this round of iteration.
  • ⁇ ( t ) is a hypemetwork parameter at the beginning of a local iteration.
  • Z t is an update weight. is a change in the parameter at the first device 110.
  • ⁇ indicates multiplication of vector components.
  • the federated learning process may be repeatedly executed T' times as required, to complete a federated training process between the second device and the plurality of first devices.
  • the second device 120 determines a training result of the subnetwork model 111 based on an indication or a mask of the structure of the first device 110.
  • a specific manner is as follows: The second device 120 calculates ⁇ ( T' ) ⁇ z k , extracts a non-zero component and transmits the non-zero component to a specific first device u k .
  • the specific first device u k receives and sets a final subnetwork parameter to
  • the subnetwork model 111 is determined by the first device 110 by adjusting the structure of the hypemetwork model 123, and the determined subnetwork model 111 can meet resource constraints and data distribution of the first devices 110.
  • An embodiment of the present disclosure further provides a method for generating such a personalized subnetwork. The method is described below with reference to FIG. 5 to FIG. 8 .
  • FIG. 5 is a schematic diagram of communication 500 for determining a structure of a subnetwork model according to an embodiment of the present disclosure.
  • a personalized subnetwork model suitable for the first device 110 is determined in a federated learning manner.
  • the second device 120 transmits the parameter of the hypemetwork 121 to the plurality of first devices 110.
  • the plurality of first devices 110 may be some of all the first devices 110, preferably first devices with high performance.
  • the first devices 110 have only a shared hypemetwork, and the first devices 110 use the received parameter to train (504) the hypemetwork.
  • the first devices 110 then transmit (506) the parameters of the trained hypemetwork to the second device 120.
  • the second device 120 may gather (508) these parameters to update the hypemetwork 121.
  • the updated parameters of the hypemetwork may be transmitted (510) to the first device 110 as preconfigured parameters.
  • the first device 120 may use the preconfigured parameters to initialize the local hypemetwork, and then adjust (for example, prune) the structure of the hypemetwork to determine a subnetwork model 111 that meets its own resource constraints and data distribution.
  • the hypemetwork 123 of the second device 120 may be updated iteratively, for example, the second device 120 may repeat actions 502 to 508 after updating the hypemetwork 123.
  • different first devices 110 may be selected for each iteration.
  • a process in which the first device 110 and the second device 120 perform federated learning to determine the personalized subnetwork 111 is described below in detail with reference to FIG. 6 to FIG. 8 .
  • FIG. 6 is a flowchart of a process 400 for initializing a hypemetwork according to some embodiments of the present disclosure.
  • the process 400 may be implemented by the second device 120, and more specifically, is performed by the control unit 123.
  • a preconfigured parameter of the hypemetwork 121 may be determined, so that the first device 110 subsequently generates the personalized subnetwork model 111.
  • Block 410 Select a hypemetwork, and generate an initial parameter.
  • the second device 21 may select a neural network model, for example, MobileNetV2, from a neural network model library as the hypemetwork 121.
  • the neural network model may include an input layer, a plurality of hidden layers, and an output layer. Each layer includes a plurality of nodes, and nodes between layers are connected by weighted edges. A weight may be referred to as a model parameter w, and may be learned through training.
  • the neural network model has a corresponding resource budget, including a model size, floating-point operations, a quantity of parameters, inference speeds, and the like.
  • the second device 120 may randomly generate an initial parameter w (0) of the hypemetwork 121.
  • Block 620 Send the hypemetwork parameter to a plurality of selected first devices 110 for a plurality of second devices 120 to locally train the hypemetwork.
  • the second device 120 may select some of the first devices from K online first devices 110, for example, may randomly select or preferably select K' clients u L 1 , ...
  • each selected first device u k obtains a local parameter of the hypemetwork by using the local training data 112. It should be noted that at this point, the first device 110 has not yet produced a personalized subnetwork 111, and the first device 110 uses the local training data 112 to train the hypemetwork determined at Block 610.
  • each selected first device u t k uses the local training data 112, and obtains a new hypemetwork parameter through S-step gradient descent, as shown in Equation (4).
  • ⁇ is a learning rate.
  • L is a cross-entropy loss function.
  • D t K , S is a random sample of a training set D L K .
  • the first device u k transmits the hypemetwork parameter generated in the current round of iteration to the second device 120.
  • Block 630 Receive local parameters of trained hypernetworks from the plurality of first devices 110.
  • Block 640 Update a global parameter of the hypemetwork.
  • the second device 120 updates a global parameter of the hypemetwork 121 based on the received local parameter of the client. For example, the second device 120 updates the global parameter according to the following Equation (5):
  • w ( t +1) represents a global parameter of the hypemetwork after t+1 iterations, and represents a local hypemetwork parameter generated by the first device 110 participating in a current iteration.
  • the function Update() may be a preset update algorithm, and in some embodiments, the global parameter is calculated based on a weighted average of amounts of training data, as shown in the following Equation (6).
  • n t k indicates a size of a training dataset of the client u k
  • an arithmetic average value of the parameters may be calculated as the global parameter of the hypemetwork.
  • Block 650 Determine whether T rounds are reached. If the T rounds are not reached, return to Block 620 to perform a next round of iteration. If the T rounds are reached: Block 660: Determine a current global parameter of the hypemetwork 121 as the preconfigured parameter of the hypemetwork 121, as shown in Equation (7).
  • ⁇ is the preconfigured parameter of the hypemetwork 121, which may be used by the first device 110 to generate the personalized subnetwork 111.
  • the second device 120 may select, based on performance levels of the first devices 110, K' first devices 110 to participate in iterations, to accelerate initialization of the hypemetwork 121.
  • the second device 120 After the global initialization of the hypemetwork is completed, the second device 120 maintains the hypemetwork 121 with the parameter ⁇ .
  • a method for personalizing a neural network model on a device is provided, which can simplify a structure of a hypemetwork 121 through pruning to generate a personalized subnetwork 111, and the generated personalized subnetwork meets data distribution and resource constraints of a client. The following provides detailed description with reference to FIG. 7 and FIG. 8 .
  • FIG. 7 is a flowchart of a process 700 for determining a structure of a subnetwork model according to an embodiment of the present disclosure.
  • the process 700 may be implemented in each device in K first devices 110.
  • the hypemetwork is simplified by iteratively deleting nodes and parameters of the hypemetwork 121.
  • the hypemetwork 121 is pruned based on accuracy of a network model. For example, a parameter that has little impact on accuracy may be selected according to a rule.
  • a resource constraint is also used as a hard constraint condition, and each pruning operation needs to delete at least one parameter to simplify the hypernetwork.
  • the hypemetwork 121 may be initialized by the plurality of first devices 110 and the second device 120 through the process described above with reference to FIG. 6 , and therefore high accuracy can be reached, to facilitate pruning to generate a personalized subnetwork model 111.
  • Block 710 Initialize a subnetwork model.
  • Embodiments of the present disclosure provide a layer-by-layer pruning method, which simplifies the structure of the hypemetwork to meet resource constraints, and ensures that an obtained subnetwork has high accuracy.
  • a pruning process of the hypemetwork is described with reference to MobileNetV2.
  • the pruning method provided in embodiments of the present disclosure is applicable to another type of neural network model.
  • FIG. 8 is a schematic diagram of pruning a hypemetwork according to an embodiment of the present disclosure.
  • MobileNetV2 is used as a hypemetwork, and a channel (neuron) pruning operation is performed, that is, pruning is performed by channel.
  • a channel (neuron) pruning operation is performed, that is, pruning is performed by channel.
  • some parts such as a depthwise layer, an input layer, and an output layer cannot be pruned, while 34 layers can be pruned.
  • prunable layers are L_1, ..., and L_34
  • a prunable layer L_i includes m_i channels C_1, ..., and C_(m_i).
  • the local data of the first device 110 may be in a 32x32x3 image format, to be specific, the input layer has three channels, corresponding to RGB values of the image.
  • a solid line arrow indicates a computation connection between adjacent layers.
  • a mark X indicates a pruned channel (node).
  • a dashed line arrow indicates a corresponding deleted connection.
  • a computation process of a channel that is not pruned remains unchanged.
  • Block 720 Adjust a structure of a plurality of layers of the subnetwork model to obtain a plurality of candidate neural network models. Specifically, for the plurality of prunable layers, parameters related to some nodes at each prunable layer of the subnetwork model 111 are deleted, to obtain the plurality of candidate network models. In some embodiments, for each layer, some nodes may be determined based on a predetermined quantity percentage (for example, 0.1, 0.2, and 0.25), and parameters related to these nodes may be deleted. Then, the local data of the first device may be used to determine accuracy of the obtained pruned candidate neural network model.
  • a predetermined quantity percentage for example, 0.1, 0.2, and 0.25
  • Block 730 Select one candidate network model based on accuracy of the plurality of candidate network models to update the subnetwork model.
  • Block 740 Determine whether a current subnetwork model meets resource constraints.
  • a calculation amount for example, floating-point operations (FLOPs), or a quantity of parameters of a model may be used as a resource constraint.
  • the first devices 110 may be classified as follows: (i) High-budget client: A FLOPs upper limit is 88 M, and a parameter quantity upper limit is 2.2 M. (ii) Medium-budget client: A FLOPs upper limit is 59 M, and a parameter quantity upper limit is 1.3 M. (iii) Low-budget client: A FLOPs upper limit is 28 M, and a parameter quantity upper limit is 0.7 M.
  • two federated learning settings may be considered: (1) High-performance configuration: A percentage of high-budget clients to medium-budget clients to low-budget clients is 5:3:2. (2) Low-performance configuration: A percentage of high-budget clients to medium-budget clients to low-budget clients is 2:3:5. This takes into consideration both the efficiency and accuracy of subsequent federated training.
  • the foregoing Blocks 720 and 730 are repeated to perform further pruning.
  • the above threshold regarding the calculation amount and the threshold regarding the quantiy of parameters depend on the performance of the first device 110. If the performance of the first device 110 is higher, the threshold may be set to a higher value, so that less pruning is performed on the subnetwork model 111. This means that higher accuracy can be achieved after training.
  • Block 750 Determine a structure of the current subnetwork model as a personalized subnetwork used by the first device 110 for federated training.
  • the structure of the subnetwork model 111 is adjusted iteratively, but a parameter value of the subnetwork model 111 remains unchanged, that is, a preconfigured parameter ⁇ (if not deleted) received from the second device 120 is maintained.
  • a process of determining the personalized subnetwork model 111 is described above.
  • the structure of the determined subnetwork model may be indicated by, for example, a mask, for federated training.
  • FIG. 9A, FIG. 9B , FIG. 9C, and FIG. 9D show test results according to some embodiments of the present disclosure.
  • the test results show that the solution (FedCS) according to embodiments of the present disclosure achieves excellent performance on classification tasks, and a personalized network architecture that has undergone federated learning and training has advantages.
  • the solution FebCS of the present disclosure has higher accuracy in both data IID and data NIID.
  • the advantages are more obvious.
  • a network architecture search framework (FedCS) of the present disclosure requires a significantly reduced amount of server-client traffic, which indicates a faster convergence speed and reduced communication costs.
  • the model obtained by searching has a smaller parameter quantity when the floating-point operations (FLOPs) constraint is equivalent, and therefore, storage space of a client is saved, as shown in FIG. 9C below.
  • FLOPs floating-point operations
  • the subnetwork federated training algorithm in the present disclosure greatly improves accuracy. It indicates that although client subnetworks have different architectures, subnetwork performance can be effectively improved by using a mask policy to perform federated training by using a hypemetwork.
  • FIG. 10 is a schematic block diagram of an apparatus 1000 for generating a neural network model according to some embodiments of the present disclosure.
  • the apparatus 1000 may be implemented in the first device 110 shown in FIG. 1 .
  • the apparatus 1000 includes a sending unit 1010.
  • the sending unit 1010 is configured to send an indication about a structure of a subnetwork model to a second device, where the subnetwork model is determined by adjusting a structure of a hypemetwork model.
  • the apparatus 1000 further includes a receiving unit 1020.
  • the receiving unit 1020 is configured to receive a parameter of the subnetwork model from the second device, where the parameter of the subnetwork model is determined by the second device based on the indication and the hypemetwork model.
  • the apparatus 1000 further includes a training unit 1030.
  • the training unit 1030 is configured to train the subnetwork model based on the received parameter of the subnetwork model.
  • the sending unit 1010 is further configured to send a parameter of the trained subnetwork model to the second device for the second device to update the hypemetwork model.
  • the receiving unit 1020 may be further configured to obtain a preconfigured parameter of the hypernetwork model.
  • the apparatus 1000 further includes a model determining unit 1040.
  • the model determining unit 1040 is configured to determine the structure of the subnetwork model based on the preconfigured parameter and by adjusting the structure of the hypemetwork model.
  • the training unit 1030 is configured to locally train the hypemetwork model to determine a local parameter of the hypemetwork model.
  • the sending unit 1010 is further configured to send the local parameter to the second device.
  • the receiving unit 1020 is further configured to receive the preconfigured parameter from the second device, where the preconfigured parameter is determined by the second device based on at least the local parameter received from a first device.
  • the model determining unit 1040 is further configured to initialize the subnetwork model to a hypemetwork model with the preconfigured parameter.
  • the model determining unit 1040 is further configured to iteratively update the subnetwork model by performing the following operations at least once: adjusting a structure of a plurality of layers of the subnetwork model to obtain a plurality of candidate network models; selecting one candidate network model based on accuracy of the plurality of candidate network models to update the subnetwork model; and if the subnetwork model meets a constraint of the first device, stopping the iterative update.
  • the model determining unit 1040 is further configured to delete parameters related to some nodes of one layer in the plurality of layers to obtain one of the plurality of candidate network models. In some embodiments, the model determining unit 1040 is configured to determine the some nodes based on a predetermined quantity percentage. In the foregoing manner: In some embodiments, a calculation amount of the subnetwork model is less than a first threshold, or a quantity of parameters of the subnetwork model is less than a second threshold.
  • the first threshold and the second threshold are both associated with performance of the first device.
  • the indication is in a form of a mask, and the mask indicates whether the subnetwork model has a corresponding parameter of the hypemetwork model.
  • the sending unit 1010 may be further configured to send a change in a parameter of the trained subnetwork model to the second device for the second device to update the hypemetwork model, where it may be determined that the change in the parameter is a difference between the parameter of the trained subnetwork model and the parameter received from the second device.
  • FIG. 11 is a schematic block diagram of an apparatus 1100 for generating a neural network model according to some embodiments of the present disclosure.
  • the apparatus 1100 may be implemented in the second device 120 shown in FIG. 1 .
  • the apparatus 1100 includes a receiving unit 1110.
  • the receiving unit 1110 is configured to receive an indication about a structure of a subnetwork model from a plurality of first devices, where the subnetwork model is determined by adjusting a structure of a hypemetwork model.
  • the apparatus 1100 further includes a unit 1120 for determining a parameter of a subnetwork model.
  • the unit 1120 for determining a parameter of a subnetwork model is configured to determine a parameter of the subnetwork model based on the indication and the hypemetwork model.
  • the apparatus 1100 further includes a sending unit 1130.
  • the sending unit 1130 is configured to send the parameter of the subnetwork model to the plurality of first devices for the plurality of first devices to separately train the subnetwork model.
  • the apparatus 1100 further includes a hypemetwork update unit 1140.
  • the hypemetwork update unit 1140 is configured to update the hypemetwork model by using the parameters that are of the trained subnetwork models and that are received from the plurality of first devices by using the receiving unit 1110.
  • the indication is in a form of a mask, and the mask indicates whether the subnetwork model has a corresponding parameter of the hypemetwork model.
  • the receiving unit 1110 may be further configured to receive, from the plurality of first devices, a change in the parameter of the trained subnetwork model.
  • the hypemetwork update unit 1140 may be further configured to update the hypemetwork model by using the received change in the parameter.
  • the hypemetwork update unit 1140 may be further configured to update the hypemetwork model based on an update weight of the parameter of the subnetwork model, where the update weight depends on a quantity of subnetwork models having the parameter.
  • the hypemetwork update unit 1140 may be further configured to determine a preconfigured parameter of the hypemetwork model for the plurality of first devices to determine respective subnetwork models from the hypemetwork model.
  • FIG. 12 is a block diagram of a computing device 1200 capable of implementing a plurality of embodiments of the present disclosure.
  • the device 1200 may be configured to implement the first device 110, the second device 120, the apparatus 1000, and the apparatus 1100.
  • the device 1200 includes a computing unit 1201 that may perform various appropriate actions and processing according to computer program instructions stored in a random access memory (RAM) and/or a read-only memory (ROM) 1202 or computer program instructions loaded from a storage unit 1207 into the RAM and/or ROM 1202.
  • RAM random access memory
  • ROM read-only memory
  • the RAM and/or ROM 1202 may further store various programs and data required for an operation of the device 1200.
  • the computing unit 1201 and the RAM and/or ROM 1202 are connected to each other by a bus 1203.
  • An input/output (I/O) interface 1204 is also connected to the bus 1203.
  • a plurality of components in the device 1200 are connected to the I/O interface 1204, and include: an input unit 1205, for example, a keyboard and a mouse; an output unit 1206, for example, various types of monitors and speakers; the storage unit 1107, for example, a magnetic disk or an optical disc; and a communication unit 1208, for example, a network card, a modem, or a wireless communication transceiver.
  • the communication unit 1208 allows the device 1200 to exchange information/data with other devices via a computer network such as the Internet and/or various telecommunications networks.
  • the computing unit 1101 may be various general-purpose and/or dedicated processing components that have processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like.
  • the computing unit 1201 performs any of the methods and processing described above.
  • the foregoing process may be implemented as a computer software program, and the computer software program is tangibly included in a machine-readable medium, for example, the storage unit 1207.
  • a part or all of the computer program may be loaded and/or installed on the device 1200 by using the RAM and/or a ROM and/or the communication unit 1208.
  • a computer program When a computer program is loaded into RAM and/or ROM and executed by the computing unit 1201, one or more steps in any of the processes described above may be performed.
  • the computing unit 1201 may be configured to perform any method and processing described above in any other appropriate manner (for example, through firmware).
  • the program code for implementing the methods of the present disclosure may be written using any combination of one or more programming languages.
  • the program code may be provided to a processor or controller of a general purpose computer, a special purpose computer, or another programmable data processing apparatus, such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented.
  • the program code may be fully executed on a machine, partially executed on a machine, partially executed on a machine as a stand-alone software package and partially executed on a remote machine or fully executed on a remote machine or server.
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus or device.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • the machine-readable medium may include, but is not limited to, an electronic system, apparatus, or device, a magnetic system, apparatus, or device, an optical system, apparatus, or device, an electromagnetic system, apparatus, or device, an infrared system, apparatus, or device, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.
  • machine-readable storage medium include one or more wire-based electrical connections, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
  • RAMs random access memories
  • ROMs read-only memories
  • EPROMs or flash memories erasable programmable read-only memories
  • CD-ROMs compact disc read-only memories
  • magnetic storage devices or any suitable combination of the foregoing.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computer And Data Communications (AREA)
EP22827695.2A 2021-06-24 2022-06-24 Procédé, appareil et système pour générer un modèle de réseau neuronal, dispositifs, support et produit-programme Pending EP4350572A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110704382.6A CN113570027A (zh) 2021-06-24 2021-06-24 用于生成神经网络模型的方法、装置、系统、设备、介质和程序产品
PCT/CN2022/101128 WO2022268203A1 (fr) 2021-06-24 2022-06-24 Procédé, appareil et système pour générer un modèle de réseau neuronal, dispositifs, support et produit-programme

Publications (1)

Publication Number Publication Date
EP4350572A1 true EP4350572A1 (fr) 2024-04-10

Family

ID=78162609

Family Applications (1)

Application Number Title Priority Date Filing Date
EP22827695.2A Pending EP4350572A1 (fr) 2021-06-24 2022-06-24 Procédé, appareil et système pour générer un modèle de réseau neuronal, dispositifs, support et produit-programme

Country Status (4)

Country Link
US (1) US20240135191A1 (fr)
EP (1) EP4350572A1 (fr)
CN (1) CN113570027A (fr)
WO (1) WO2022268203A1 (fr)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113570027A (zh) * 2021-06-24 2021-10-29 华为技术有限公司 用于生成神经网络模型的方法、装置、系统、设备、介质和程序产品
CN115086399B (zh) * 2022-07-28 2022-12-06 深圳前海环融联易信息科技服务有限公司 基于超网络的联邦学习方法、装置及计算机设备
CN115017377B (zh) * 2022-08-05 2022-11-08 深圳比特微电子科技有限公司 用于搜索目标模型的方法、装置和计算设备
CN115099393B (zh) * 2022-08-22 2023-04-07 荣耀终端有限公司 神经网络结构搜索方法及相关装置
CN115860135B (zh) * 2022-11-16 2023-08-01 中国人民解放军总医院 基于超网的异构联邦学习方法、设备及介质
CN116070714B (zh) * 2023-01-10 2024-03-08 天津大学 基于联邦学习和神经架构搜索的云边协同训练方法及系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11593642B2 (en) * 2019-09-30 2023-02-28 International Business Machines Corporation Combined data pre-process and architecture search for deep learning models
CN111340242B (zh) * 2020-05-15 2020-08-04 支付宝(杭州)信息技术有限公司 保护隐私的模型联合训练方法及装置
CN112818207A (zh) * 2021-02-26 2021-05-18 深圳前海微众银行股份有限公司 网络结构搜索方法、装置、设备、存储介质及程序产品
CN113570027A (zh) * 2021-06-24 2021-10-29 华为技术有限公司 用于生成神经网络模型的方法、装置、系统、设备、介质和程序产品

Also Published As

Publication number Publication date
WO2022268203A1 (fr) 2022-12-29
US20240135191A1 (en) 2024-04-25
CN113570027A (zh) 2021-10-29

Similar Documents

Publication Publication Date Title
EP4350572A1 (fr) Procédé, appareil et système pour générer un modèle de réseau neuronal, dispositifs, support et produit-programme
CN111461226A (zh) 对抗样本生成方法、装置、终端及可读存储介质
WO2019111118A1 (fr) Schémas de compression de poids de gradient robuste pour des applications d'apprentissage profond
WO2018153806A1 (fr) Formation de modèles d'apprentissage automatique
CN111932386B (zh) 用户账号确定方法及装置、信息推送方法及装置、电子设备
US11423307B2 (en) Taxonomy construction via graph-based cross-domain knowledge transfer
CN111611085B (zh) 基于云边协同的人机混合增强智能系统、方法、装置
CN116523079A (zh) 一种基于强化学习联邦学习优化方法及系统
CN114065863A (zh) 联邦学习的方法、装置、系统、电子设备及存储介质
CN112560985A (zh) 神经网络的搜索方法、装置及电子设备
CN113627536A (zh) 模型训练、视频分类方法，装置，设备以及存储介质
CN111090740B (zh) 一种用于对话系统的知识图谱生成方法
JP7361121B2 (ja) デュアル・ネットワークと共に訓練された主ネットワークを介した多目的タスクの実行
CN113657468A (zh) 预训练模型的生成方法、装置、电子设备和存储介质
CN111957053A (zh) 游戏玩家匹配方法、装置、存储介质与电子设备
CN115186738B (zh) 模型训练方法、装置和存储介质
CN117033997A (zh) 数据切分方法、装置、电子设备和介质
CN115292044A (zh) 数据处理方法、装置、电子设备及存储介质
CN114118358A (zh) 图像处理方法、装置、电子设备、介质及程序产品
CN112927810A (zh) 基于大数据的智慧医疗响应方法及智慧医疗云计算系统
US11861490B1 (en) Decoupled machine learning training
US20220300852A1 (en) Method and System for Automating Scenario Planning
CN113836438B (zh) 用于帖子推荐的方法、电子设备和存储介质
CN115827526B (zh) 数据处理方法、装置、设备以及存储介质
CN113572679B (zh) 账户亲密度的生成方法、装置、电子设备和存储介质

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20231220

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR