CN112784961A - Training method and device for hyper network, electronic equipment and storage medium - Google Patents
Training method and device for hyper network, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112784961A CN112784961A CN202110080416.9A CN202110080416A CN112784961A CN 112784961 A CN112784961 A CN 112784961A CN 202110080416 A CN202110080416 A CN 202110080416A CN 112784961 A CN112784961 A CN 112784961A
- Authority
- CN
- China
- Prior art keywords
- network
- hyper
- sub
- parameters
- probability model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Multimedia (AREA)
- Computer And Data Communications (AREA)
Abstract
The disclosure provides a training method and device for a hyper-network, electronic equipment and a storage medium, and relates to the field of artificial intelligence, in particular to the fields of computer vision, deep learning and the like. The specific implementation scheme is as follows: carrying out N times of iteration operation based on a preset hyper network to obtain a target hyper network; wherein, the ith iteration operation in the N iterations operations comprises: selecting m sub-networks in a search space of a preset hyper-network using the i-1 th group of hyper-network parameters; evaluating the m sub-networks based on the (i-1) th group of super-network parameters to obtain an ith probability model; obtaining an ith group of hyper-network parameters based on mutual information between the hyper-parameters of the (i-1) th probability model and the hyper-parameters of the ith probability model; and under the condition that i is equal to the preset threshold value N, taking the preset hyper network using the ith group of hyper network parameters as a target hyper network. According to the scheme of the embodiment of the disclosure, the performance of the searched sub-network can be improved.
Description
Technical Field
The present disclosure relates to the field of artificial intelligence, and more particularly to the field of computer vision and deep learning.
Background
The principle of NAS (Neural Architecture Search) is to Search for an optimal network structure from a set of Neural network structures, called a Search space, based on a Search strategy. Early NAS searches were very inefficient and required very large resource consumption. Therefore, the NAS method based on parameter sharing is receiving attention because of its high search efficiency. In the NAS method based on parameter sharing, a super network is trained, and parameters of the super network are shared by all network structures in a search space corresponding to the super network, namely sub-networks of the super network. Thus, the performance of the super network affects the performance of each sub-network.
Disclosure of Invention
The disclosure provides a training method and device of a hyper-network, electronic equipment and a storage medium.
According to an aspect of the present disclosure, there is provided a training method of a super network, including:
carrying out N times of iteration operation based on a preset hyper network to obtain a target hyper network; wherein N is an integer greater than or equal to 2;
wherein, the ith iteration operation in the N iterations operations comprises:
selecting m sub-networks from a search space of a preset hyper-network using the i-1 th group of hyper-network parameters by using the i-1 th probability model; wherein i is a positive integer less than or equal to N, and m is a positive integer;
evaluating the m sub-networks based on the i-1 group of super-network parameters to obtain performance information of the m sub-networks;
obtaining an ith probability model based on the performance information of the m sub-networks;
obtaining an ith group of hyper-network parameters based on mutual information between the hyper-parameters of the (i-1) th probability model and the hyper-parameters of the ith probability model;
and under the condition that i is equal to the preset threshold value N, taking the preset hyper network using the ith group of hyper network parameters as a target hyper network.
According to another aspect of the present disclosure, there is provided a training apparatus of a super network, including:
the iteration module is used for carrying out N times of iteration operations based on a preset hyper network to obtain a target hyper network; wherein N is an integer greater than or equal to 2;
wherein the iteration module comprises:
the sub-network unit is used for selecting m sub-networks in a search space of a preset super-network using the i-1 th group of super-network parameters by using the i-1 th probability model; wherein i is a positive integer less than or equal to N, and m is a positive integer;
the performance information unit is used for evaluating the m sub-networks based on the i-1 group of super-network parameters to obtain the performance information of the m sub-networks;
the probability model unit is used for obtaining an ith probability model based on the performance information of the m sub-networks;
the hyper-network parameter unit is used for obtaining an ith group of hyper-network parameters based on mutual information between the hyper-parameters of the (i-1) th probability model and the hyper-parameters of the (i) th probability model;
and the target hyper-network unit is used for taking the preset hyper-network using the ith group of hyper-network parameters as the target hyper-network under the condition that i is equal to the preset threshold value N.
According to another aspect of the present disclosure, there is provided an electronic device including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method according to any one of the embodiments of the present disclosure.
According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform a method in any of the embodiments of the present disclosure.
According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method in any of the embodiments of the present disclosure.
According to the technology disclosed by the invention, the probability model and the hyper-network are mutually influenced, positive feedback can be formed, and the performance of the hyper-network is gradually improved. Moreover, because the hyper-network parameters are updated based on the mutual information before and after the probability model is updated, and then the probability model is updated, the hyper-network parameters and the probability model can be updated with the maximum mutual information as a target, so that the sampled sub-networks can better represent each sub-network of the hyper-network, and the consistency between the performance of each sub-network sharing the hyper-network parameters and the performance of the sub-network obtained by training independently is higher. The performance of the searched sub-network is improved while the searching efficiency of the neural network is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
FIG. 1 is a schematic diagram of a training method for a hyper-network provided by one embodiment of the present disclosure;
FIG. 2 is a schematic diagram of a training method for a hyper-network provided by another embodiment of the present disclosure;
FIG. 3 is a schematic diagram of a training method for a hyper-network provided by yet another embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a training apparatus for a super network provided by one embodiment of the present disclosure;
FIG. 5 is a schematic diagram of a training apparatus for a super network provided by another embodiment of the present disclosure;
FIG. 6 is a block diagram of an electronic device for implementing a training method for a hyper-network of an embodiment of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram illustrating a training method of a super network according to an embodiment of the present disclosure. As shown in fig. 1, the method includes:
step S110, carrying out N times of iteration operations based on a preset hyper network to obtain a target hyper network; wherein N is an integer greater than or equal to 2;
wherein, the ith iteration operation in the N iterations operations comprises:
s111, selecting m sub-networks from a search space of a preset super-network using the i-1 group of super-network parameters by using the i-1 th probability model; wherein i is a positive integer less than or equal to N, and m is a positive integer;
step S112, evaluating the m sub-networks based on the i-1 group of super-network parameters to obtain performance information of the m sub-networks;
step S113, obtaining an ith probability model based on the performance information of the m sub-networks;
step S114, obtaining an ith group of hyper-network parameters based on mutual information between the hyper-parameters of the (i-1) th probability model and the hyper-parameters of the (i) th probability model;
and step S115, taking the preset hyper network using the ith group of hyper network parameters as a target hyper network under the condition that i is equal to the preset threshold value N.
Illustratively, the initial hyper-network parameters, or set 0 hyper-network parameters, may be the hyper-network parameters corresponding to the initial search space. In one example, an initial search space, such as a MobileNet-like search space or a ResNet-like search space, may be predetermined. The initial search space includes a plurality of model structures or subnetworks. Based on the training data, forward propagation and backward propagation are respectively performed on all or part of the sub-networks in the initial search space, for example, on a plurality of sub-networks obtained by random sampling, so as to obtain initial hyper-network parameters. The super-network can be determined based on the super-network parameters and each sub-network in the search space of the super-network can be evaluated to obtain the performance information of the sub-network.
Illustratively, the probabilistic model may characterize the relationship of model structure and model performance. In one example, the type of the probability model may be predetermined, for example, a multidimensional gaussian distribution model is used as the probability model. Probabilistic models can be used to sample sub-networks. In the 1 st iteration, m subnetworks may be randomly sampled using the initial probability model, i.e., the 0 th probability model. Since the hyper-parameters of the probabilistic model are updated on the basis of m sub-networks in each iteration operation, the probabilistic model samples the m sub-networks on the basis of the updated hyper-parameters in each iteration operation 2 and thereafter. For example, the probability model may predict each sub-network based on the updated hyper-parameters to obtain the predicted performance of each sub-network, and select m sub-networks in the search space according to the predicted performance of each sub-network. Alternatively, the probability model may select m subnetworks according to a certain principle, such as an entropy maximization principle, based on the updated maximum likelihood estimation value of the hyper-parameter.
In the embodiment of the disclosure, a sub-network is sampled by using a probability model, the sub-network obtained by sampling is evaluated based on a super-network parameter, the probability model is updated according to performance information of the sub-network obtained by evaluation, and then the super-network parameter is updated based on mutual information between super-parameters of the probability model before and after updating, namely, the super-network parameter is updated based on mutual information between the super-parameter of the (i-1) th probability model and the super-parameter of the (i) th probability model, so that the (i) th group of super-network parameters is obtained. The mutual information may refer to an amount of information related to a hyper-parameter of the probability model before update (hyper-parameter of the i-th probability model) included in the hyper-parameter of the probability model after update (hyper-parameter of the i-th probability model). Based on the mutual information, the amount of change of the hyper-parameters of the probabilistic model in the updating process can be determined.
Since the hyper-network parameters are updated according to the mutual information of the probabilistic model before and after the update, the hyper-network parameters are updated subsequently as the probabilistic model is updated. If the accuracy of the super-network parameters is improved, the sub-network sampled by the probability model based on the super-network parameters is more accurately evaluated, the accuracy of the updated probability model is also improved, and then the super-network parameters are updated by utilizing the mutual information before and after the probability model is updated, so that the super-network parameters with higher accuracy can be obtained. And after a plurality of rounds of updating, if the updating times i of the super network reach a preset threshold value N, outputting the target super network. The preset threshold N may be 200 times, 500 times, etc.
Therefore, according to the method disclosed by the embodiment of the disclosure, the probability model and the super network are mutually influenced, the positive feedback can be formed, and the performance of the super network is gradually improved. Moreover, because the hyper-network parameters are updated based on the mutual information before and after the probability model is updated, and then the probability model is updated, the hyper-network parameters and the probability model can be updated with the maximum mutual information as a target, so that the sampled sub-networks can better represent each sub-network of the hyper-network, and the consistency between the performance of each sub-network sharing the hyper-network parameters and the performance of the sub-network obtained by training independently is higher. The performance of the searched sub-network is improved while the searching efficiency of the neural network is improved. Based on this, the searched sub-network is applied to the scene of image processing (such as image classification, image recognition) and the like, and the speed, the precision and the like of the image processing can be improved. Better hardware performance may be achieved using the searched subnets on specific hardware, such as higher processing accuracy on processors with limited operating speed, or faster operation on processors with limited processing accuracy, etc. In some scenarios, hardware cost may also be reduced, for example, hardware performance may be achieved with less hardware than before, and hardware cost may be reduced by reducing the amount of hardware used.
In an exemplary embodiment, in step S111, selecting m subnetworks in the search space of the preset piconet using the i-1 th set of piconet parameters by using the i-1 th probabilistic model includes:
and selecting m sub-networks which maximize the information entropy from the search space of the preset hyper-network using the i-1 group of hyper-network parameters by using the i-1 probability model.
That is, the information entropy gains of all sub-networks in the search space of the current preset hyper-network are evaluated by using the (i-1) th probability model, and the m sub-networks with the largest information entropy gains are selected from the search space. Illustratively, m sub-networks can be sampled according to the principle of information entropy maximization based on the maximum likelihood estimated value of the hyper-parameter of the (i-1) th probability model.
According to the exemplary embodiment, the ith probability model obtained based on m sub-networks can be made to be more consistent with the actual situation, and the consistency between the performance of each sub-network sharing the parameters of the super-network and the performance of the sub-network obtained by training alone is higher.
In an exemplary embodiment, in step S113, obtaining an ith probability model based on the performance information of the m subnetworks, includes:
performing maximum likelihood estimation based on the performance information of the m sub-networks to obtain an ith group of probability model hyperparameters;
and obtaining an ith probability model based on the hyper-parameters of the ith group of probability models.
As an example, the maximum likelihood estimation may be directly performed based on the performance information of the m subnetworks to obtain the i-th group of probability model hyper-parameters. As another example, the corresponding loss function may be obtained based on the performance information of the m subnetworks, and the maximum likelihood estimation may be performed based on the loss functions of the m subnetworks to obtain the i-th group of probability model hyper-parameters.
The maximum likelihood estimation is a method for more accurately estimating model parameters aiming at the condition that the model structure is determined and the parameters are unknown. By adopting a maximum likelihood estimation mode, the accuracy of the probability model hyper-parameters can be improved, the accuracy of the probability model and the hyper-network parameters is further improved, and the performance of the hyper-network and the sub-network is further improved.
In the embodiment of the disclosure, mutual information maximization can be taken as an updating target, and the hyper-network parameters and the probability model are updated. In an exemplary embodiment, in step S114, obtaining an ith group of hyper-network parameters based on mutual information between the hyper-parameter of the i-1 th probabilistic model and the hyper-parameter of the ith probabilistic model, includes:
taking the reciprocal of mutual information between the hyperparameter of the (i-1) th probability model and the hyperparameter of the (i) th probability model as a loss function;
and updating the (i-1) th group of super-network parameters based on the loss function to obtain the (i) th group of super-network parameters.
According to the above embodiment, the larger the mutual information, the smaller the reciprocal thereof, the smaller the loss function. For example, if the mutual information is 2, the loss function is 1/2; the mutual information is 3, the loss function is 1/3.
Since the loss function is smaller by using the loss function update model or parameters, according to the above embodiment, after the super-network parameters and the probability model are updated, mutual information is larger, and consistency between the performance of each sub-network sharing the super-network parameters and the performance of the sub-network obtained by training alone can be higher.
After the target super network is obtained through training, a target sub-network can be selected from a search space of the target super network based on a preset algorithm, so that the target sub-network is applied to application scenes such as image processing and the like. The preset algorithm may be a genetic algorithm, etc. In an exemplary embodiment, the target sub-network may also be selected with reference to the steps shown in fig. 2. As shown in fig. 2, the method may further include:
step S210, selecting k sub-networks in the search space of the target super-network by using the Nth probability model; wherein k is a positive integer;
step S220, evaluating k sub-networks based on the super-network parameters of the target super-network to obtain performance information of the k sub-networks;
in step S230, the subnet corresponding to the maximum value among the performance information of k subnets is determined as the target subnet.
For example, in step S210, L sub-networks may be randomly sampled based on the target super-network, where the number L of sampled sub-networks is an integer greater than or equal to k. And predicting the L sub-networks by using the Nth probability model, namely the latest probability model, to obtain the prediction performances of the L sub-networks, and selecting k sub-networks from the L sub-networks according to the prediction performances of the L sub-networks, for example, the k sub-networks with the best prediction performance. And then evaluating the k sub-networks based on the hyper-network parameters of the target hyper-network, and obtaining the target sub-network according to the evaluation result.
According to the embodiment, the sub-networks can be predicted and evaluated respectively based on the probability model and the super-network parameters of the target super-network, so that the sub-network with the optimal performance is selected, and the hardware performance of the target sub-network in a specific application scene is favorably improved, for example, higher processing precision can be obtained on a processor with limited operation speed, or the hardware can be operated faster on a processor with limited processing precision, and the like.
In an exemplary embodiment, as shown in fig. 3, the method further includes:
step S310, acquiring an image to be processed;
and step S320, processing the image to be processed by using the target sub-network to obtain an image processing result of the image to be processed.
The image to be processed may be, for example, an image to be recognized or classified. And processing the image to be processed by using the target sub-network to obtain an image processing result, including an identification result or a classification result of the image to be processed. The recognition result may include a target detection result, a semantic segmentation result, and the like.
According to the embodiment, the target sub-network obtained based on the hyper-network search is applied to the image processing scene, so that the processing precision and/or speed can be improved.
It should be understood that the application scope of the present disclosure is not limited to the above image processing field, but may also be applied to artificial intelligence fields such as natural language processing, cloud computing, and the like. Illustratively, PaddleSlim, Paddlecloud for cloud computing, easyll for image recognition, and the like, may be applicable to model compression. According to the method disclosed by the embodiment of the invention, the probability model and the super network are mutually influenced, the positive feedback can be formed, and the performance of the super network is gradually improved. Moreover, because the hyper-network parameters are updated based on the mutual information before and after the probability model is updated, and then the probability model is updated, the hyper-network parameters and the probability model can be updated with the maximum mutual information as a target, so that the sampled sub-networks can better represent each sub-network of the hyper-network, and the consistency between the performance of each sub-network sharing the hyper-network parameters and the performance of the sub-network obtained by training independently is higher. The performance of the searched sub-network is improved while the searching efficiency of the neural network is improved.
As the realization of the above methods, the present disclosure also provides a training device of the super network. Fig. 4 is a schematic diagram of a training apparatus of a super network according to an embodiment of the present disclosure. As shown in fig. 4, the apparatus includes:
an iteration module 410, configured to perform N iteration operations based on a preset hyper network to obtain a target hyper network; wherein N is an integer greater than or equal to 2;
wherein the iteration module 410 includes:
a sub-network unit 411, configured to select m sub-networks from a search space of a preset super-network using the i-1 th group of super-network parameters by using the i-1 th probabilistic model; wherein i is a positive integer less than or equal to N, and m is a positive integer;
a performance information unit 412, configured to evaluate the m subnetworks based on the i-1 th group of hyper-network parameters to obtain performance information of the m subnetworks;
a probability model unit 413, configured to obtain an ith probability model based on the performance information of the m subnetworks;
a hyper-network parameter unit 414, configured to obtain an ith group of hyper-network parameters based on mutual information between a hyper-parameter of the i-1 th probabilistic model and a hyper-parameter of the i-th probabilistic model;
a target hyper-network unit 415, configured to take the preset hyper-network using the ith set of hyper-network parameters as the target hyper-network, if i is equal to the preset threshold N.
Illustratively, the performance information unit 412 is configured to:
and selecting m sub-networks which maximize the information entropy from the search space of the preset hyper-network using the i-1 group of hyper-network parameters by using the i-1 probability model.
Exemplarily, the probabilistic model unit 413 is configured to:
performing maximum likelihood estimation based on the performance information of the m sub-networks to obtain an ith group of probability model hyperparameters;
and obtaining an ith probability model based on the hyper-parameters of the ith group of probability models.
Illustratively, the super network parameters unit 414 is configured to:
taking the reciprocal of mutual information between the hyperparameter of the (i-1) th probability model and the hyperparameter of the (i) th probability model as a loss function;
and updating the (i-1) th group of super-network parameters based on the loss function to obtain the (i) th group of super-network parameters.
Illustratively, as shown in fig. 5, the training apparatus of the super network further includes:
a sub-network selecting module 510, configured to select k sub-networks from the search space of the target super-network by using the nth probability model; wherein k is a positive integer;
a performance evaluation module 520, configured to evaluate k subnetworks based on the hyper-network parameters of the target hyper-network to obtain performance information of the k subnetworks;
and a target determining module 530, configured to determine the sub-network corresponding to the maximum value in the performance information of the k sub-networks as a target sub-network.
As shown in fig. 5, the apparatus further includes:
an image obtaining module 540, configured to obtain an image to be processed;
and an image processing module 550, configured to process the image to be processed by using the target subnetwork, so as to obtain an image processing result of the image to be processed.
The training method of the super network provided by the embodiment of the disclosure can realize the training device of the super network provided by the embodiment of the disclosure, and has corresponding beneficial effects.
The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.
FIG. 6 illustrates a schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 6, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 can also be stored. The calculation unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Various components in the electronic device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.
The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 601 performs the various methods and processes described above, such as the training method of the hyper-network. For example, in some embodiments, the training method of the hypernetwork can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the training method of the hypernetwork described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured by any other suitable means (e.g., by means of firmware) to perform the training method of the hyper-network.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel or sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.
The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.
Claims (15)
1. A training method of a super network, comprising:
carrying out N times of iteration operation based on a preset hyper network to obtain a target hyper network; wherein N is an integer greater than or equal to 2;
wherein the ith iteration operation of the N iterations operations comprises:
selecting m sub-networks from a search space of a preset hyper-network using the i-1 th group of hyper-network parameters by using the i-1 th probability model; wherein i is a positive integer less than or equal to N, and m is a positive integer;
evaluating the m sub-networks based on the i-1 group of hyper-network parameters to obtain performance information of the m sub-networks;
obtaining an ith probability model based on the performance information of the m sub-networks;
obtaining an ith group of hyper-network parameters based on mutual information between the hyper-parameters of the (i-1) th probability model and the hyper-parameters of the (i) th probability model;
and taking the preset hyper-network using the ith group of hyper-network parameters as the target hyper-network under the condition that i is equal to a preset threshold value N.
2. The method of claim 1, wherein said selecting m subnetworks in the search space of the pre-set hyper-network using the i-1 st set of hyper-network parameters using the i-1 st probabilistic model comprises:
and selecting m sub-networks which enable the information entropy to be maximized in the search space of the preset super-network using the i-1 th group of super-network parameters by using the i-1 th probability model.
3. The method of claim 1, wherein the deriving an ith probability model based on the performance information of the m subnetworks comprises:
performing maximum likelihood estimation based on the performance information of the m sub-networks to obtain an ith group of probability model hyperparameters;
and obtaining the ith probability model based on the hyper-parameters of the ith group of probability models.
4. The method according to claim 1, wherein the obtaining of the ith set of hyper-network parameters based on mutual information between the hyper-parameters of the (i-1) th probabilistic model and the hyper-parameters of the (i) th probabilistic model comprises:
taking the reciprocal of mutual information between the hyperparameter of the i-1 th probability model and the hyperparameter of the i-th probability model as a loss function;
and updating the i-1 th group of hyper-network parameters based on the loss function to obtain the i-th group of hyper-network parameters.
5. The method of any of claims 1-4, further comprising:
selecting k sub-networks in the search space of the target super-network by using the Nth probability model; wherein k is a positive integer;
evaluating the k sub-networks based on the hyper-network parameters of the target hyper-network to obtain performance information of the k sub-networks;
and determining the sub-network corresponding to the maximum value in the performance information of the k sub-networks as a target sub-network.
6. The method of claim 5, further comprising:
acquiring an image to be processed;
and processing the image to be processed by using the target sub-network to obtain an image processing result of the image to be processed.
7. A training apparatus for a super network, comprising:
the iteration module is used for carrying out N times of iteration operations based on a preset hyper network to obtain a target hyper network; wherein N is an integer greater than or equal to 2;
wherein the iteration module comprises:
the sub-network unit is used for selecting m sub-networks in a search space of a preset super-network using the i-1 th group of super-network parameters by using the i-1 th probability model; wherein i is a positive integer less than or equal to N, and m is a positive integer;
a performance information unit, configured to evaluate the m subnetworks based on the i-1 th group of hyper-network parameters to obtain performance information of the m subnetworks;
a probability model unit, configured to obtain an ith probability model based on the performance information of the m subnetworks;
a hyper-network parameter unit, configured to obtain an i-th group of hyper-network parameters based on mutual information between a hyper-parameter of the i-1 th probabilistic model and a hyper-parameter of the i-th probabilistic model;
and the target hyper-network unit is used for taking the preset hyper-network using the ith group of hyper-network parameters as the target hyper-network under the condition that i is equal to a preset threshold value N.
8. The apparatus of claim 7, wherein the performance information unit is to:
and selecting m sub-networks which enable the information entropy to be maximized in the search space of the preset super-network using the i-1 th group of super-network parameters by using the i-1 th probability model.
9. The apparatus of claim 7, wherein the probabilistic model unit is to:
performing maximum likelihood estimation based on the performance information of the m sub-networks to obtain an ith group of probability model hyperparameters;
and obtaining the ith probability model based on the hyper-parameters of the ith group of probability models.
10. The apparatus of claim 7, wherein the hyper-network parameter unit is to:
taking the reciprocal of mutual information between the hyperparameter of the i-1 th probability model and the hyperparameter of the i-th probability model as a loss function;
and updating the i-1 th group of hyper-network parameters based on the loss function to obtain the i-th group of hyper-network parameters.
11. The apparatus of any of claims 7-10, further comprising:
a sub-network selection module, configured to select k sub-networks from the search space of the target super-network by using the nth probability model; wherein k is a positive integer;
the performance evaluation module is used for evaluating the k sub-networks based on the super-network parameters of the target super-network to obtain performance information of the k sub-networks;
and the target determining module is used for determining the sub-network corresponding to the maximum value in the performance information of the k sub-networks as a target sub-network.
12. The apparatus of claim 11, further comprising:
the image acquisition module is used for acquiring an image to be processed;
and the image processing module is used for processing the image to be processed by utilizing the target sub-network to obtain an image processing result of the image to be processed.
13. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.
14. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any one of claims 1-6.
15. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110080416.9A CN112784961A (en) | 2021-01-21 | 2021-01-21 | Training method and device for hyper network, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110080416.9A CN112784961A (en) | 2021-01-21 | 2021-01-21 | Training method and device for hyper network, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112784961A true CN112784961A (en) | 2021-05-11 |
Family
ID=75757692
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110080416.9A Pending CN112784961A (en) | 2021-01-21 | 2021-01-21 | Training method and device for hyper network, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112784961A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113657465A (en) * | 2021-07-29 | 2021-11-16 | 北京百度网讯科技有限公司 | Pre-training model generation method and device, electronic equipment and storage medium |
| CN113657468A (en) * | 2021-07-29 | 2021-11-16 | 北京百度网讯科技有限公司 | Pre-training model generation method and device, electronic equipment and storage medium |
| CN114707471A (en) * | 2022-06-06 | 2022-07-05 | 浙江大学 | Artificial intelligent courseware making method and device based on hyper-parameter evaluation graph algorithm |
| WO2023056802A1 (en) * | 2021-10-08 | 2023-04-13 | 上海交通大学 | Image classification method for maximizing mutual information, and device, medium and system |
-
2021
- 2021-01-21 CN CN202110080416.9A patent/CN112784961A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113657465A (en) * | 2021-07-29 | 2021-11-16 | 北京百度网讯科技有限公司 | Pre-training model generation method and device, electronic equipment and storage medium |
| CN113657468A (en) * | 2021-07-29 | 2021-11-16 | 北京百度网讯科技有限公司 | Pre-training model generation method and device, electronic equipment and storage medium |
| CN113657465B (en) * | 2021-07-29 | 2024-04-09 | 北京百度网讯科技有限公司 | Pre-training model generation method and device, electronic equipment and storage medium |
| WO2023056802A1 (en) * | 2021-10-08 | 2023-04-13 | 上海交通大学 | Image classification method for maximizing mutual information, and device, medium and system |
| CN114707471A (en) * | 2022-06-06 | 2022-07-05 | 浙江大学 | Artificial intelligent courseware making method and device based on hyper-parameter evaluation graph algorithm |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112784961A (en) | Training method and device for hyper network, electronic equipment and storage medium | |
| CN113657465B (en) | Pre-training model generation method and device, electronic equipment and storage medium | |
| CN111582454B (en) | Method and device for generating neural network model | |
| JP2018536920A (en) | Text information processing method and device | |
| CN112560996A (en) | User portrait recognition model training method, device, readable storage medium and product | |
| CN111368973A (en) | Method and apparatus for training a hyper-network | |
| CN114936323B (en) | Training method and device of graph representation model and electronic equipment | |
| CN114818913A (en) | Decision generation method and device | |
| CN113657468A (en) | Pre-training model generation method and device, electronic equipment and storage medium | |
| CN112784962B (en) | Super-network training method and device, electronic equipment and storage medium | |
| CN115329748B (en) | Log analysis method, device, equipment and storage medium | |
| CN113012682B (en) | False wake-up rate determination method, device, apparatus, storage medium, and program product | |
| CN115664814A (en) | Network intrusion detection method and device, electronic equipment and storage medium | |
| CN113313049A (en) | Method, device, equipment, storage medium and computer program product for determining hyper-parameters | |
| CN111563591B (en) | Super network training method and device | |
| CN114610953A (en) | Data classification method, device, equipment and storage medium | |
| CN113920404A (en) | Training method, image processing method, device, electronic device and storage medium | |
| CN114120180A (en) | Method, device, equipment and medium for generating time sequence nomination | |
| CN113590774A (en) | Event query method, device and storage medium | |
| CN113010571A (en) | Data detection method, data detection device, electronic equipment, storage medium and program product | |
| CN112905885A (en) | Method, apparatus, device, medium, and program product for recommending resources to a user | |
| CN112784165A (en) | Training method of incidence relation estimation model and method for estimating file popularity | |
| CN111667028A (en) | Reliable negative sample determination method and related device | |
| CN114037057B (en) | Pre-training model generation method and device, electronic equipment and storage medium | |
| CN112528123A (en) | Model searching method, model searching apparatus, electronic device, storage medium, and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |