CN112069370A - Neural network structure search method, device, medium and device - Google Patents

Abstract

A neural network structure search method, device, medium and device are disclosed. The method for searching a neural network structure includes: acquiring a first neural network including a plurality of blocks with different numbers of channels, wherein at least one block of the first neural network is connected to at least three blocks, and at least one block includes a block for executing The first layer of block channel number and spatial resolution conversion; according to the sample data in the first data set, using a gradient-based search strategy, the first neural network is subjected to neural network structure search processing, and the first neural network is obtained. The structure parameters of the first neural network are determined; according to the structure parameters of the first neural network, the structure of the second neural network obtained by the search is determined. The technical solution provided by the present disclosure is beneficial to improve the flexibility of the neural network structure search, thereby improving the performance and diversity of the neural network obtained by the search.

Description

Neural network structure search method, device, medium and device

technical field

The present disclosure relates to computer vision technology, in particular to a neural network structure search method, a neural network structure search device, a storage medium, and an electronic device.

Background technique

In the field of computer vision technology, designing the structure of a neural network and adjusting the parameters of the neural network with a well-designed structure often requires a lot of labor costs, computing power costs, and time costs.

How to conveniently design the structure of neural network is a technical issue worthy of attention.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned technical problems, the present disclosure is made. Embodiments of the present disclosure provide a neural network structure search method, a neural network structure search apparatus, a storage medium, and an electronic device.

According to an aspect of the embodiments of the present disclosure, a method for searching a neural network structure is provided, including: acquiring a first neural network including a plurality of blocks with different numbers of channels, wherein at least one block in the first neural network is related to at least one block in the first neural network. Three blocks are connected, and at least one block includes a head layer for performing block channel number and spatial resolution conversion; according to the sample data in the first data set, a gradient-based search strategy is used to perform a neural network on the first neural network. In the structure search process, the structure parameters of the first neural network are obtained; the second neural network structure obtained by searching is determined according to the structure parameters of the first neural network.

According to another aspect of the embodiments of the present disclosure, there is provided an apparatus for searching a neural network structure, including: an acquisition module configured to acquire a first neural network including a plurality of blocks with different numbers of channels, wherein the first neural network At least one of the blocks is connected to at least three blocks, and at least one block includes a head layer for performing block channel number and spatial resolution conversion; a search module for using a gradient-based search strategy according to the sample data in the first data set , performing a neural network structure search process on the first neural network obtained by the obtaining module to obtain the structural parameters of the first neural network; the determining module is used for obtaining the structural parameters of the first neural network according to the searching module, The second neural network structure obtained by the search is determined.

According to yet another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, where the storage medium stores a computer program, and the computer program is used to execute the above-mentioned neural network structure search method.

According to yet another aspect of the embodiments of the present disclosure, there is provided an electronic device, the electronic device comprising: a processor; a memory for storing instructions executable by the processor; the processor for retrieving from the memory The executable instructions are read and executed to implement the above-described neural network structure search method.

Based on the neural network structure search method and neural network structure search device provided by the above embodiments of the present disclosure, since the blocks in the present disclosure can be connected to at least three blocks, the first neural network of the present disclosure can be considered to be based on dense Connected search space; since the blocks in this disclosure contain a head layer for performing block channel number conversion, interconnected blocks can have different numbers of channels, so the present disclosure allows the first neural network to include different numbers of channels For the same multiple blocks, by performing the neural network structure search process in the first neural network including multiple blocks with different number of channels, not only the deep search of the neural network structure can be realized, but also the deep search of the neural network structure can be realized. Channel number search. It can be seen from this that the technical solution provided by the present disclosure is beneficial to improve the flexibility of the neural network structure search, thereby improving the performance and diversity of the neural network obtained by the search.

The technical solutions of the present disclosure will be further described in detail below through the accompanying drawings and embodiments.

Description of drawings

The accompanying drawings, which form a part of the specification, illustrate embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure.

The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, wherein:

1 is a flowchart of an example of the disclosed neural network structure search method;

FIG. 2 is a schematic structural diagram of an example of the first neural network of the present disclosure;

3 is a schematic structural diagram of an example of an MBConv structure included in the parallel layer of the present disclosure;

FIG. 4 is a schematic structural diagram of an example of an MBConv structure included in the stacked layers of the present disclosure;

5 is a schematic structural diagram of an embodiment of a variable grouping-based convolution module of the present disclosure;

6 is a schematic structural diagram of an example of the disclosed neural network structure search apparatus;

FIG. 7 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure.

Detailed ways

Example embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present disclosure, not all of the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited by the example embodiments described herein.

It should be noted that the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

Those skilled in the art can understand that terms such as "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, etc., and neither represent any specific technical meaning, nor represent any difference between them. the necessary logical order of .

It should also be understood that, in the embodiments of the present disclosure, "a plurality" may refer to two or more than two, and "at least one" may refer to one, two or more than two.

It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more in the case of no explicit definition or contrary indications given in the context.

In addition, the term "and/or" in the present disclosure is only an association relationship to describe associated objects, which means that there can be three kinds of relationships, such as A and/or B, which can mean that A exists alone, A and B exist simultaneously, There are three cases of B alone. In addition, the character "/" in the present disclosure generally indicates that the related objects are an "or" relationship.

It should also be understood that the description of the various embodiments in the present disclosure emphasizes the differences between the various embodiments, and the same or similar points can be referred to each other, and for the sake of brevity, they will not be repeated.

Meanwhile, it should be understood that, for the convenience of description, the dimensions of various parts shown in the accompanying drawings are not drawn in an actual proportional relationship.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application or uses in any way.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

Embodiments of the present disclosure may be applied to electronic devices such as terminal devices, computer systems, servers, etc., which may operate in conjunction with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal equipment, computing systems, environments and/or configurations suitable for use with electronic equipment such as terminal equipment, computer systems or servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients computer, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the foregoing, among others.

Electronic devices such as terminal devices, computer systems, servers, etc., may be described in the general context of computer system-executable instructions, such as program modules, being executed by the computer system. Generally, program modules may include routines, programs, object programs, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks may be performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

SUMMARY OF THE DISCLOSURE

In the process of realizing the present disclosure, the inventors found that performing Neural Architecture Search (NAS) processing in the search space (Search Space) to obtain the structure of the neural network has become an important aspect of computer vision technology. subject.

In the existing neural network structure search technology, a super network (Super Network) is usually used to represent the search space. A super network includes multiple blocks (Blocks), and a sequential single connection method is usually adopted between multiple blocks, that is, one block is usually connected to an upstream block and a downstream block respectively. A block usually includes multiple layers. The possible layers of the target neural network can be obtained by searching in the super network. However, this method usually only realizes the deep search of the neural network, but cannot realize the width search of the neural network, which will limit the structure of the target neural network obtained from the super network search.

Exemplary overview

It is assumed that the target neural network to be obtained includes multiple sequentially connected blocks, and the number of channels and spatial resolutions of all the blocks contained in the target neural network need to be obtained by searching. In an environment with low computing power, the target neural network can be quickly obtained from the super network by using the neural network structure search technology provided by the present disclosure. For example, in a computing power environment containing four GPUs (Graphics Processing Unit, graphics processing unit), it is possible to realize the width search and depth search of the target neural network at a time cost of only about tens of hours. Finally, the target neural network is obtained.

Exemplary method

FIG. 1 is a flowchart of the disclosed neural network structure search method. As shown in FIG. 1 , the method of this embodiment includes steps: S100 , S101 and S102 . Each step will be described below.

S100. Acquire a first neural network including a plurality of blocks with different numbers of channels.

The first neural network in this disclosure may be referred to as a search space or a super network. The first neural network includes: a plurality of blocks. At least one block of the first neural network is connected to a plurality of blocks, eg, one of the blocks of the first neural network is connected to at least three blocks. For another example, all blocks except the predetermined block in the first neural network are connected with at least three blocks. The predetermined blocks may include: two blocks located at the head of the first neural network, two blocks located at the tail of the first neural network, and the like. The first neural network in this disclosure may be referred to as a dense connection based super network. The number of channels of all blocks included in the first neural network may be different.

At least one block (eg, all blocks) in the first neural network of the present disclosure includes a head layer and at least one stacked layer. The header layer can be thought of as the first layer in a block. The header layer is used to perform block channel number and spatial resolution conversion, so that the layers behind the header layer in the block can process the output of all upstream blocks connected to the block in which it is located.

The number of channels of a block in the present disclosure may be referred to as the width of the block. The number of channels in a block can be considered as the number of input channels and the number of output channels corresponding to each layer located after the head layer in the block. The spatial resolution of a block can be referred to as the length and height of the block. The spatial resolution of a block can be considered as the spatial resolution corresponding to each layer located after the head layer in the block. For example, the number of channels and the height of the input feature maps of the layers located after the head layer in the block can be considered as the number of channels and the length of the block. It can be seen from this that each layer located after the head layer in the block of the present disclosure corresponds to the same number of channels and spatial resolution.

S110. According to the sample data in the first data set, using a gradient-based search strategy, perform a neural network structure search process on the first neural network to obtain structural parameters of the first neural network.

The gradient-based search strategy in the present disclosure can be considered as: derivation of the loss function, and gradually approach the minimum value of the loss function after the derivation. The present disclosure can provide the sample data in the first data set to the first neural network, obtain the processing result output by the first neural network, and perform the reverse operation according to the difference between the processing result and the sample data and the loss function after derivation. A forward propagation operation to adjust the structural parameters of the first neural network. The structural parameters of the first neural network in the present disclosure may refer to: parameters set for the structure of the first neural network, and the structural parameters of the first neural network may generally reflect the corresponding structure in the first neural network, and become the second neural network. Possibilities of structures in neural networks.

S102. Determine the second neural network structure obtained by searching according to the structural parameters of the first neural network.

The second neural network in the present disclosure may be referred to as a target neural network. That is, the neural network obtained by searching the neural network structure in the search space. The second neural network can be considered as a substructure of the first neural network structure.

Since a block in the present disclosure can be connected to at least three blocks, for example, one block can be connected to at least three upstream blocks respectively; therefore, the first neural network of the present disclosure can be considered as a dense connection-based search space or a dense-connection-based search space the super network; the present disclosure can make all upstream blocks connected to a block have different channel numbers by setting a head layer for performing block channel number conversion within the block, so the present disclosure allows the first neural network in the first neural network. All upstream blocks of a block may include multiple blocks with different numbers of channels. By performing the neural network structure search process in the first neural network based on dense connection, not only the depth search of the neural network, but also the channel number search of the neural network can be realized. It can be seen from this that the technical solution provided by the present disclosure is beneficial to improve the flexibility of the neural network structure search, thereby improving the performance and diversity of the neural network obtained by the search.

In an optional example, the manner of obtaining the first neural network in the present disclosure includes but is not limited to: obtaining the first neural network by means of data import, or generating the first neural network according to preset information. The preset information can be considered as input parameters of the software program for generating the first neural network. The preset information may include, but is not limited to, the number of channels of each block, the number of upstream blocks connected to the block, and the like. In addition, the preset information may further include: the spatial resolution of each block, the number of stacked layers included in the block, and the like.

In an optional example, the process of obtaining the first neural network including a plurality of blocks with different numbers of channels in the present disclosure may include the following two steps:

Step 1. Obtain the number of channels of all blocks required by the first neural network and the number of upstream/downstream blocks connected to the block.

Optionally, the number of channels of a block in the present disclosure may be considered as: the number of input channels and the number of output channels corresponding to each layer located after the head layer in the block. The spatial resolution of the block may be referred to as the length and height of the block, and the length and height of the block may be equal. Of course, the present disclosure does not exclude the case where the length and height are not equal. The spatial resolution of a block can be considered as the spatial resolution corresponding to each layer located after the head layer in the block. For example, the number of channels and the length and height of the input feature maps of the layers located after the head layer in the block can be considered as the number of channels and the length and height of the block. The layers located after the header layer in the block of the present disclosure correspond to the same number of channels and spatial resolution.

Optionally, all blocks in the present disclosure have different numbers of channels. That is, all blocks in the present disclosure do not have the same number of channels. For example, the number of channels of a part of the blocks in all the blocks is the first number of channels, and the number of channels of another part of the blocks of all the blocks is the number of the second channels, and the number of the first channels and the number of the second channels are different. For another example, there are no two blocks with the same number of channels in all blocks. That is to say, assuming that the number of all blocks required by the first neural network is N, there are M different channel numbers among the channel numbers of the N blocks, where N and M are both positive integers greater than 2, And M is less than or equal to N. The block set Arch formed by N blocks in the present disclosure can be expressed as: Arch={B ₁ , B ₂ , B ₃ , . . . , B _N }.

Optionally, for the ith block (that is, B _i ) in all blocks, the number of upstream blocks connected to the block in the present disclosure refers to: other blocks, for example, other than the first two blocks located at the head of the first neural network, the number of predetermined blocks depends on the number of upstream blocks, such as the number of predetermined blocks is the number of upstream blocks minus one) before , and the number of blocks adjacent and connected to the ith block. Correspondingly, the number of downstream blocks connected to the block in the present disclosure refers to: blocks located in the i-th block (except for the predetermined block located at the tail of the first neural network, for example, except for the last block located at the tail of the first neural network. For blocks other than the two blocks, the number of predetermined blocks depends on the number of downstream blocks, for example, the number of predetermined blocks is the number of blocks adjacent to and connected to the ith block after the number of downstream blocks minus one). When the number of upstream blocks connected to the block is obtained, it indicates that the number of upstream/downstream blocks in the first neural network is the same except for the first block to the xth block at the front. where x is the number of upstream blocks minus one. Correspondingly, when the number of downstream blocks connected to the block is obtained, it indicates that the number of downstream blocks of other blocks in the first neural network except the last block to the last xth block is the same. where x is the number of downstream blocks minus one. The present disclosure can set the number of channels of all blocks and the number of upstream/downstream blocks connected to the blocks required by the first neural network according to actual requirements. The present disclosure does not limit this.

It should be noted that, when all the blocks required by the first neural network of the present disclosure are sorted according to the increasing number of channels of the blocks, multiple adjacent blocks may have the same increase in the number of channels. The number of channels is increased. After sorting all the blocks according to the number of channels of the block, all the blocks can be divided into multiple groups according to the increase of the number of channels. All blocks in each group correspond to the same increase of the number of channels, and different groups correspond to Different channel count increments. Optionally, all blocks in the same group usually have the same spatial resolution, and the number of channels of any two blocks in all blocks in the same group is usually different.

Step 2: Determine the connection relationship of all the blocks required by the first neural network according to the above-mentioned number of upstream blocks and increasing the number of channels of the blocks.

Optionally, the present disclosure may sort all blocks according to the number of channels of all blocks and in an increasing order of the number of channels of the blocks, and determine the upstream/downstream blocks of each block by using the above-mentioned number of upstream/downstream blocks. The connection relationship between the blocks and the corresponding upstream/downstream blocks forms the block connection architecture of the first neural network.

Optionally, it is assumed that the number of all blocks required by the first neural network in the present disclosure is n+1, where n is a positive integer greater than 5, and the channel numbers of the n+1 blocks are: C ₀ , C ₀ +c, C ₀ +2c, C ₀ +3c, C ₀ +4c, C ₀ +5c, ..., C ₀ +nc, where C ₀ is a positive integer greater than 1 (eg C ₀ =3) , assuming that the number of upstream/downstream blocks connected to the block is 4, an example of the block connection architecture of the first neural network formed by the present disclosure is shown in FIG. 2 .

The present disclosure obtains the number of channels of all blocks and the number of upstream/downstream blocks connected to the block, and determines the connection relationship of all blocks according to the number of upstream/downstream blocks and in an increasing order of the number of channels of the blocks, so as to form a first neural network. The network provides a feasible implementation and helps to reduce the computational power consumed by the neural network structure search process.

In an optional example, the present disclosure may divide the formed first neural network into multiple stages according to the spatial resolution of the block and the increase in the number of channels of the block, and one stage may include one block or multiple blocks. The spatial resolution of all blocks in a stage is usually the same. The number of channels is usually not the same for all blocks in a stage. The increment of the number of channels is usually the same for all blocks in a stage, ie a stage has a increment of the number of channels. A stage in the first neural network can be thought of as a group of the aforementioned. In the case where each stage in the first neural network has an increase in the number of channels, for two adjacent stages in the first neural network, the increase in the number of channels in the former stage is usually smaller than that in the latter stage. increase in the number of channels. Optionally, for the ith block, if the ith block is adjacent to and connected to the ith block, and the ith block is the upstream block of the ith block, the length of the ith block is the same as that of the ith block. The ratio of the length of the -1 block is usually not greater than 2, and the ratio of the height of the i-th block to the height of the i-1-th block is usually not greater than 2.

In an optional example, among the three sequentially connected blocks in the first neural network, the increase in the number of channels of the first block relative to the intermediate block is usually not greater than the number of channels of the intermediate block relative to the last block. increase. For example, if three blocks connected in sequence belong to the same stage, the increase in the number of channels of the first block relative to the middle block is usually equal to the increase of the number of channels of the middle block relative to the last block. For another example, if three blocks connected in sequence belong to two stages, the increase in the number of channels of the first block relative to the middle block is usually smaller than the increase of the number of channels of the middle block relative to the last block.

By making the increase of the number of channels between the two blocks arranged in the first neural network smaller than the increase of the number of channels between the two blocks arranged in the back, the present disclosure is beneficial to reduce the number of neural networks in the first neural network. The computing power consumed by the network structure search processing.

In an optional example, in the process of forming the first neural network, the present disclosure also needs to set the head layer in each block and the layers after the head layer. The header layer in the present disclosure may be composed of multiple parallel layers, and the number of parallel layers included in the header layer is generally the same as the preset number of upstream/downstream blocks. The parallel layers in the present disclosure refer to layers arranged in parallel. That is, there is no upstream-downstream relationship between parallel layers in a head layer. A parallel layer corresponds to an upstream block that is adjacent to and connected to its own block. Different parallel layers in a block correspond to different upstream blocks adjacent to and connected to the block in which it resides. All parallel layers in a block correspond to all upstream blocks adjacent to and connected to the block in which it resides. That is to say, a block processes the output information (for example, feature maps, etc.) of each upstream block through its header layer, so that the layers located after the header layer in the block can process the output information of each upstream block. deal with.

Optionally, for the ith block in the first neural network, the present disclosure may determine the number of parallel layers included in the head layer of the ith block according to the preset number of upstream/downstream blocks, and determine the number of parallel layers included in the head layer of the ith block according to the preset number of upstream and downstream blocks. and the number of channels and spatial resolution of each upstream block adjacent to and connected to the ith block, determine the number of input channels and spatial resolution corresponding to each parallel layer in the ith block, and Number of output channels and spatial resolution. That is, the number of input channels and spatial resolution corresponding to any parallel layer in the ith block usually depends on the number of channels and spatial resolution of the upstream block corresponding to the parallel layer, and any parallel layer in the ith block corresponds to The number of output channels and spatial resolution of , usually depends on the number of channels and spatial resolution of the i-th block. The number of input channels corresponding to the parallel layer can be considered as the number of channels of input information (such as feature maps) received by the parallel layer. The spatial resolution corresponding to the parallel layer can be considered as the spatial resolution of the input information (such as feature maps) received by the parallel layer.

As an example, assuming that the number of upstream blocks of the i-th block (ie, B _i ) is a, the a upstream blocks are denoted as B _i-1 , B _i-2 , . . . , B _ia , and B _i The number of channels and spatial resolution of B i-1 are denoted as C _i and H _i ×W _i , respectively, and the number of channels and spatial resolution of B _i-1 are denoted as C _i-1 and H _i-1 ×W _i-1 , B _i The number of channels and spatial resolution of _-2 are denoted as C _i-2 and H _i-2 ×W _i-2 , and so on, the number of channels and spatial resolution of B _ia are denoted as C _ia and H _ia ×W, respectively _ia ; Under the above-mentioned assumptions, the head layer of B _i in the present disclosure may include a parallel layers, wherein the first parallel layer corresponds to B _i-1 (eg, connected with B _i-1 ), and the first parallel layer corresponds to B i-1. The parallel layer is used to convert the C _i-1 ×H _i-1 ×W _i-1- based feature map output by B _i-1 into a C _i ×H _i ×W _i -based feature map; the second parallel layer corresponds to B _i-2 , and the second parallel layer is used to convert the C _i-2 ×H _i-2 ×W _i-2- based feature map output by B _i-2 to a C _i ×H _i ×W _i based feature map The feature map of , and so on, the a-th parallel layer corresponds to B _ia , and the a-th parallel layer is used to convert the feature map based on C _ia ×H _ia ×W _ia output by B _ia to C _i ×H _i Feature map of ×W _i .

In the present disclosure, a plurality of parallel layers are set in the header layer of a block, and each parallel layer corresponds to an upstream block adjacent to and connected to the block where it is located, so that the multiple upstream blocks adjacent to and connected to the block where it is located each output The information with different channel numbers and spatial resolutions can be converted to the channel number and spatial resolution of the block in which it is located, so that the layers (such as stacked layers) located after the head layer in the block can The information output by multiple upstream blocks with different channel numbers and spatial resolutions are processed separately, so that the first neural network can be formed by multiple blocks with different channel numbers and spatial resolutions, which is beneficial to improve the search efficiency of the neural network structure. flexibility.

In an optional example, the operations performed by the parallel layer in the header layer of the block in the present disclosure may be set according to a preset candidate operation set. For example, the present disclosure can set the operations included in each parallel layer in the head layer according to all the candidate operations in the preset candidate operation set. The method of the present disclosure to set the output of any parallel layer in the i-th block in the first neural network may be as follows: first, according to the operation weight of each candidate operation in the parallel layer and the output of each candidate operation, perform calculation, and obtain the calculation method. The result; then, the calculation result is processed (such as convolution processing, etc.), and the processed result is used as the output of the parallel layer. The number of channels in the processing result is the same as the number of channels in the block in which it is located. That is, in the process of processing the calculation result, the channel number and spatial resolution of the calculation result are converted into the channel number and spatial resolution of the block where the parallel layer is located.

In an optional example, the present disclosure is preset with a set of candidate operations. The set of candidate operations may refer to: a set formed by the parallel layer and all candidate operations that may be involved in the layers located after the head layer. The number of parallel layers, the number of stacked layers, and the operations performed by the parallel layers and the stacked layers in each block included in the second neural network obtained through the neural network structure search are usually selected from the candidate operations. determined by the operation.

Optionally, the parallel layer in the present disclosure may adopt, but is not limited to, the MBConv structure. When the parallel layer adopts the MBConv structure, an example of the MBConv structure included in the parallel layer in the present disclosure is shown in FIG. 3 .

The number of input channels of this structure in the parallel layer shown in FIG. 3 is C, the number of output channels is C', and C and C' may be different. The structure in the parallel layer in Figure 3 consists of 3 parts. The leftmost trapezoidal box represents the convolution operation with a convolution kernel size of 1×1 and the ReLU6 activation function. The number of input channels is C and the number of output channels is tC, where t is the expansion coefficient. The rectangular box in the middle represents a depthwise separable convolution with stride 2 and a kernel size of k×k and a ReLU6 activation function, whose number of input channels and number of output channels are both tC. The rightmost trapezoid box represents a convolution operand with a kernel size of 1×1, whose number of input channels is tC and the number of output channels is C. where t and k×k depend on the candidate operation.

Optionally, an example of all the candidate operations included in the candidate operation set in the present disclosure may be shown in the following table:

Table 1

Mbconv_k3e3 in Table 1 represents the Mbconv convolution operation with a convolution kernel size of 3×3 and an expansion factor of 3; Mbconv_k3e6 represents an Mbconv convolution operation with a convolution kernel size of 3×3 and an expansion factor of 6; By analogy, I will not explain them one by one; skip means skip, that is, skip means that the input of the candidate operation is not processed, and it can be considered that the output of the candidate operation is the input of the candidate operation.

For any parallel layer in the head layer of any block, the output of the parallel layer can be expressed as:

表示块的头层中的第l个并行层中的第o个候选操作的操作权重。In the above formula (1), x _lout represents the output of the l-th parallel layer in the head layer of the block; o(x _lin ) represents the o-th candidate operation in the l-th parallel layer in the head layer of the block for The input _xlin of the lth parallel layer is processed, and the result obtained is,

Represents the operation weight of the o-th candidate operation in the l-th parallel layer in the head layer of the block.

通常是经过归一化处理后的操作权重。本公开可以采用下述公式获得归一化处理后的第l个并行层中的第o个候选操作的操作权重：Optionally, in this disclosure

Usually it is the normalized operation weight. The present disclosure can use the following formula to obtain the operation weight of the 0th candidate operation in the 1th parallel layer after normalization:

表示头层中的第l个并行层中的第o个候选操作的操作权重(即归一化处理后的操作权重)；

表示头层中的第l个并行层中的第o个候选操作的归一化处理前的操作权重；O表示候选操作集合；

表示头层中的第l并行层中的第o'个候选操作的归一化处理前的操作权重。In the above formula (2),

Represents the operation weight of the o-th candidate operation in the l-th parallel layer in the head layer (that is, the operation weight after normalization);

Represents the operation weight before normalization of the o-th candidate operation in the l-th parallel layer in the head layer; O represents the candidate operation set;

Represents the operation weight before normalization of the 0'th candidate operation in the lth parallel layer in the head layer.

In the present disclosure, by using the operation weights of the candidate operations to perform fusion processing on the outputs of each candidate operation in the parallel layer, the processing results of the parallel layer can fully reflect all the candidate operations in the parallel layer, and the processing results of the parallel layer can fully reflect all the candidate operations in the parallel layer. , which is beneficial to use the loss function to adjust the operation weights of all candidate operations in the parallel layer, which is beneficial to search and obtain the structure of the second neural network from the first neural network.

In an optional example, for any block (such as the i-th block) in the first neural network, since the head layer includes multiple parallel layers, the head layer in the present disclosure needs to output multiple parallel layers Perform fusion processing, and provide the result of fusion processing as the output of the head layer to the first stack layer after the head layer. In the process of fusing the outputs of all parallel layers included in the head layer, the weight of each block connected to each parallel layer can be considered. The weight of a block in the present disclosure may be referred to as a block connection weight. That is to say, the present disclosure can perform calculation according to the corresponding block connection weight of each parallel layer in the head layer of the ith block and the output of each parallel layer, for example, perform weighted average calculation, and the obtained calculation result is used as the ith block. The output of the head layer of the i-th block, the output of the head layer of the i-th block can be used as the input of the first stacked layer located after the head layer in the i-th block.

Optionally, the operation performed by the head layer of the i-th block in the present disclosure can be represented by the following formula:

In the above formula (3), x _i represents the output information of the head layer of the ith block; m' represents the number of upstream blocks connected to the ith block; k=1,...,m'; p _{ik , k} represents the block connection weight of the ik th upstream block connected to the ith block; H _ik (x _ik ) represents the processing result of the output information of the ik th upstream block by the corresponding parallel layer in the head layer in the ith block, i.e. The information output by the corresponding parallel layer.

Optionally, p _ik,k in the present disclosure is usually the block connection weight after normalization. Assuming that any upstream block connected to the i-th block can be represented as the j-th block, the present disclosure can obtain the block connection weight of the j-th block after normalization by the following formula:

In the above formula (4), p _ij represents: when the jth block is the upstream block of the ith block, for the connection between the jth block and the ith block, the normalized value of the jth block Block connection weight; m represents the number of upstream blocks connected to the ith block; k=1,...,m; β _ij represents: when the jth block is the upstream block of the ith block, for the ith block For the connection between the j-th block and the i-th block, the block connection weight of the j- _th block before normalization; For the connection of the i block, the block connection weight of the kth block before normalization is processed.

By using the block connection weight, the present disclosure performs fusion processing on the outputs of each parallel layer in the head layer of the block. On the one hand, each stacked layer in the block can fuse the outputs of each upstream block with different channel numbers and spatial resolutions. On the other hand, the processing results of the stacked layers can fully reflect the output information of multiple upstream blocks with different block connection weights, which affects the processing results of the stacked layers, which is beneficial to use the loss function to adjust the block connection. The weights are further beneficial to obtain the structure of the second neural network by searching from the first neural network.

In an optional example, any block in the present disclosure may include: a plurality of stacked layers. Stacked layers are layers in a block that follow the header layer. Stacked layers can be used to process information from the outputs of all upstream blocks adjacent to and connected to the block on which it resides. All stacked layers included in a block are connected in sequence, and the input of the stacked layer arranged at the front among all stacked layers included in a block is the output of the head layer of the block, and all stacked layers included in a block are The permutation in the output of the last stacked layer can be thought of as the output of the block.

In an optional example, for the ith block in the first neural network, the present disclosure may determine, according to the number of channels and the spatial resolution of the ith block, the stacked layers in the ith block that are sequentially located after the head layer The corresponding number of input channels and spatial resolution and the number of output channels and spatial resolution. For example, the number of input channels and the number of output channels corresponding to each stacked layer in the ith block are the number of channels of the ith block, and the corresponding input spatial resolution and output spatial resolution of each stacked layer in the ith block are the spatial resolution of the i-th block. The number of stacked layers included in the i-th block can be set according to actual requirements. For example, the number of stacked layers included in the i-th block is usually not less than 3. In addition, the number of stacked layers contained in all blocks in the first neural network is usually the same. Of course, the present disclosure also does not exclude the situation that all blocks in the first neural network contain different numbers of stacked layers.

In the present disclosure, by setting the format of the input information and output information corresponding to the stacked layers in the ith block according to the number of channels and the spatial resolution of the ith block, it is beneficial to normalize the operations performed by each block in the first neural network, so that the The structure of each block is standardized, thereby facilitating the convenient formation of the first neural network and improving the maintainability of the first neural network.

Optionally, the present disclosure may set the operations included in each stack layer sequentially located after the head layer in each block according to all the candidate operations in the preset candidate operation set. An example of all candidate operations is shown in Table 1 above. For any stacked layer in the block, the manner in which the present disclosure sets the output of the stacked layer may include: calculating (eg, weighted average) according to the operation weight for each candidate operation in the stacked layer and the output of each candidate operation calculation) to obtain the calculation result; after that, the calculation result can be processed (such as convolution processing, etc.), and the processed result can be used as the output of the stacked layer.

Optionally, the operation weight of any candidate operation in the stacked layer may indicate the possibility that the candidate operation in the stacked layer is selected to become a structure in the second neural network.

For any stack in any block, the output of the stack can be expressed as:

表示块中的第l堆叠层中的第o个候选操作的操作权重。In the above formula (5), x _l+1 represents the output of the lth stack layer in the block, and can also be considered as the input of the l+1th layer in the block; o(x _l ) represents the lth stack in the block. The 0th candidate operation in the layer is processed against the output of the l-1th layer (may be the stacked layer or the head layer) in the block, and the result obtained,

Represents the operation weight of the 0th candidate operation in the lth stack layer in the block.

通常是经过归一化处理后的操作权重。本公开可以采用下述公式获得归一化处理后的第l堆叠层中的第o个候选操作的操作权重：Optionally, in this disclosure

Usually it is the normalized operation weight. The present disclosure can use the following formula to obtain the operation weight of the 0th candidate operation in the 1th stack layer after normalization:

表示块中的第l堆叠层中的第o个候选操作的操作权重；

表示块中的第l堆叠层中的第o个候选操作的归一化处理前的操作权重；O表示候选操作集合；

表示块中的第l堆叠层中的第o'个候选操作的归一化处理前的操作权重。In the above formula (6),

represents the operation weight of the o-th candidate operation in the l-th stacked layer in the block;

Represents the operation weight before normalization of the o-th candidate operation in the l-th stacked layer in the block; O represents the candidate operation set;

Represents the operation weight before normalization of the 0'th candidate operation in the lth stack layer in the block.

Optionally, the stacked layers of the present disclosure may adopt, but not limited to, an MBConv structure. When the stacked layer adopts the MBConv structure, an example of the structure of the MBConv structure included in the stacked layer is shown in FIG. 4 . Figure 4 includes three sections. The leftmost trapezoidal box represents the convolution operation with a kernel size of 1×1 and the ReLU6 activation function. The rectangle in the middle represents the depthwise separable convolution with kernel size k×k and the ReLU6 activation function. The rightmost trapezoidal box represents the convolution operands with a kernel size of 1×1. The number of input channels and the number of output channels in Figure 4 are both C.

In the present disclosure, by using the operation weights of the candidate operations to perform fusion processing on the outputs of each candidate operation in the stacked layer, the processing result of the stacked layer can fully reflect all the candidate operations in the stacked layer, and the processing result of the stacked layer can fully reflect all the candidate operations in the stacked layer. , which is beneficial to use the loss function to adjust the operation weights of all candidate operations, which is beneficial to search and obtain the structure of the second neural network from the first neural network.

In an optional example, an example of the first neural network formed by the present disclosure is shown in FIG. 5 . The lower graph in Figure 5 is the first neural network comprising 17 blocks. Each block in the first neural network is connected with 3 downstream blocks, that is, the number of upstream/downstream blocks connected with the block is 3. The first neural network in Figure 5 can be divided into 5 stages, namely:

1. The first stage at the far left of Figure 5. The first stage includes a block, namely the box marked 16 in Fig. 5, the number of channels of this block is 16, and the spatial resolution of this block is 112×112.

2. The second stage located on the left side of Figure 5. The second stage includes a block, that is, the box marked 24 in Figure 5. The number of channels in this block is 24, and the spatial resolution of this block is 56×56. The number of channels between the blocks of the first stage and the blocks of the second stage is increased by 8.

3. The third stage located at the left-middle position in FIG. 5 . The third stage includes three blocks, namely the three blocks marked 32 , 40 and 48 in FIG. 5 . The number of channels of the first block, the second block and the third block in the third stage are 32, 40 and 48 respectively, and the number of channels of the first block, the second block and the third block in the third stage The spatial resolutions are all 28×28. The number of channels between adjacent blocks in the third stage is increased by 8.

4. The fourth stage located in the middle of FIG. 5 , the fourth stage includes six blocks, namely the six blocks marked 64, 80, 96, 112, 128 and 144 in FIG. 5 . The channel numbers of the first to sixth blocks in the fourth stage are 64, 80, 96, 112, 128, and 144, respectively, and the spatial resolutions of the first to sixth blocks in the fourth stage are all is 14×14. The number of channels between adjacent blocks in the fourth stage is increased by 16.

5. The fifth stage at the far right position in FIG. 5 , the fifth stage includes six blocks, namely the six blocks marked 160 , 224 , 288 , 352 , 416 and 480 in FIG. 5 . The number of channels from the first block to the sixth block in the fifth stage are 160, 224, 288, 352, 416 and 480, respectively, and the spatial resolutions of the first block to the sixth block in the fifth stage are all is 7×7. The number of channels between adjacent blocks in the fifth stage is increased by 64.

The structure of any block in the first neural network in FIG. 5 is shown in the upper left figure of FIG. 5 . That is, the head layer of each block includes three parallel layers and three sequentially connected stacked layers. The inputs of the three parallel layers are the outputs of the three upstream blocks connected to the block. After the block connection weights perform a weighted average calculation on the outputs of the three parallel layers, the result of the weighted average calculation can be used as the input of the first stacked layer of the block.

The structure contained in any stacked layer in any block of the first neural network in FIG. 5 and any parallel layer in any head layer (the middle box in FIG. 3 and the middle box in FIG. 2 ) is shown in the upper right of FIG. 5 as shown in the figure. That is, each parallel layer and each stacking layer include all candidate operations. Each candidate operation processes its input separately to form the output of each candidate operation. When the candidate operation is skipped, the output of the candidate operation is the candidate operation. Operation input. After the output of each candidate operation is weighted and averaged according to the operation weight of each candidate operation, the result of the weighted average calculation can be obtained, and the result of the calculation is used to form the output of the layer, for example, the result of the weighted average calculation is rolled Product processing, and the result of convolution processing as the output of this layer. The output of the last stacked layer can be used as the output of this block.

In an optional example, before using the sample data in the first data set to perform the neural network structure search processing on the first neural network, the present disclosure should first ensure that the processing of the input information by the first neural network has a certain accuracy . Therefore, the present disclosure can use the sample data in the second data set to train the first neural network before performing the neural network structure search process on the first neural network to adjust each layer ( Including the operation parameters of each candidate operation in each parallel layer and each stacked layer), so that the first neural network can process the input information with certain accuracy. Since the process of using sample data to adjust the structural parameters in the first neural network can also be referred to as the training process of the first neural network, the training process of the first neural network in the present disclosure may include two stages, the first stage For the training process for the operation parameters, the second stage is the training process for the structural parameters.

It should be noted that the second stage can be trained only for structural parameters, and the second stage can also be trained for operation parameters and structural parameters at the same time (for example, a multi-objective optimization method is used to train operation parameters and structural parameters at the same time). In addition, the first stage and the second stage can be performed iteratively, that is, the first stage and the second stage can be performed alternately. For example, the first stage of training is performed first, and when the first predetermined iterative condition is reached (such as the number of sample data used reaches a predetermined number or the accuracy of the processing of the input information by the first neural network meets certain requirements, etc.), the first stage is stopped. The first-stage training process, and then start the second-stage training, when the second predetermined iterative condition is reached (for example, the number of sample data used reaches a predetermined number or the convergence of the structural parameters of the first neural network meets certain requirements, etc.) , stop the training process of the second stage, start the training of the first stage again, and so on, so as to realize the iterative progress of the first stage and the second stage.

表示对操作参数w求导，L_train(w,α,β)表示针对操作参数w、候选操作的操作权重α以及块连接权重β的第二交叉熵损失函数。该第二交叉熵损失函数在计算时，会使用到操作权重α和块连接权重β，但是，在第一阶段的反向传播过程中，通常只更新操作参数w，而并不会对操作权重α和块连接权重β进行更新。Optionally, the operation parameters of the candidate operation in the present disclosure include, but are not limited to: the weight of the convolution kernel in the candidate operation, and the like. The present disclosure can provide the sample data in the second data set to the first neural network, and process the sample data via the first neural network; after that, the present disclosure can process the sample data according to the processing result of the sample data and the sample data (such as the processing result and the sample data). The difference between the data), using the second loss function (such as cross entropy loss function, etc., hereinafter referred to as the second cross entropy loss function), adjust each layer in each block in the first neural network (including each parallel layer and Operation parameters for each candidate operation in each stack layer). Before the first stage of training is performed for the first time, the present disclosure may use a random initialization method to assign values to the structural parameters of the first neural network. The second loss function can be expressed as

represents the derivation of the operation parameter w, and L _train (w, α, β) represents the second cross-entropy loss function for the operation parameter w, the operation weight α of the candidate operation, and the block connection weight β. The second cross-entropy loss function will use the operation weight α and the block connection weight β in the calculation, but in the back-propagation process of the first stage, only the operation parameter w is usually updated, and the operation weight is not affected. α and block connection weights β are updated.

Optionally, before the first stage of training is performed for the first time, the operation parameter w, the operation weight α of the candidate operation, and the block connection weight β can be assigned initial values (for example, initial values assigned by random initialization) . During the first training process of the first stage, the values of the operation weight α and the block connection weight β in the second cross-entropy loss function are still the initial values given above. In the subsequent process of performing the first-stage training again, the values of the operation weight α and the block connection weight β in the second cross-entropy loss function are usually the values obtained in the most recent second-stage training.

Optionally, the sample data included in the first data set and the second data set in the present disclosure are generally different. For example, the present disclosure can divide a complete sample data set into two parts, one part is used as the training set (ie the second data set), the other part is used as the validation set (ie the first data set), the sample data in the training set Used to adjust the operation parameters of each candidate operation in each layer in each block in the first neural network. The sample data in the validation set is used to adjust the structural parameters in the first neural network.

The present disclosure trains the first neural network by using the sample data in the second data set to adjust the operation of each candidate operation in each layer (including each parallel layer and each stacked layer) in each block in the first neural network The parameters can make the first neural network process the input information with certain accuracy, so in the second stage of training process, it is beneficial to make the structural parameters of the first neural network converge as soon as possible, which is beneficial to improve the neural network structure search efficiency.

In an optional example, the training of the first stage of the present disclosure can be implemented in the following two ways:

Mode 1: Provide the sample data in the second data set (for example, the training set) to the first neural network, and pass through all layers (including parallel layers and stacks) in all paths formed by all blocks in the first neural network All candidate operations in the layer), respectively process the sample data, and obtain the processing result of the first neural network on the sample data. That is to say, the input sample data flows through all block connections in the first neural network, and flows through all layers in each block, and is processed by all parallel layers in all blocks and all candidates in each stacked layer. The operation is processed.

In the case where the first method is adopted in the first stage, the present disclosure can adjust the first neural network by using the second loss function and the gradient descent method according to the processing result of the sample data and the sample data (such as the difference between the two) Operation parameters of all candidate operations in each layer in each block (including each parallel layer and each stacked layer). That is, during backpropagation of training, the operation parameters of all candidate operations in all layers in all blocks are updated.

Mode 2: The sample data in the second data set is provided to the first neural network, and selected ones of all layers (including each parallel layer and each stacked layer) in all paths formed by all blocks in the first neural network A candidate operation of , respectively processes the sample data, and obtains the processing result of the first neural network on the sample data. That is, the input sample data flows through all block connections in the first neural network and through all layers in each block, but is only selected by each of the parallel layers in all the blocks and the stacked layers One of the selected candidate operations is processed. The present disclosure can use a random selection method to select candidate operations, and the present disclosure can also select candidate operations according to the operation weights of the candidate operations. The present disclosure does not limit this.

In the case where the above-mentioned method 2 is adopted in the first stage, the present disclosure can use the second loss function and the gradient descent method according to the processing result of the sample data by the first neural network and the sample data (such as the difference between the two). to adjust the operation parameters of the selected candidate operations in each layer in each block in the first neural network. That is, during a backpropagation process in the first stage of training, the operation parameters of the selected candidate operation in all layers in all blocks (including each parallel layer and each stacked layer) will be updated. Operation parameters for candidate operations that are not picked are not updated. In the present disclosure, by selecting candidate operations to process sample data during the training process of the first stage, it is beneficial to improve the training efficiency of the first stage.

In an optional example, the present disclosure may start to perform the training process of the second stage after completing the training of the first stage. Specifically, the present disclosure can provide sample data in a first data set (eg, a validation set) to a first neural network, and process the sample data through the first neural network; after that, the sample data is processed according to the first neural network. The processing results and sample data (such as the difference between the two) are used to adjust the first loss function (for example, the cross entropy loss function, hereinafter referred to as the first cross entropy loss function), and the gradient descent method is used to adjust the first loss function. Block connection weights in the neural network and operation weights for each candidate operation in each parallel layer in each block and in each stacked layer.

其中的

表示对候选操作的操作权重α(归一化处理前的操作权重)以及块连接权重β(归一化处理前的块连接权重)求导，

表示针对操作参数w、候选操作的操作权重α以及块连接权重β的第一交叉熵损失函数。第一交叉熵损失函数在计算时，会使用到操作参数w，但是，在第二阶段的反向传播过程中，本公开通常只会对操作权重α和块连接权重β进行更新，并不会对操作参数w进行更新。Optionally, the first loss function can be expressed as

one of them

represents the derivation of the operation weight α of the candidate operation (the operation weight before normalization processing) and the block connection weight β (the block connection weight before normalization processing),

Represents the first cross-entropy loss function for the operation parameter w, the operation weight α of the candidate operation, and the block connection weight β. The operation parameter w is used in the calculation of the first cross-entropy loss function. However, during the back-propagation process of the second stage, the present disclosure usually only updates the operation weight α and the block connection weight β, and does not Update the operating parameter w.

Optionally, when the training of the second stage is started, the value of the operation parameter w in the first cross-entropy loss function is usually the value obtained in the most recent training of the first stage.

In the present disclosure, by using the first loss function to update the block connection weight in the first neural network and the operation weight of each candidate operation in each parallel layer in each block and each stack layer, the blocks in the first neural network can be The connection weight and the operation weight of each candidate operation converge toward the direction of gradient descent, which is beneficial to accurately reflect the structure of the second neural network from the first neural network.

In an optional example, the training of the second stage of the present disclosure can be implemented in the following two ways:

Mode 1: Provide the sample data in the first data set to the first neural network, and pass through all candidates in all layers (including each parallel layer and each stacked layer) in all paths formed by all blocks in the first neural network operation, respectively process the sample data, and obtain the processing result of the sample data by the first neural network.

In the case where the above-mentioned way 1 is adopted in the second stage, the present disclosure can use the first loss function (such as the first cross-entropy loss function) according to the processing result of the sample data and the sample data (such as the difference between the two), and adopt The gradient descent method is used to adjust the connection weight of each block in the first neural network and the operation weight of each candidate operation in each parallel layer in each block and each stack layer. That is, during the back-propagation process of the second stage training, all block connection weights and operation weights of all parallel layers in all blocks and all candidate operations in all stacked layers are updated.

Optionally, the present disclosure may set delay parameters of each layer (including parallel layers and stacked layers) in the above-mentioned first loss function, and the present disclosure may, according to the processing result of the sample data and the sample data (such as the difference between the two), Using the first loss function set with the delay parameters of each layer, the gradient descent method is used to perform multi-objective optimization processing to update the block connection weights in the first neural network and each layer in each block (including parallel layers and stacked layers) The operation weight of each candidate operation in .

The present disclosure can control the processing speed of the finally obtained second neural network by setting the delay parameters of each layer in the first loss function, and updating the fast connection weight and the operation weight through a multi-objective optimization method, thereby helping to improve the first Second, the applicability of neural networks to practical application environments.

Optionally, the first loss function L(w, α, β) set with delay parameters of each layer can be expressed as:

L(w,α,β)=L _CE +λlog _τ latency Formula (7)

In the above formula (7), L _CE represents the loss function of the operation parameter, for example, L _CE can be the second cross-entropy loss function described in the above embodiment; λ and τ represent hyper-parameters, used for Controls the magnitude of the stacking layer delay optimization term (latency); λ and τ can be known values; λ can be in the range of 0.05-0.7; for example, λ can be 0.2; τ can be in the range of 10-20 , such as τ can be 15; λ and τ can be positively correlated with latency; latency represents the delay time of all blocks in the first neural network.

Optionally, the delay time latency of all blocks in the first neural network can be calculated and obtained by using the following formula (8):

In the above formula (8), latency _l represents the delay time of the lth layer in the block; the minimum value in the value range of l can be 1, and the maximum value in the value range can be all the first neural network in the first neural network. The number of layers formed by all header layers and all stacked layers in the block.

Optionally, when the lth layer is a stacked layer, the latency _l in the above formula (8) can be obtained by calculating the following formula (9):

表示块中的第l堆叠层层中的第o个候选操作的操作权重(归一化处理后的操作权重)；

表示块中的第l堆叠层中的第o个候选操作的延迟时间；O表示候选操作集合。In the above formula (9),

represents the operation weight of the 0th candidate operation in the lth stack layer in the block (the operation weight after normalization);

represents the delay time of the 0th candidate operation in the lth stack layer in the block; O represents the set of candidate operations.

Optionally, when the lth layer is the head layer, the latency _l in the above formula (8) can be calculated by the following formula (10):

表示第i块的头层中的第l并行层中的第o个候选操作的操作权重；

表示第i块的头层中的第l并行层中的第o个候选操作的延迟时间；O表示候选操作集合。In the above formula (10), p _j,i represents the block connection weight between the i-th block and the j-th block (the normalized block connection weight), and the j-th block is adjacent to and connected to the i-th block the upstream block;

represents the operation weight of the o-th candidate operation in the l-th parallel layer in the head layer of the i-th block;

represents the delay time of the o-th candidate operation in the l-th parallel layer in the head layer of the i-th block; O represents the set of candidate operations.

Mode 2: The sample data in the first data set is provided to the first neural network, and the sample data is processed separately through two selected candidate operations in all layers in all paths formed by all blocks in the first neural network. Perform processing to obtain the processing result of the sample data. That is, the input sample data flows through all block connections in the first neural network and through all layers in each block, but is only selected by each of the parallel layers in all the blocks and the stacked layers The two selected candidate operations are processed. The present disclosure may adopt a random selection method to select two candidate operations, and the present disclosure may also select two candidate operations according to the operation weights of the candidate operations. The present disclosure does not limit this.

In the case where the above-mentioned method 2 is adopted in the second stage, the present disclosure can use the first loss function and the gradient descent method according to the processing result of the sample data by the first neural network and the sample data (such as the difference between the two). to adjust the connection weight of each block in the first neural network and the operation weight of the selected two candidate operations in each layer in each block. That is, in a back-propagation process of the second-stage training, the connection weights of all blocks and the operation weights of the two candidate operations selected this time in all layers (including each parallel layer and each stacked layer) in all blocks will be updated, but the operation weights of the candidate operations that are not selected this time will not be updated. In the present disclosure, in the training process of the second stage, two candidate operations are selected to process the sample data, which is beneficial to improve the training efficiency of the second stage.

In an optional example, in the case where the above-mentioned method 2 is adopted in the second stage, after updating the above-mentioned block connection weight and operation weight, the present disclosure may further adjust the operation weight of the two selected candidate operations , for example, using an offset to adjust the operation weights of the two selected candidate operations.

Optionally, the present disclosure can calculate the offsets of the two selected candidate operations according to the updated operation weights; after that, adjust the layers in the blocks in the first neural network according to the calculated offsets. The operation weights of the two selected candidate operations in , for example, the offsets are respectively added to the operation weights of the two selected candidate operations.

Optionally, the present disclosure can use the following formula (11) to calculate the offsets of the two selected candidate operations:

表示更新前的第l层中的第o个候选操作的操作权重；

表示更新后的第l层中的第o个候选操作的操作权重。In the above formula (11), O _s represents the selected two candidate operations; o represents the selected o-th candidate operation;

Represents the operation weight of the o-th candidate operation in the l-th layer before the update;

Represents the operation weight of the o-th candidate operation in the updated l-th layer.

The present disclosure helps to rationalize the operation weights by adjusting the operation weights of the two selected candidate operations in each layer of each block in the first neural network by using the offset, thereby improving the training in the second stage efficiency.

Optionally, the present disclosure may determine the blocks belonging to the second neural network and the parallel layers in the blocks by comparing the connection weights of the blocks in the first neural network. The present disclosure can determine the layers in the blocks belonging to the second neural network and the operations included in the layers by comparing the operation weights of all candidate operations in each layer (including parallel layers and stacked layers) in the blocks belonging to the second neural network . In addition, the present disclosure can determine the location of the upsampling in the second neural network according to the parallel layers in the blocks belonging to the second neural network.

It should be particularly noted that the present disclosure may preset some layers in the first neural network according to actual requirements, and the preset partial layers will be directly derived to become some layers in the second neural network. For example, at least the previous layer (eg, the first two layers) in the first neural network; for another example, at least the last-to-last layer in the first neural network, and so on. That is, some layers in the second neural network may be obtained without searching through the neural network structure. The operations performed by the pre-set layers can be set according to actual needs. For example, the first two layers in the first/second neural network are preset, where the first layer can be an ordinary convolutional layer whose output is based on 16×112×112 information (such as 16×112×112 features picture). The second layer can use an MBConv structure with a convolution kernel size of 3 × 3 and an expansion coefficient of 1, and the output of the second layer can be based on 24 × 56 × 56 information (such as a 24 × 56 × 56 feature map) . The pre-set layers in the first/second neural network can be considered as pre-set blocks, for example, the above-mentioned first layer can be considered as the first block of the first/second neural network, and the above-mentioned second layer can be considered as the first block. /Second block of the second neural network. During the training process of the first stage of the first neural network, the operation parameters in some of the preset layers may be adjusted.

Exemplary device

FIG. 6 is a schematic structural diagram of an embodiment of a neural network structure search apparatus of the present disclosure. The apparatus of this embodiment can be used to implement the above method embodiments of the present disclosure. The apparatus shown in FIG. 6 mainly includes: an acquisition module 600 , a search module 601 and a determination module 602 .

The obtaining module 600 is configured to obtain a first neural network including a plurality of blocks with different numbers of channels. At least one block in the first neural network is connected with at least three blocks, and at least one block includes a head layer for performing block channel number and spatial resolution conversion.

Optionally, the acquisition module 600 can acquire the number of channels of all blocks required by the first neural network and the number of upstream/downstream blocks connected to the block, and according to the number of upstream/downstream blocks, increment the number of channels of the block to determine all blocks. The connection relationship forms the first neural network.

Optionally, in the first neural network formed by the acquisition module 600, if there are three blocks connected in sequence, in these three blocks, the increase in the number of channels of the first block relative to the middle block is not greater than that of the middle block. The number of channels increments of the block relative to the last block.

Optionally, for the ith block in the first neural network, the acquisition module 600 may determine the number of parallel layers included in the head layer of the ith block according to the number of upstream blocks of the ith block, and determine the number of parallel layers included in the head layer of the ith block according to the number of upstream blocks of the ith block, and The number and spatial resolution, as well as the channel number and spatial resolution of each upstream block connected to the i-th block, determine the corresponding input channel number and spatial resolution, as well as the output channel number and spatial resolution of each parallel layer. Among them, different parallel layers correspond to different upstream blocks.

Optionally, the acquisition module 600 can set the operations included in each parallel layer in the head layer according to all the candidate operations in the preset candidate operation set; for any parallel layer in the i-th block in the first neural network, The process of setting the output of the parallel layer by the obtaining module 600 may include: performing calculation according to the operation weight of each candidate operation in the parallel layer and the output of each candidate operation, and obtaining the calculation result.

Optionally, for the i-th block in the first neural network, the process that the acquisition module 600 sets the output of the head layer in the i-th block may include: The corresponding block connection weight and the output of each parallel layer are calculated to obtain the calculation result.

Optionally, for the i-th block in the first neural network, the acquisition module 600 can determine the respective inputs of the stacked layers in the i-th block that are sequentially located after the head layer according to the channel number and spatial resolution of the i-th block. Number of channels and spatial resolution and number of output channels and spatial resolution.

Optionally, the obtaining module 600 may set the operations included in the stacking layers sequentially located after the head layer in each block according to all the candidate operations in the preset candidate operation set. For the jth stacked layer in the ith block in the first neural network, the process of setting the output of the jth stacked layer by the acquisition module 600 may include: according to the operation weight of each candidate operation in the jth stacked layer and the weight of each candidate operation Output, perform calculations, and obtain calculation results.

Optionally, the search module 601 in the present disclosure can also perform the following operations before using a gradient-based search strategy to perform a neural network structure search process on the first neural network according to the sample data in the first data set: The search module 601 provides the sample data in the second data set to the first neural network, and processes the sample data via the first neural network; the search module 601 uses the first loss function to adjust the processing results of the sample data and the sample data. Operation parameters for each candidate operation in each layer in each block in the first neural network.

Optionally, the search module 601 can adjust the operation parameters of each candidate operation in each layer of each block in the first neural network in two ways, so as to realize the training in the first stage.

Mode 1: The search module 601 provides the sample data in the second data set to the first neural network, and performs all candidate operations on the sample data through all candidate operations in all layers in all paths formed by all blocks in the first neural network. Process to obtain the processing result of the sample data.

Mode 2: The search module 601 provides the sample data in the second data set to the first neural network, and through a selected candidate operation in all layers in all paths formed by all blocks in the first neural network, respectively The sample data is processed to obtain the processing result of the sample data. In the case of adopting the second method, the search module 601 can use the second loss function and gradient descent method according to the processing result of the sample data and the sample data to adjust the parameters in the layers of the blocks in the first neural network. The operation parameters of the selected candidate operation.

Optionally, the search module 601 can provide the sample data in the first data set to the first neural network, and process the sample data through the first neural network, and then the search module 601 can process the sample data according to the processing result of the sample data and the sample data. Using the first loss function and gradient descent method, the block connection weight in the first neural network and the operation weight of each candidate operation in each layer in each block are updated.

Optionally, the search module 601 can implement the second stage of training in the following two ways.

Mode 1: The search module 601 provides the sample data in the first data set to the first neural network, and through all candidate operations in all layers in all paths formed by all blocks in the first neural network, respectively, for the sample The data is processed to obtain the processing result of the sample data.

Mode 2: The search module 601 provides the sample data in the first data set to the first neural network, and the selected two candidate operations in all the layers in all the paths formed by all the blocks in the first neural network, respectively. The sample data is processed to obtain the processing result of the sample data.

Optionally, the search module can perform multi-objective optimization processing according to the processing results of the sample data and the sample data, using the first loss function including the delay parameters of each layer, and adopt the gradient descent method to update the blocks in the first neural network. Connection weights and operation weights for each candidate operation in each layer in each block.

Optionally, in the case where the search module 601 adopts the above-mentioned method 2, the search module 601 may also calculate the offsets of the two selected candidate operations according to the updated operation weights; and adjust the offsets according to the offsets. The operation weights of the two selected candidate operations in each layer in each block in a neural network.

The determining module 602 is configured to determine the second neural network structure obtained by searching according to the structural parameters of the first neural network obtained by the searching module 601 .

Exemplary Electronics

An electronic device according to an embodiment of the present disclosure is described below with reference to FIG. 7 . 7 shows a block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 7 , electronic device 71 includes one or more processors 711 and memory 77 .

Processor 711 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in electronic device 71 to perform desired functions.

Memory 77 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory (cache). The non-volatile memory may include, for example, read only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 711 may execute the program instructions to implement the neural network structure search method and/or the various embodiments of the present disclosure described above. or other desired functionality. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

In one example, the electronic device 71 may further include: an input device 713 and an output device 714, etc., these components are interconnected through a bus system and/or other forms of connection mechanisms (not shown). In addition, the input device 713 may also include, for example, a keyboard, a mouse, and the like. The output device 714 can output various information to the outside. The output devices 714 may include, for example, displays, speakers, printers, and communication networks and their connected remote output devices, among others.

Of course, for simplicity, only some of the components in the electronic device 71 related to the present disclosure are shown in FIG. 7 , and components such as buses, input/output interfaces, and the like are omitted. Besides, the electronic device 71 may also include any other appropriate components according to the specific application.

Exemplary computer program product and computer readable storage medium

In addition to the methods and apparatus described above, embodiments of the present disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the "exemplary method" described above in this specification The steps in the neural network structure search method according to various embodiments of the present disclosure are described in the section.

The computer program product may write program code for performing operations of embodiments of the present disclosure in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc. , also includes conventional procedural programming languages, such as "C" language or similar programming languages. The program code may execute entirely on the user computing device, partly on the user device, as a stand-alone software package, partly on the user computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.

In addition, embodiments of the present disclosure may also be computer-readable storage media having computer program instructions stored thereon that, when executed by a processor, cause the processor to perform the above-described "Example Method" section of this specification Steps in a neural network structure search method according to various embodiments of the present disclosure described in .

The computer-readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses or devices, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media may include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), Erase programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

The basic principles of the present disclosure have been described above with reference to specific embodiments. However, it should be pointed out that the advantages, advantages, effects, etc. mentioned in the present disclosure are only examples rather than limitations, and these advantages, advantages, effects, etc. should not be considered to be A must-have for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of example and easy understanding, but not for limitation, and the above details do not limit the present disclosure to be implemented by the above specific details.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the points that are different from other embodiments, and the same or similar parts between the various embodiments may be referred to each other. As for the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.

The block diagrams of devices, apparatuses, apparatuses, and systems referred to in this disclosure are merely illustrative examples and are not intended to require or imply that the connections, arrangements, or configurations must be in the manner shown in the block diagrams. As those skilled in the art will appreciate, these devices, apparatus, apparatuses, and systems may be connected, arranged, and configured in any manner. Words such as "including," "including, "having," etc. are open-ended words meaning "including but not limited to," and are used interchangeably therewith. The words "or" and "and" as used herein refer to the words " and/or" and are used interchangeably therewith unless the context clearly dictates otherwise. The word "such as" as used herein refers to, and is used interchangeably with, the phrase "such as but not limited to".

The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Furthermore, in some embodiments, the present disclosure can also be implemented as programs recorded in a recording medium, the programs including machine-readable instructions for implementing methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

It should also be noted that, in the apparatus, device and method of the present disclosure, each component or each step may be decomposed and/or recombined. These disaggregations and/or recombinations should be considered equivalents of the present disclosure.

The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications and the like to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for the purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims (17)

1. A neural network structure search method, comprising: Obtain a first neural network including a plurality of blocks with different numbers of channels, wherein at least one block of the first neural network is connected to at least three blocks, and at least one block includes a head layer; According to the sample data in the first data set, using a gradient-based search strategy, a neural network structure search process is performed on the first neural network to obtain structural parameters of the first neural network; According to the structural parameters of the first neural network, the second neural network structure obtained by searching is determined. 2. The method according to claim 1, wherein the obtaining the first neural network comprising a plurality of blocks with different channel numbers comprises: Get the number of channels of all blocks required for the first neural network and the number of upstream/downstream blocks connected to the block; The connection relationship of all the blocks is determined according to the number of upstream/downstream blocks, and the number of channels of the blocks is incremented. 3. The method according to claim 2, wherein, among the three sequentially connected blocks in the first neural network, the increase in the number of channels of the first block relative to the middle block is not greater than that of the middle block relative to the last block Increases the number of channels for the block. 4. The method according to claim 2 or 3, wherein the acquiring the first neural network comprising a plurality of blocks with different numbers of channels comprises: For the ith block in the first neural network, determine the number of parallel layers included in the head layer of the ith block according to the number of upstream blocks of the ith block, and determine the number of parallel layers included in the head layer of the ith block according to the channel of the ith block. number and spatial resolution, as well as the number of channels and spatial resolution of each upstream block connected to the i-th block, determine the number of input channels and spatial resolution, and the number of output channels and spatial resolution corresponding to each parallel layer; Among them, different parallel layers correspond to different upstream blocks. 5. The method according to claim 4, wherein the obtaining the first neural network comprising a plurality of blocks with different numbers of channels comprises: According to all the candidate operations in the preset candidate operation set, set the operations included in each parallel layer in the head layer; For any parallel layer in the i-th block in the first neural network, setting the output of the parallel layer includes: According to the operation weight of each candidate operation in the parallel layer and the output of each candidate operation, the calculation is performed to obtain the calculation result. 6. The method according to any one of claims 1 to 5, wherein the obtaining the first neural network comprising a plurality of blocks with different numbers of channels comprises: For the ith block in the first neural network, setting the output of the head layer in the ith block includes: According to the block connection weights corresponding to each parallel layer in the head layer of the i-th block and the output of each parallel layer, the calculation is performed to obtain the calculation result. 7. The method according to any one of claims 1 to 6, wherein the obtaining the first neural network comprising a plurality of blocks with different numbers of channels comprises: For the ith block in the first neural network, according to the number of channels and the spatial resolution of the ith block, determine the number of input channels corresponding to each stacked layer in the ith block sequentially located after the head layer and spatial resolution as well as the number of output channels and spatial resolution. 8. The method according to any one of claims 1 to 7, wherein the obtaining the first neural network comprising a plurality of blocks with different numbers of channels comprises: According to all the candidate operations in the preset candidate operation set, set the operations included in the stacking layers in the blocks that are sequentially located after the head layer; For the jth stacked layer in the ith block in the first neural network, setting the output of the jth stacked layer includes: according to the operation weight of each candidate operation in the jth stacked layer and the output of each candidate operation, Do the calculation and get the calculation result. 9. The method according to any one of claims 1 to 8, wherein, according to the sample data in the first data set, the method uses a gradient-based search strategy to perform a neural network on the first neural network Structure search processing, before, also includes: providing the sample data in the second data set to the first neural network, and processing the sample data via the first neural network; According to the processing result of the sample data and the sample data, using the first loss function, the operation parameters of each candidate operation in each layer in each block in the first neural network are adjusted. 10. The method according to claim 9, wherein, providing the sample data in the second data set to the first neural network, and processing the sample data via the first neural network, comprising: Provide the sample data in the second data set to the first neural network, and process the sample data through all candidate operations in all layers in all paths formed by all blocks in the first neural network to obtain samples. the result of the processing of the data; or The sample data in the second data set is provided to the first neural network, and the sample data is processed separately through a selected candidate operation in all layers in all paths formed by all blocks in the first neural network , to obtain the processing results of the sample data; The operation parameters of each candidate operation in each layer in each block in the first neural network are adjusted by using the first loss function according to the processing result of the sample data and the sample data, including: According to the processing result of the sample data and the sample data, the second loss function is used to adjust the operation parameters of the selected candidate operations in each layer of each block in the first neural network by using a gradient descent method. 11. The method according to any one of claims 1 to 10, wherein, according to the sample data in the first data set, using a gradient-based search strategy, the first neural network is searched for a neural network structure processing, including: providing sample data in the first data set to a first neural network, and processing the sample data via the first neural network; According to the processing result of the sample data and the sample data, the first loss function is used, and the gradient descent method is used to update the block connection weight in the first neural network and the operation weight of each candidate operation in each layer in each block. 12. The method of claim 11, wherein the providing the sample data in the first data set to a first neural network, and processing the sample data via the first neural network, comprises: Provide the sample data in the first data set to the first neural network, and process the sample data through all candidate operations in all layers in all paths formed by all blocks in the first neural network to obtain samples. the result of the processing of the data; or The sample data in the first data set is provided to the first neural network, and the sample data are respectively subjected to two selected candidate operations in all layers in all paths formed by all blocks in the first neural network. Process to obtain the processing result of the sample data. 13. The method according to claim 12, wherein, according to the processing result of the sample data and the sample data, the first loss function is used to update the block connection weight and each block connection weight in the first neural network by using a gradient descent method. Operation weights for each candidate operation in each layer in the block, including: According to the processing result of the sample data and the sample data, the first loss function including the delay parameters of each layer is used, and the gradient descent method is used to perform multi-objective optimization processing, so as to update the block connection weights and the block connection weights in the first neural network. The operation weight of each candidate operation in each layer in . 14. The method according to claim 13, wherein, according to the processing result of the sample data and the sample data, the second loss function is used to update the block connection weight and each block connection weight in the first neural network by using a gradient descent method. Operation weights for each candidate operation in each layer in the block, also including: Calculate the offsets of the two selected candidate operations according to the updated operation weights; According to the offset, the operation weights of the selected two candidate operations in each layer in each block in the first neural network are adjusted. 15. A neural network structure search device, comprising: An acquisition module for acquiring a first neural network including a plurality of blocks with different numbers of channels, wherein at least one block of the first neural network is connected to at least three blocks, and at least one block includes a number of channels for performing block and Head layer for spatial resolution conversion; A search module, configured to perform a neural network structure search process on the first neural network acquired by the acquisition module according to the sample data in the first data set, using a gradient-based search strategy, to obtain structural parameters of the first neural network ; A determining module, configured to determine the second neural network structure obtained by searching according to the structural parameters of the first neural network obtained by the searching module. 16. A computer-readable storage medium storing a computer program for performing the method of any one of the preceding claims 1-14. 17. An electronic device comprising: processor; a memory for storing the processor-executable instructions; The processor, configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of the preceding claims 1-14.

Images