WO2021008206A1 - 神经网络结构的搜索方法、图像处理方法和装置 - Google Patents

神经网络结构的搜索方法、图像处理方法和装置 Download PDF

Info

Publication number
WO2021008206A1
WO2021008206A1 PCT/CN2020/089403 CN2020089403W WO2021008206A1 WO 2021008206 A1 WO2021008206 A1 WO 2021008206A1 CN 2020089403 W CN2020089403 W CN 2020089403W WO 2021008206 A1 WO2021008206 A1 WO 2021008206A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature map
node
neural network
processed
input
Prior art date
Application number
PCT/CN2020/089403
Other languages
English (en)
French (fr)
Inventor
徐宇辉
谢凌曦
张晓鹏
陈鑫
齐国君
田奇
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021008206A1 publication Critical patent/WO2021008206A1/zh
Priority to US17/573,220 priority Critical patent/US12026938B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means

Definitions

  • This application relates to the field of artificial intelligence, and more specifically, to a search method, image processing method, and device of a neural network structure.
  • Artificial intelligence is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.
  • artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a similar way to human intelligence.
  • Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.
  • Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theories.
  • neural networks for example, deep neural networks
  • a neural network with good performance often has a sophisticated network structure, which requires human experts with superb skills and rich experience to spend a lot of energy to construct.
  • NAS neural architecture search
  • a search network is generally constructed based on a certain number of building units, and then the connection relationship between the nodes of the building units in the search network is optimized in the search space to obtain the optimized building units, and finally Build the target neural network according to the optimized building unit.
  • the output feature map of each node needs to be processed through all optional operations, and the amount of data to be processed is large, and the search overhead is high.
  • This application provides a neural network structure search method, image processing method, device, computer readable storage medium and chip to reduce search overhead.
  • a search method of a neural network structure includes: determining a search space and multiple building units; stacking multiple building units to obtain a search network; The network structure is optimized, and the optimized building unit is obtained; the target neural network is built according to the optimized building unit.
  • the above-mentioned building unit is a network structure obtained by connecting multiple nodes through the basic operation of a neural network.
  • the above-mentioned building unit is a basic module for building a neural network.
  • the building unit can be used to build a neural network.
  • the partial channel of the output feature map of each node is processed by the candidate operation to obtain the processed feature map.
  • the input feature map of the next node of each node is A feature map obtained by splicing the processed feature map with the remaining feature maps of each node that have not been processed by the candidate operation.
  • the next node of each of the foregoing nodes may refer to a node where each node has a connection relationship, and a partial channel of the output feature map of each node is processed by the waiting operation to obtain the input node of the feature map.
  • the output feature map of each node has c channels, and the c/K channels of the output feature map of each node are processed by the to-be-selected operation to be processed
  • the input feature map of the next node of each node is a feature map obtained by splicing the processed feature map and the remaining feature map.
  • the processed feature map has c/K channels, and the remaining feature map has c(K-1)/K channels.
  • the remaining feature map is the feature map output by each node that has not been processed by the operation to be selected. Both c and K are integers greater than 1, and the foregoing candidate operations include all operations in the search space.
  • each construction unit of the above multiple construction units only part of the channel of the output feature map of each node has been processed by the candidate operation, and the processed feature map and the remaining feature maps are input to the The next node of each node is used as the input feature map of the next node.
  • a construction unit contains the first node and the second node, the second node is the next node of the first node, the feature graph output by the first node has c channels, and the feature graph output by the first node is c/K Two channels are processed to be selected to obtain the processed feature map.
  • the input feature map of the second node is the feature map obtained by splicing the processed feature map and the remaining feature map.
  • the remaining feature map is the unprocessed feature map output by the first node.
  • the feature map processed by the operation to be selected.
  • the aforementioned search space is determined according to the application requirements of the target neural network to be constructed.
  • the aforementioned search space may be determined according to the type of processed data of the target neural network.
  • the type and number of operations contained in the search space should be adapted to the processing of image data.
  • the aforementioned search space may include convolution operations, pooling operations, skip-connect operations, and so on.
  • the target neural network is a neural network for processing voice data
  • the type and number of operations contained in the search space should be adapted to the processing of voice data.
  • the target neural network is a neural network for processing voice data
  • the above search space may include activation functions (such as ReLU, Tanh) and so on.
  • the above-mentioned search space is determined according to the application requirements of the target neural network and the video memory resource conditions of the device performing the neural network structure search.
  • the above video memory resource condition of the device that performs the neural network structure search may refer to the size of the video memory resource of the device that performs the neural network structure search.
  • the types and numbers of operations included in the above search space can be comprehensively determined according to the application requirements of the target neural network and the video memory resource conditions of the device performing the neural network structure search.
  • the types and numbers of operations included in the search space can be determined according to the application requirements of the target neural network, and then the types and numbers of operations included in the search space can be adjusted in combination with the video memory resource conditions of the device performing the neural network structure search to determine the search The type and number of operations that the space ultimately contains.
  • the device performing the neural network structure search has less video memory resources, then some less important operations in the search space can be deleted ; And if the video memory resources of the device performing the neural network structure search are sufficient, the types and numbers of operations included in the search space can be maintained, or the types and numbers of operations included in the search space can be increased.
  • the above-mentioned video memory resource can be replaced with a cache resource.
  • the cache resource is a memory or storage unit used to store computing data in the optimization process of the equipment used to construct the neural network.
  • the cache resource may specifically include video memory resources.
  • the number of the aforementioned building units is determined according to the video memory resource condition of the device performing the neural network structure search.
  • the number of building units can be less, and when the device performing the neural network structure search has sufficient video memory resources, the number of building units can be more.
  • the number of the aforementioned construction units is determined according to the application requirements of the target neural network to be constructed and the video memory resource conditions of the device performing the neural network structure search.
  • the initial number of construction units can be determined according to the application requirements of the target neural network, and then the initial number of construction units can be further adjusted according to the video memory resources of the device performing the neural network structure search, thereby determining the final number of construction units.
  • the initial number of building units is the final number of building units.
  • the output feature map of each node in the construction unit since the output feature map of each node in the construction unit has only partial channel feature maps for processing to be selected, the number of channels of feature maps to be selected can be reduced. In turn, the video memory occupied during the search is reduced, and the search overhead is reduced.
  • this application can reduce the video memory occupied in the search process, this application can increase the data volume of each batch of data processed in the network search process under the same video memory resources, so as to achieve a more complex neural network structure. search for.
  • the foregoing stacking of multiple building units to obtain a search network includes: stacking the multiple building units in sequence in a preset stacking manner to obtain a search network, wherein, in the search network, the search network is located The output of the building unit in front of the network is the input of the building unit located in the back of the search network.
  • the foregoing preset stacking manner may include information such as where and what type of building units are stacked and the number of stacks.
  • the input feature map of the next node of each node is a feature map obtained by splicing the processed feature map and the remaining feature map, including: each node
  • the input feature map of the next node is the feature map obtained after the processed feature map and the remaining feature map are spliced and the channel sequence is exchanged.
  • the aforementioned switching of the channel sequence may be to readjust the channel sequence of the spliced feature maps.
  • the processed feature map can be spliced with the remaining feature maps to obtain the spliced feature map, and the channel order of the spliced feature map is exchanged, and then the channel order is exchanged
  • the obtained feature map is input to the next node.
  • the channel order can be exchanged first, and then the feature maps obtained after the channel order exchange are spliced together, and then the spliced feature map is input to the next node for processing .
  • the channel order of the spliced feature maps can be shuffled, so that the next node can randomly select the feature maps of some channels for processing, which can enhance the input data In order to avoid over-fitting in the final target neural network as much as possible.
  • each of the above-mentioned multiple building units includes an input node and a plurality of intermediate nodes, and the connection between the nodes of each building unit constitutes an edge, the The input of each intermediate node in the multiple intermediate nodes is the sum of the products of the corresponding multiple input feature maps and the respective corresponding edge weight parameters, where each input corresponds to an edge weight parameter, and each input corresponds to an edge
  • the weight parameter is used to indicate the weight of each input to each intermediate node.
  • the number of the aforementioned input nodes can be one or more.
  • the edge weight parameters by setting the edge weight parameters, the importance of different edges can be measured, and during the optimization process, the edges with the corresponding edge weight parameter values can be selected, and the edges with the smaller edge weight parameter values can be discarded, so that The finally constructed target neural network maintains certain stability.
  • the network structure of the building unit in the search network is optimized in the search space to obtain the optimized building unit, including: in the search space, The network structure parameters of the building units in the search network are adjusted to obtain the optimized building units.
  • the network structure parameters of the above-mentioned construction unit include the weight parameters of the candidate operations and the weight parameters of the edges.
  • the optimized construction unit retains the edges corresponding to some of the edge weight parameters, and the optimized construction unit retains the operations corresponding to the weight parameters of the selected operations. .
  • edges corresponding to the weight parameter values of the edges with larger parameter values may be retained, and some operations corresponding to the weight parameters of the candidate operations with larger parameter values may be retained.
  • each of the above-mentioned multiple building units includes an input node and multiple intermediate nodes, and each intermediate node of the multiple intermediate nodes corresponds to a level.
  • the intermediate node of the first level is connected to the input node
  • the intermediate node of the i-th level is connected to the input node
  • the intermediate node of the i-th level is connected to the intermediate nodes of the previous i-1 levels
  • i is an integer greater than 1.
  • the number of the above-mentioned input nodes may be one or more.
  • the intermediate node of the first level is connected to each input node.
  • K is determined according to the size of the video memory resource of the device that executes the above method.
  • the value of K can be set as a multiple of 2, for example, 2, 4, 6, 8, and so on.
  • K when the video memory resource of the device that executes the above method is large, K can be set to a smaller value, and when the video memory resource of the device that executes the above method is less, K can be set to a larger value.
  • the value of K is 4, and when the video memory resource of the device that executes the above method is small, the value of K is 8.
  • the value of K may be set first according to the size of the video memory resources of the device that executes the above method, and then the value of K may be adjusted according to the performance of the target neural network in the case of the K value.
  • an image processing method includes: acquiring an image to be processed; and classifying the image to be processed according to a target neural network to obtain a classification result of the image to be processed.
  • the target neural network is constructed by multiple optimized building units, and the multiple optimized building units are obtained by optimizing the network structure of multiple building units in the search network.
  • the c/K channels of the output feature map of each node are processed by the candidate operation to obtain the processed feature map.
  • the input feature map of the next node of each node is the processed feature map and the remaining
  • the feature map obtained after the feature map is spliced, the output feature map of each node has c channels, the processed feature map has c/K channels, the remaining feature map has c(K-1)/K channels, and the remaining features
  • the figure is a feature graph output by each node that has not been processed by the operation to be selected.
  • the operation to be selected includes all operations in the search space. Both c and K are integers greater than 1.
  • the target neural network is constructed by optimized construction units, and the target neural network may be trained on training data (including training images and classification results of training images). Neural network.
  • the target neural network processes the image to be processed, all channels of the feature map output by each node in each optimized construction unit will be processed by the candidate operation, and the feature map processed by the candidate operation will be input to the next A node, which is the difference between using the target neural network for image processing and neural network structure search processing to obtain the target neural network.
  • the input feature map of the next node of each node is the feature map obtained by splicing the processed feature map and the remaining feature map, including:
  • the input feature map of the next node is the feature map obtained after the processed feature map and the remaining feature map are spliced and the channel sequence is exchanged.
  • the target neural network since the target neural network is constructed in the process of changing the order of the channels of the spliced feature maps, the channel order of the spliced feature maps can be disrupted, so that the next node can randomly select part of the channel.
  • the feature map processing can enhance the randomness of the input data, so as to avoid the over-fitting phenomenon of the final target neural network as much as possible. Therefore, the use of the target neural network can better perform image classification.
  • each of the multiple building units includes an input node and multiple intermediate nodes, and the connections between the nodes of each building unit constitute edges, and multiple The input of each intermediate node in the intermediate node is the sum of the products of the corresponding multiple inputs and the respective corresponding edge weight parameters, where each input corresponds to an edge weight parameter, and the edge weight parameter corresponding to each input is used for Indicates the weight of each input to each intermediate node.
  • the target neural network is constructed in the process of setting edge weight parameters, the importance of different edges can be measured, and during the optimization process, the corresponding edge with a relatively large edge weight parameter value can be selected, and the edge weight can be discarded.
  • the edge of the parameter value is relatively small, so that the finally constructed target neural network maintains a certain stability.
  • each of the multiple building units includes an input node and multiple intermediate nodes, and each intermediate node of the multiple intermediate nodes corresponds to a level, where , The intermediate node of the first level is connected to the input node, the intermediate node of the i-th level is connected to the input node, the intermediate node of the i-th level is connected to the intermediate nodes of the previous i-1 levels, i is an integer greater than 1.
  • an image processing method includes: acquiring an image to be processed; processing the image to be processed according to a target neural network to obtain a processing result of the image to be processed.
  • the above-mentioned image processing may refer to image recognition, classification, detection and so on.
  • an image processing method includes: obtaining a road image; performing convolution processing on the road image according to a target neural network to obtain multiple convolution feature maps of the road image; and processing the road image according to the target neural network Deconvolution processing is performed on multiple convolution feature maps of to obtain the semantic segmentation result of the road image.
  • the above-mentioned target neural network is a target neural network constructed according to any one of the realization methods in the first aspect.
  • an image processing method includes: obtaining a face image; performing convolution processing on the face image according to a target neural network to obtain a convolution feature map of the face image; The product feature map is compared with the convolution feature map of the ID image to obtain the verification result of the face image.
  • the convolution feature map of the aforementioned ID image may be obtained in advance and stored in the corresponding database. For example, perform convolution processing on the image of the ID document in advance, and store the obtained convolution feature map in the database.
  • the above-mentioned target neural network is a target neural network constructed according to any one of the implementation methods in the first aspect.
  • a neural network structure search device in a sixth aspect, includes: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, The processor is used to execute the method in any one of the implementation manners in the first aspect.
  • an image processing device which includes: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processing The device is used to execute the method in any one of the second aspect to the fifth aspect.
  • a computer-readable medium stores program code for device execution, and the program code includes a method for executing any one of the first to fifth aspects. .
  • a computer program product containing instructions is provided.
  • the computer program product runs on a computer, the computer executes the method in any one of the foregoing first to fifth aspects.
  • a chip in a tenth aspect, includes a processor and a data interface.
  • the processor reads instructions stored in a memory through the data interface, and executes any one of the first to fifth aspects above The method in the implementation mode.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute the instructions stored in the memory.
  • the processor is configured to execute the method in any one of the implementation manners of the first aspect to the fifth aspect.
  • FIG. 1 is a schematic diagram of an artificial intelligence main body framework provided by an embodiment of the present application
  • Figure 2 is a schematic diagram of a specific application provided by an embodiment of the application.
  • FIG. 3 is a schematic diagram of a specific application provided by an embodiment of the application.
  • FIG. 4 is a schematic structural diagram of a system architecture provided by an embodiment of the application.
  • FIG. 5 is a schematic structural diagram of a convolutional neural network provided by an embodiment of the application.
  • FIG. 6 is a schematic structural diagram of a convolutional neural network provided by an embodiment of this application.
  • FIG. 7 is a schematic diagram of the hardware structure of a chip provided by an embodiment of the application.
  • FIG. 8 is a schematic diagram of a system architecture provided by an embodiment of the application.
  • FIG. 9 is a schematic flowchart of a search method of a neural network structure according to an embodiment of the present application.
  • FIG. 10 is a schematic diagram of a construction unit of an embodiment of the present application.
  • FIG. 11 is a schematic diagram of the processing procedure of the feature map
  • Figure 12 is a schematic diagram of the channel sequence exchange of the feature map
  • Figure 13 is a schematic diagram of the channel sequence exchange of the feature map
  • Figure 14 is a schematic diagram of input nodes after different feature maps are superimposed
  • Figure 15 is a schematic diagram of the structure of the search network
  • 16 is a schematic diagram of a search method of a neural network structure according to an embodiment of the present application.
  • FIG. 17 is a schematic diagram of a search method of a neural network structure according to an embodiment of the present application.
  • FIG. 18 is a schematic flowchart of an image processing method according to an embodiment of the present application.
  • Fig. 19 is a schematic block diagram of a neural network structure search device according to an embodiment of the present application.
  • 20 is a schematic block diagram of an image processing device according to an embodiment of the present application.
  • FIG. 21 is a schematic block diagram of a neural network training device according to an embodiment of the present application.
  • Figure 1 shows a schematic diagram of an artificial intelligence main framework, which describes the overall workflow of the artificial intelligence system and is suitable for general artificial intelligence field requirements.
  • Intelligent Information Chain reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has gone through the condensing process of "data-information-knowledge-wisdom".
  • Infrastructure provides computing power support for artificial intelligence systems, realizes communication with the outside world, and realizes support through the basic platform.
  • the infrastructure can communicate with the outside through sensors, and the computing power of the infrastructure can be provided by smart chips.
  • the smart chip here can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), and an application specific integrated circuit (application specific).
  • Hardware acceleration chips such as integrated circuit (ASIC) and field programmable gate array (FPGA).
  • the basic platform of infrastructure can include distributed computing framework and network and other related platform guarantees and support, and can include cloud storage and computing, interconnection networks, etc.
  • data can be obtained through sensors and external communication, and then these data can be provided to the smart chip in the distributed computing system provided by the basic platform for calculation.
  • the data in the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence.
  • This data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.
  • the aforementioned data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other processing methods.
  • machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, training, etc.
  • Reasoning refers to the process of simulating human intelligent reasoning in a computer or intelligent system, using formal information to conduct machine thinking and solving problems based on reasoning control strategies.
  • the typical function is search and matching.
  • Decision-making refers to the decision-making process of intelligent information after reasoning, and usually provides functions such as classification, ranking, and prediction.
  • some general capabilities can be formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image Recognition and so on.
  • Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. It is an encapsulation of the overall solution of artificial intelligence, productizing intelligent information decision-making and realizing landing applications. Its application fields mainly include: intelligent manufacturing, intelligent transportation, Smart home, smart medical, smart security, autonomous driving, safe city, smart terminal, etc.
  • the embodiments of this application can be applied to many fields in artificial intelligence, for example, smart manufacturing, smart transportation, smart home, smart medical care, smart security, automatic driving, safe cities and other fields.
  • the embodiments of the present application can be specifically applied in fields that require the use of (deep) neural networks, such as image classification, image retrieval, image semantic segmentation, image super-resolution, and natural language processing.
  • deep neural networks such as image classification, image retrieval, image semantic segmentation, image super-resolution, and natural language processing.
  • recognizing the images in the album can facilitate the user or the system to classify and manage the album and improve the user experience.
  • the neural network structure search method of the embodiment of the present application can search for a neural network structure suitable for album classification, and then train the neural network according to the training pictures in the training picture library to obtain the album classification neural network.
  • the album classification neural network can be used to classify the pictures, so that different categories of pictures can be labeled for users to view and find.
  • the classification tags of these pictures can also be provided to the album management system for classification management, saving users management time, improving the efficiency of album management, and enhancing user experience.
  • a neural network suitable for album classification can be constructed by a neural network structure search system (corresponding to the neural network structure search method in the embodiment of the present application).
  • the network structure of the building unit in the search network can be optimized by using the training image library to obtain the optimized building unit, and then the optimized building unit can be used to build the neural network.
  • the neural network can be trained according to the training pictures to obtain the album classification neural network.
  • the album classification neural network processes the input pictures, and the picture category is tulip.
  • a neural network suitable for data processing in an autonomous driving scenario can be constructed, and then the neural network can be trained through the data in the autonomous driving scenario to obtain The sensor data processing network can finally use the sensor processing network to process the input road images to identify different objects in the road images.
  • the neural network structure search system can construct a neural network according to the vehicle detection task.
  • the sensor data can be used to optimize the network structure of the building units in the search network to obtain the optimized Build a unit, and then use the optimized building unit to build a neural network.
  • the neural network can be trained according to the sensor data to obtain the sensor data processing network.
  • the sensor data processing network processes the input road picture, and can identify the vehicle in the road picture (as shown in the rectangular frame in the lower right corner of Fig. 3).
  • a neural network can be composed of neural units.
  • a neural unit can refer to an arithmetic unit that takes x s and intercept 1 as inputs.
  • the output of the arithmetic unit can be:
  • s 1, 2,...n, n is a natural number greater than 1
  • W s is the weight of x s
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field.
  • the local receptive field can be a region composed of several neural units.
  • Deep neural network also called multi-layer neural network
  • DNN can be understood as a neural network with multiple hidden layers.
  • DNN is divided according to the position of different layers.
  • the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer.
  • the first layer is the input layer
  • the last layer is the output layer
  • the number of layers in the middle are all hidden layers.
  • the layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer.
  • DNN looks complicated, it is not complicated in terms of the work of each layer. Simply put, it is the following linear relationship expression: among them, Is the input vector, Is the output vector, Is the offset vector, W is the weight matrix (also called coefficient), and ⁇ () is the activation function.
  • Each layer is just the input vector After such a simple operation, the output vector is obtained Due to the large number of DNN layers, the coefficient W and the offset vector The number is also relatively large.
  • the definition of these parameters in the DNN is as follows: Take the coefficient W as an example: Suppose that in a three-layer DNN, the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.
  • the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as
  • the input layer has no W parameter.
  • more hidden layers make the network more capable of portraying complex situations in the real world. Theoretically speaking, a model with more parameters is more complex and has a greater "capacity", which means it can complete more complex learning tasks.
  • Training a deep neural network is also a process of learning a weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by vectors W of many layers).
  • Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with convolutional structure.
  • the convolutional neural network contains a feature extractor composed of a convolution layer and a sub-sampling layer.
  • the feature extractor can be regarded as a filter.
  • the convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network.
  • a neuron can be connected to only part of the neighboring neurons.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels.
  • Sharing weight can be understood as the way to extract image information has nothing to do with location.
  • the convolution kernel can be initialized in the form of a matrix of random size. During the training of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.
  • RNN Recurrent Neural Networks
  • RNN can process sequence data of any length.
  • the training of RNN is the same as the training of traditional CNN or DNN.
  • the neural network can use an error back propagation (BP) algorithm to modify the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forwarding the input signal to the output will cause error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss is converged.
  • the backpropagation algorithm is a backpropagation motion dominated by error loss, and aims to obtain the optimal neural network model parameters, such as the weight matrix.
  • an embodiment of the present application provides a system architecture 100.
  • the data collection device 160 is used to collect training data.
  • the training data may include training images and classification results corresponding to the training images, where the results of the training images may be manually pre-labeled results.
  • the data collection device 160 stores the training data in the database 130, and the training device 120 trains to obtain the target model/rule 101 based on the training data maintained in the database 130.
  • the training device 120 processes the input original image and compares the output image with the original image until the output image of the training device 120 differs from the original image. The difference is less than a certain threshold, thereby completing the training of the target model/rule 101.
  • the above-mentioned target model/rule 101 can be used to implement the image processing method of the embodiment of the present application.
  • the target model/rule 101 in the embodiment of the present application may specifically be a neural network.
  • the training data maintained in the database 130 may not all come from the collection of the data collection device 160, and may also be received from other devices.
  • the training device 120 does not necessarily perform the training of the target model/rule 101 completely based on the training data maintained by the database 130. It may also obtain training data from the cloud or other places for model training.
  • the above description should not be used as a reference to this application. Limitations of Examples.
  • the target model/rule 101 trained according to the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG. 4, which can be a terminal, such as a mobile phone terminal, a tablet computer, notebook computers, augmented reality (AR) AR/virtual reality (VR), vehicle-mounted terminals, etc., can also be servers or clouds.
  • the execution device 110 is configured with an input/output (input/output, I/O) interface 112 for data interaction with external devices.
  • the user can input data to the I/O interface 112 through the client device 140.
  • the input data in this embodiment of the application may include: the image to be processed input by the client device.
  • the preprocessing module 113 and the preprocessing module 114 are used for preprocessing according to the input data (such as the image to be processed) received by the I/O interface 112.
  • the preprocessing module 113 and the preprocessing module may not be provided.
  • 114 there may only be one preprocessing module, and the calculation module 111 is directly used to process the input data.
  • the execution device 110 may call data, codes, etc. in the data storage system 150 for corresponding processing .
  • the data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 150.
  • the I/O interface 112 returns the processing result, such as the denoising processed image obtained as described above, to the client device 140 to provide it to the user.
  • the training device 120 can generate corresponding target models/rules 101 based on different training data for different goals or tasks, and the corresponding target models/rules 101 can be used to achieve the above goals or complete The above tasks provide the user with the desired result.
  • the user can manually set input data, and the manual setting can be operated through the interface provided by the I/O interface 112.
  • the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send the input data and the user's authorization is required, the user can set the corresponding authority in the client device 140.
  • the user can view the result output by the execution device 110 on the client device 140, and the specific presentation form may be a specific manner such as display, sound, and action.
  • the client device 140 can also be used as a data collection terminal to collect the input data of the input I/O interface 112 and the output result of the output I/O interface 112 as new sample data, and store it in the database 130 as shown in the figure.
  • the I/O interface 112 directly uses the input data input to the I/O interface 112 and the output result of the output I/O interface 112 as a new sample as shown in the figure.
  • the data is stored in the database 130.
  • FIG. 4 is only a schematic diagram of a system architecture provided by an embodiment of the present application.
  • the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data The storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.
  • the target model/rule 101 is obtained by training according to the training device 120.
  • the target model/rule 101 may be the neural network in this application in the embodiment of this application, specifically, the neural network provided in the embodiment of this application Can be CNN, deep convolutional neural networks (deep convolutional neural networks, DCNN), recurrent neural networks (recurrent neural network, RNNS) and so on.
  • CNN is a very common neural network
  • the structure of CNN will be introduced in detail below in conjunction with Figure 5.
  • a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture.
  • a deep learning architecture refers to a machine learning algorithm. Multi-level learning is carried out on the abstract level of
  • CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the input image.
  • a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer/pooling layer 220 (where the pooling layer is optional), and a neural network layer 230.
  • the input layer 210 can obtain the image to be processed, and pass the obtained image to be processed to the convolutional layer/pooling layer 220 and the subsequent neural network layer 230 for processing, and the image processing result can be obtained.
  • the convolutional layer/pooling layer 220 may include layers 221-226, for example: in an implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, and layer 223 is a convolutional layer. Layers, 224 is the pooling layer, 225 is the convolutional layer, and 226 is the pooling layer; in another implementation, 221 and 222 are the convolutional layers, 223 is the pooling layer, and 224 and 225 are the convolutional layers. Layer, 226 is the pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.
  • the convolution layer 221 can include many convolution operators.
  • the convolution operator is also called a kernel. Its function in image processing is equivalent to a filter that extracts specific information from the input image matrix.
  • the convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. In the process of convolution on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. ...It depends on the value of stride) to complete the work of extracting specific features from the image.
  • the size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix and the depth dimension of the input image are the same.
  • the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension convolution output, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row ⁇ column) are applied. That is, multiple homogeneous matrices.
  • the output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" mentioned above.
  • Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to eliminate unwanted noise in the image.
  • the multiple weight matrices have the same size (row ⁇ column), the size of the convolution feature maps extracted by the multiple weight matrices of the same size are also the same, and then the multiple extracted convolution feature maps of the same size are combined to form The output of the convolution operation.
  • weight values in these weight matrices need to be obtained through a lot of training in practical applications.
  • Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions. .
  • the initial convolutional layer (such as 221) often extracts more general features, which can also be called low-level features; with the convolutional neural network
  • the features extracted by the subsequent convolutional layers (for example, 226) become more and more complex, such as features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.
  • the 221-226 layers as illustrated by 220 in Figure 5 can be a convolutional layer followed by a layer
  • the pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers.
  • the only purpose of the pooling layer is to reduce the size of the image space.
  • the pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain a smaller size image.
  • the average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling.
  • the maximum pooling operator can take the pixel with the largest value within a specific range as the result of maximum pooling.
  • the operators in the pooling layer should also be related to the image size.
  • the size of the image output after processing by the pooling layer can be smaller than the size of the image of the input pooling layer, and each pixel in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.
  • the convolutional neural network 200 After processing by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 200 needs to use the neural network layer 230 to generate one or a group of required classes of output. Therefore, the neural network layer 230 may include multiple hidden layers (231, 232 to 23n as shown in FIG. 5) and an output layer 240. The parameters contained in the multiple hidden layers can be based on specific task types. The relevant training data of the, for example, the task type can include image recognition, image classification, image super-resolution reconstruction and so on.
  • the output layer 240 After the multiple hidden layers in the neural network layer 230, that is, the final layer of the entire convolutional neural network 200 is the output layer 240.
  • the output layer 240 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error.
  • a convolutional neural network (CNN) 200 may include an input layer 110, a convolutional layer/pooling layer 120 (the pooling layer is optional), and a neural network layer 130.
  • CNN convolutional neural network
  • FIG. 5 multiple convolutional layers/pooling layers in the convolutional layer/pooling layer 120 in FIG. 6 are parallel, and the respectively extracted features are input to the full neural network layer 130 for processing.
  • the convolutional neural network shown in FIGS. 5 and 6 is only used as an example of two possible convolutional neural networks in the image processing method of the embodiment of the application.
  • the application implements
  • the convolutional neural network used in the image processing method of the example can also exist in the form of other network models.
  • the structure of the convolutional neural network obtained by the search method of the neural network structure of the embodiment of the present application may be as shown in the convolutional neural network structure in FIG. 5 and FIG. 6.
  • FIG. 7 is a hardware structure of a chip provided by an embodiment of the application.
  • the chip includes a neural network processor 50.
  • the chip may be set in the execution device 110 as shown in FIG. 1 to complete the calculation work of the calculation module 111.
  • the chip can also be set in the training device 120 as shown in FIG. 1 to complete the training work of the training device 120 and output the target model/rule 101.
  • the algorithms of each layer in the convolutional neural network as shown in FIG. 5 or FIG. 6 can be implemented in the chip as shown in FIG. 7.
  • the neural network processor NPU 50 NPU is mounted as a coprocessor to a main central processing unit (central processing unit, CPU) (host CPU), and the main CPU distributes tasks.
  • the core part of the NPU is the arithmetic circuit 50.
  • the controller 504 controls the arithmetic circuit 503 to extract data from the memory (weight memory or input memory) and perform calculations.
  • the arithmetic circuit 503 includes multiple processing units (process engines, PE). In some implementations, the arithmetic circuit 503 is a two-dimensional systolic array. The arithmetic circuit 503 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.
  • the arithmetic circuit fetches the corresponding data of matrix B from the weight memory 502 and buffers it on each PE in the arithmetic circuit.
  • the arithmetic circuit fetches matrix A data and matrix B from the input memory 501 to perform matrix operations, and the partial or final result of the obtained matrix is stored in an accumulator 508.
  • the vector calculation unit 507 can perform further processing on the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on.
  • the vector calculation unit 507 can be used for network calculations in the non-convolutional/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .
  • the vector calculation unit 507 can store the processed output vector in the unified buffer 506.
  • the vector calculation unit 507 may apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate the activation value.
  • the vector calculation unit 507 generates a normalized value, a combined value, or both.
  • the processed output vector can be used as an activation input to the arithmetic circuit 503, for example for use in subsequent layers in a neural network.
  • the unified memory 506 is used to store input data and output data.
  • the weight data directly transfers the input data in the external memory to the input memory 501 and/or the unified memory 506 through the storage unit access controller 505 (direct memory access controller, DMAC), and stores the weight data in the external memory into the weight memory 502, And the data in the unified memory 506 is stored in the external memory.
  • DMAC direct memory access controller
  • the bus interface unit (BIU) 510 is used to implement interaction between the main CPU, the DMAC, and the fetch memory 509 through the bus.
  • An instruction fetch buffer 509 connected to the controller 504 is used to store instructions used by the controller 504;
  • the controller 504 is configured to call the instructions cached in the memory 509 to control the working process of the computing accelerator.
  • the unified memory 506, the input memory 501, the weight memory 502, and the instruction fetch memory 509 are all on-chip (On-Chip) memories, and the external memory is a memory external to the NPU.
  • the external memory can be a double data rate synchronous dynamic random access memory. Memory (double data rate synchronous dynamic random access memory, referred to as DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.
  • DDR SDRAM double data rate synchronous dynamic random access memory
  • HBM high bandwidth memory
  • each layer in the convolutional neural network shown in FIG. 5 or FIG. 6 can be executed by the arithmetic circuit 303 or the vector calculation unit 307.
  • the execution device 110 in FIG. 4 introduced above can execute each step of the image processing method of the embodiment of this application.
  • the CNN model shown in FIGS. 5 and 6 and the chip shown in FIG. 7 can also be used to execute the implementation of this application. Examples of the various steps of the image processing method.
  • the image processing method of the embodiment of the present application and the image processing method of the embodiment of the present application will be described in detail below with reference to the accompanying drawings.
  • an embodiment of the present application provides a system architecture 300.
  • the system architecture includes a local device 301, a local device 302, an execution device 210 and a data storage system 250, where the local device 301 and the local device 302 are connected to the execution device 210 through a communication network.
  • the execution device 210 may be implemented by one or more servers.
  • the execution device 210 can be used in conjunction with other computing devices, such as data storage, routers, load balancers and other devices.
  • the execution device 210 may be arranged on one physical site or distributed on multiple physical sites.
  • the execution device 210 may use the data in the data storage system 250 or call the program code in the data storage system 250 to implement the method for searching the neural network structure of the embodiment of the present application.
  • the execution device 210 may perform the following process: determine a search space and multiple building units; stack the multiple building units to obtain a search network, which is a neural network used to search for a neural network structure;
  • the network structure of the building units in the search network is optimized in the search space to obtain optimized building units, wherein the search space gradually decreases during the optimization process, the number of building units gradually increases, and the search space decreases
  • the increase in the number of construction units makes the video memory consumption generated in the optimization process within a preset range
  • the target neural network is built according to the optimized construction unit.
  • a target neural network can be built, and the target neural network can be used for image classification or image processing.
  • Each local device can represent any computing device, such as personal computers, computer workstations, smart phones, tablets, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices, set-top boxes, game consoles, etc.
  • Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism/communication standard.
  • the communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.
  • the local device 301 and the local device 302 obtain the relevant parameters of the target neural network from the execution device 210, deploy the target neural network on the local device 301 and the local device 302, and use the target neural network for image classification Or image processing and so on.
  • the target neural network can be directly deployed on the execution device 210.
  • the execution device 210 obtains the image to be processed from the local device 301 and the local device 302, and classifies the image to be processed according to the target neural network or other types of images. deal with.
  • the above-mentioned execution device 210 may also be referred to as a cloud device. At this time, the execution device 210 is generally deployed in the cloud.
  • the method shown in FIG. 9 can be executed by a neural network structure search device.
  • the neural network structure search device can be a computer, a server, a cloud device, and other devices with sufficient computing power to realize a neural network structure search.
  • the method shown in FIG. 9 includes steps 1001 to 1004, which are described in detail below.
  • the aforementioned search space is determined according to the application requirements of the target neural network to be constructed.
  • the above-mentioned search space can be determined according to the application requirements of the target neural network. Specifically, the aforementioned search space can be determined according to the data type of the data to be processed by the target neural network.
  • the type and number of operations contained in the search space should match the application requirements of the target neural network.
  • the types and numbers of operations included in the search space should be adapted to the processing of image data.
  • the target neural network is used to process voice data
  • the types and numbers of operations included in the search space The quantity should be adapted to the processing of voice data.
  • the aforementioned search space may include convolution operations, pooling operations, skip-connect operations, and so on.
  • the target neural network is a neural network for processing voice data
  • the above search space may include activation functions (such as ReLU, Tanh), and so on.
  • the above-mentioned search space is determined according to the application requirements of the target neural network and the video memory resource conditions of the device performing the neural network structure search.
  • the video memory resource condition of the device performing the neural network structure search may refer to the size of the video memory resource of the device performing the neural network structure search.
  • the above-mentioned search space can be comprehensively determined according to the application requirements of the target neural network and the video memory resource conditions of the device performing the neural network structure search.
  • the type and number of operations included in the search space can be determined according to the application requirements of the target neural network, and then the types and number of operations included in the search space can be adjusted according to the video memory resource conditions of the device performing the neural network structure search to determine The type and number of operations ultimately contained in the search space.
  • the device performing the neural network structure search has less video memory resources, then some less important operations in the search space can be deleted . If the video memory resources of the device performing the neural network structure search are sufficient, the types and numbers of operations included in the search space can be maintained, or the types and numbers of operations included in the search space can be increased.
  • the number of the aforementioned building units is determined according to the video memory resource condition of the device performing the neural network structure search.
  • the number of building units can be determined according to the size of the video memory resources of the device performing the neural network structure search.
  • the number of construction units can also be relatively large.
  • the number of construction units can be set smaller.
  • the number of the aforementioned building units can also be set based on experience. For example, it can be determined based on experience how many building units generally need to be stacked to form a search network.
  • Each of the above multiple building units can be a network structure obtained by connecting multiple nodes through the basic operation of a neural network.
  • the building unit is a basic module for building a neural network.
  • the building unit can be used to build a neural network. .
  • FIG. 10 The following briefly introduces the structure of the building unit of the embodiment of the present application with reference to FIG. 10. It should be understood that the building unit shown in FIG. 10 is only a possible building unit, and the building unit shown in FIG. The structure creates any limitations.
  • the building unit may include nodes c_ ⁇ k-2 ⁇ and c_ ⁇ k-1 ⁇ and node 0, node 1 and node 2, where nodes c_ ⁇ k-2 ⁇ and c_ ⁇ k-1 ⁇ Is the input node, nodes 0 and 1 are intermediate nodes, and node 2 is the output node.
  • the construction unit shown in Figure 10 can receive the data output by nodes c_ ⁇ k-2 ⁇ and c_ ⁇ k-1 ⁇ , and process the input data by nodes 0 and 1, respectively. Among them, the data output by node 0 is also It will be input to node 1 for processing, and the data output by node 0 and node 1 will be sent to node 2 for processing, and node 2 will finally output the data processed by the building unit.
  • each node may be a feature graph.
  • the thick arrow in Figure 10 represents one or more basic operations. The calculation results of the basic operations that are imported into the same intermediate node are added at the intermediate node.
  • the thin arrow in Figure 10 represents the feature map connection of the channel dimension, and the output node 2
  • the output feature map is formed by connecting the outputs of two intermediate nodes (node 0 and node 1) in the channel dimension of the feature map in order.
  • the aforementioned search space may include basic operations or a combination of basic operations in a preset convolutional neural network, and these basic operations or combinations of basic operations may be collectively referred to as basic operations.
  • the above search space can contain the following 8 basic operations:
  • Zero setting operation (Zero, all neurons in the corresponding position are set to zero).
  • the search network in the above step 1002 may be a neural network for searching a neural network structure.
  • the foregoing stacking of multiple building units to obtain a search network includes: stacking the multiple building units in sequence in a preset stacking manner to obtain a search network, wherein, in the search network, the search network is located The output of the building unit in front of the network is the input of the building unit located in the back of the search network.
  • the above-mentioned preset stacking mode may be what type of building units are stacked at which position, and the number of stacked building units of each type, and so on.
  • a partial channel of the output feature map of each node is processed by the to-be-selected operation to obtain the processed feature map.
  • the input feature map of the next node of each node is a feature map obtained by splicing the processed feature map with the remaining feature maps of each node that have not been processed by the candidate operation.
  • the next node of each of the foregoing nodes may refer to a node where each node has a connection relationship, and a partial channel of the output feature map of each node is processed by the waiting operation to obtain the input node of the feature map.
  • the output feature map of each node has c channels, and the c/K channels of the output feature map of each node undergo the processing to be selected to obtain the processed feature map,
  • the input feature map of the next node of each node is a feature map obtained by splicing the processed feature map and the remaining feature map.
  • the processed feature map has c/K channels
  • the remaining feature map has c(K-1)/K channels.
  • the remaining feature map is the feature map output by each node that has not been processed by the operation to be selected.
  • Both c and K are integers greater than 1, and the foregoing candidate operations include all operations in the search space.
  • a construction unit contains a first node and a second node, the second node is the next node of the first node, the feature map output by the first node has c channels, and the feature map output by the first node is c/K Two channels are processed to be selected to obtain the processed feature map.
  • the input feature map of the second node is the feature map obtained by splicing the processed feature map and the remaining feature map.
  • the remaining feature map is the unprocessed feature map output by the first node.
  • the feature map processed by the operation to be selected.
  • node 0 outputs the first feature map.
  • the second feature map is obtained.
  • the second feature map can be input to node 1 after the channel sequence is exchanged Continue processing.
  • the width of the first feature map is w
  • the height is h
  • the number of channels is c.
  • the first feature map can be divided into feature map A and feature map B according to the number of channels.
  • the number of channels of feature map A is c/K
  • the number of channels of feature map B is c(K-1)/K.
  • the feature map A is processed by a candidate operation, and the candidate operations here include operation 1 to operation 8. After the output after the 8 operation processing is respectively weighted and summed with the weight parameters (W1 to W8) of the respective operations, Obtain feature map A'.
  • the feature map B described above has not undergone a waiting operation. After the feature map A'is obtained, the feature map A'and the feature map B can be spliced in the dimension of the channel to obtain the second feature map.
  • K can be 2 or a multiple of 2.
  • K can be 2, 4, 6, 8, and so on.
  • the value of K is determined according to the size of the video memory resource of the device that executes the method shown in FIG. 9.
  • K when the video memory resource of the device that executes the above method is large, K can be set to a smaller value, and when the video memory resource of the device that executes the above method is less, K can be set to a larger value.
  • the value of K is 4, and when the video memory resource of the device that executes the above method is small, the value of K is 8.
  • the value of K may be set first according to the size of the video memory resources of the device that executes the above method, and then the value of K may be adjusted according to the performance of the target neural network in the case of the K value.
  • the feature map output by node 0 includes c channels. Then, in the feature map of the c channels output by node 0, there may be c/4 channels that have undergone the operation to be selected. Then, the processed feature map is obtained, and the remaining feature maps in the feature map output by node 0 that have not been processed by the operation to be selected have 3c/4 channels, then the processed feature map (the number of channels is c/4) and The remaining feature maps (the number of channels is 3c/4) are spliced to obtain a spliced feature map, the number of channels of the spliced feature map is c, and the spliced feature map can be input into node 1.
  • the output feature map of each node in the construction unit since the output feature map of each node in the construction unit has only partial channel feature maps for processing to be selected, the number of channels of feature maps to be selected can be reduced. In turn, the video memory occupied during the search is reduced, and the search overhead is reduced.
  • this application can reduce the video memory occupied in the search process, this application can increase the data volume of each batch of data processed in the network search process under the same video memory resources, so as to achieve a more complex neural network structure. search for.
  • the feature maps of part of the channels in the spliced feature map can be randomly selected for processing, the spliced feature maps can be exchanged in order. Enter to the next node.
  • the input feature map of the next node of each node is a feature map obtained by splicing the processed feature map and the remaining feature maps, including: the input feature map of the next node of each node It is the feature map obtained after the processed feature map and the remaining feature map are spliced and the channel sequence is exchanged.
  • the aforementioned switching of the channel sequence may be to readjust the channel splicing sequence of the spliced feature maps.
  • the aforementioned switching of the channel sequence may be to readjust the channel splicing sequence of the spliced feature maps.
  • the processed feature map can be spliced with the remaining feature maps to obtain the spliced feature map, and then the channel order of the spliced feature map is exchanged, and then the channel order is exchanged
  • the resulting feature map is input to the next node.
  • the channel order can be exchanged first, and then the feature maps obtained after the order exchange can be spliced together, and the spliced feature map is input to the next node for processing.
  • the c/K channels of the processed feature map can be shifted, and the c/K channels of the processed feature map can be moved from one end of the feature map to the other end, thereby obtaining Feature map after splicing and channel sequence exchange.
  • the sequence of the feature map A'and the feature map B can be exchanged.
  • the c/K channels in the spliced feature map can also be inserted into the c(K-1)/K channels in the spliced feature map to obtain splicing and channel sequence exchange
  • the latter feature map specifically, in FIG. 13, the feature map B can be split into a feature map B1 and a feature map B2, and the feature map A'is inserted between the feature map B1 and the feature map B2.
  • the processed feature map can also be divided into multiple sub feature maps according to the channels, and then the sub feature maps can be distributed among the different channels of the remaining feature maps.
  • the feature map after splicing and channel sequence exchange is obtained.
  • the channel order of the spliced feature maps can be shuffled, so that the next node can randomly select the feature maps of some channels for processing, which can enhance the input data In order to avoid over-fitting in the final target neural network as much as possible.
  • each of the above-mentioned multiple building units includes an input node and a plurality of intermediate nodes, the connection between the nodes of each building unit constitutes an edge, and the input of each intermediate node of the multiple intermediate nodes It is the sum of the products of the corresponding multiple input feature maps and their corresponding edge weight parameters, where each input corresponds to an edge weight parameter, and the edge weight parameter corresponding to each input is used to indicate that each input is to each middle The weight of the node.
  • node 3 is an intermediate node
  • some channels in the output feature maps of node 0, node 1, and node 2 are processed and spliced with the remaining feature maps and then input to node 3.
  • Set different weights for the edges set the edge weight parameter between 0 and 3 to E1, set the edge weight parameter between 1 and 3 to E2, and set the edge weight parameter between 2 and 3 to E3,
  • the output feature maps on each edge are multiplied by their respective weights and then superimposed to obtain the input feature map of node 3.
  • the feature maps output by the three edges are F1, F2, and F3, respectively. Then, the input feature map of node 3 can be obtained according to formula (1).
  • F represents the input feature map of node 3
  • F1, F2, and F3 are the output feature maps of 3 edges respectively, which are the edge weight parameters corresponding to the 3 edges.
  • the edge weight parameters by setting the edge weight parameters, the importance of different edges can be measured, and during the optimization process, the edges with the corresponding edge weight parameter values can be selected, and the edges with the smaller edge weight parameter values can be discarded, so that The finally constructed target neural network maintains certain stability.
  • optimizing the network structure of the building unit to obtain the optimized building unit specifically includes: adjusting the network structure parameters of the building unit in the search network in the search space to obtain the optimized building unit.
  • the optimized construction unit retains some edges corresponding to the edge weight parameters, and the optimized construction unit retains some operations corresponding to the weight parameters of the candidate operations.
  • training data and verification data can be selected.
  • the training data is used to train the convolution parameters
  • the verification data trains the network structure parameters.
  • each of the above-mentioned multiple building units includes an input node and multiple intermediate nodes, and each intermediate node in the multiple intermediate nodes corresponds to a level, wherein the intermediate node of the first level and the input node The nodes are connected, the intermediate node of the i-th level is connected to the input node, the intermediate node of the i-th level is connected to the intermediate nodes of the previous i-1 levels, and i is an integer greater than 1.
  • the above search network can contain multiple types of building units. The following briefly introduces the common building units included in the search network.
  • the building units in the search network include the first type of building units.
  • the first type of construction unit is a construction unit in which the number of input feature maps (specifically, the number of channels) and the size are the same as the number and size of output feature maps.
  • the input of a certain first type of construction unit is a feature map of size C ⁇ D1 ⁇ D2 (C is the number of channels, D1 and D2 are width and height respectively), and the output is processed by the first type of construction unit
  • the size of the feature map is still C ⁇ D1 ⁇ D2.
  • the above-mentioned first type of building unit may specifically be a normal cell (normal cell)
  • the building unit in the search network includes the second type of building unit.
  • the resolution of the output feature map of the second type of construction unit is 1/M of the input feature map
  • the number of output feature maps of the second type of construction unit is M times the number of input feature maps
  • M is a positive value greater than 1. Integer.
  • the value of M can generally be 2, 4, 6, and 8.
  • the input of a certain second type of construction unit is 1 size C ⁇ D1 ⁇ D2 (C is the number of channels, D1 and D2 are width and height respectively, and the product of C1 and C2 can represent the resolution of the feature map) Feature map, then, after the second type of building unit is processed, the size of 1 obtained is Characteristic map.
  • the above-mentioned second type of construction unit may specifically be a down-sampling unit (redution cell).
  • the structure of the search network may be as shown in FIG. 15.
  • the search network is composed of five building units stacked in sequence. Among them, the first type of building unit is located at the front and the last of the search network, and there is a second type of building unit between every two first building units.
  • the first building unit in the search network in Figure 15 can process the input image. After the first type of building unit processes the image, the processed feature map is input to the second type of building unit for processing, and so on. Transfer backwards until the last first-type construction unit in the search network outputs the feature map.
  • the feature map output by the last first-type construction unit of the search network is sent to the classifier for processing, and the classifier classifies the image according to the feature map.
  • the size of the search space and the number of construction units are determined, and the construction units are stacked to obtain the search network.
  • the network structure of the building unit in the search network can be optimized (training data can be used for optimization).
  • the channel sampling structure search can be used to optimize the network structure of the building unit.
  • the main improvements in the sampling structure search include channel sampling (that is, only part of the output feature map of each node has been processed by the candidate operation, and the processed feature map and the unprocessed feature map are spliced and input to the next node) and Edge regularization (in the construction unit, the input of each intermediate node is the sum of the products of the corresponding multiple input feature maps and the respective edge weight parameters).
  • the optimized building unit can form a neural network and output.
  • the optimization process in the dashed box in FIG. 16 is equivalent to the optimization process of step 1003 above.
  • the neural network structure search method of the embodiment of the application can be executed by the neural network structure search system.
  • FIG. 17 shows the process of the neural network structure search system executing the neural network structure search method of the embodiment of the application. The content shown in Figure 17 will be described in detail below.
  • the operation warehouse 101 may include the basic operations in the convolutional neural network that are preset.
  • the operation warehouse 101 can contain the following 8 basic operations:
  • Zero setting operation (Zero, all neurons in the corresponding position are set to zero).
  • the channel sampling structure search module 102 is used to optimize the network structure of the construction unit of the search network. In the optimization process, in each construction unit, only some channels of the output feature map of each node have been processed by the operation to be selected, and The processed feature map and the unprocessed feature map are joined as the input feature map of the next node.
  • the construction unit 103 is an optimized construction unit and can be used to build a target neural network.
  • the size of the operation warehouse 101 (equivalent to the search space above) and the initial number of building units can be determined according to the target task, and then the search network can be obtained by stacking the initial number of building units.
  • the building unit can be optimized to obtain the optimized building unit.
  • the output feature map of each node is realized through channel sampling 1022 and only some channels are processed for the candidate operation, and the edge regularization 1023 can realize the edge weight parameters corresponding to the connected nodes in the construction unit.
  • the final optimized building unit is the building unit 103, and the number of the building units 103 is generally multiple. According to the multiple building units 103, the target neural network can be built.
  • step 1021 the optimization process of the construction unit 1021 is equivalent to the optimization process in step 1003 in the method shown in FIG. 9.
  • Table 1 shows the classification accuracy and search overhead of the neural network structure search method and the existing solution on the image classification data set under similar constraints.
  • NASNet-A, AmoebaNet-B, ENAS, PNAS, DARTS (2ND) and SNAS respectively represent the network structure of the traditional solution.
  • Table 1 shows the traditional solution and the search overhead of the application under the data set CIFAR10. It can be seen from Table 1 that compared with the traditional solution, the search overhead of the solution of this application is greatly reduced.
  • ProxylessNAS is a network architecture of the existing solution
  • Table 2 shows the search overhead of the existing solution and the solution of the application under the dataset ImageNet.
  • CIFAR10 and ImageNet are different data sets
  • ImageNetTop1 and ImageNetTop5 are sub-indicators, which refer to the proportion (accuracy rate) of correct results in the first 1 or 5 results in the ImageNet data set.
  • NASNet-A, AmoebaNet-B, ENAS, PNAS, DARTS (2ND) and SNAS respectively represent different network structures.
  • the data below the column where CIFR10, ImageNetTop1, ImageNetTop5 are located represents the classification accuracy.
  • the neural network structure search method of the embodiment of the present application in detail in combination with the data set CIFAR10.
  • Table 4 when the neural network structure search method does not use channel sampling or edge regularization (equivalent to a traditional scheme), the classification accuracy is 97%, and the search cost is 0.4GDs.
  • the classification accuracy rate is 97.18, the accuracy rate is improved, and the search cost is still 0.4GDs, which does not change.
  • the neural network structure search method uses both channel sampling and edge regularization, the classification accuracy is 97.43, the classification accuracy is improved, the search cost is 0.1GDs, and the search cost is greatly reduced.
  • the search method of the neural network structure of the embodiment of the present application is described in detail above with reference to the accompanying drawings.
  • the neural network constructed by the search method of the neural network structure of the embodiment of the present application can be used for image processing (for example, image classification), etc. , These specific applications are introduced below.
  • FIG. 18 is a schematic flowchart of an image processing method according to an embodiment of the present application. It should be understood that the above definitions, explanations, and extensions of the relevant content of the method shown in FIG. 9 are also applicable to the method shown in FIG. 18, and repeated descriptions are appropriately omitted when the method shown in FIG. 18 is introduced below.
  • the method shown in Figure 18 includes:
  • the target neural network is constructed by multiple optimized building units, and the multiple optimized building units are obtained by optimizing the network structure of multiple building units in the search network.
  • the c/K channels of the output feature map of each node are processed by the candidate operation to obtain the processed feature map.
  • the input feature map of the next node of each node is the processed feature map and the remaining
  • the feature map obtained after the feature map is spliced, the output feature map of each node has c channels, the processed feature map has c/K channels, the remaining feature map has c(K-1)/K channels, and the remaining features
  • the figure is a feature graph output by each node that has not been processed by the operation to be selected.
  • the operation to be selected includes all operations in the search space. Both c and K are integers greater than 1.
  • the target neural network is constructed by optimized construction units, and the target neural network may be trained on training data (including training images and classification results of training images). After the neural network.
  • the target neural network processes the image to be processed, all channels of the feature map output by each node in each optimized construction unit will be processed by the candidate operation, and the feature map processed by the candidate operation will be input to the next A node, which is the difference between using the target neural network for image processing and neural network structure search processing to obtain the target neural network.
  • the input feature map of the next node of each node is a feature map obtained by splicing the processed feature map with the remaining feature maps, including: the input feature map of the next node of each node is processed The feature map and the remaining feature maps are spliced and the channel sequence is exchanged to obtain the feature map.
  • the target neural network since the target neural network is constructed in the process of changing the order of the channels of the spliced feature maps, the channel order of the spliced feature maps can be disrupted, so that the next node can randomly select part of the channel.
  • the feature map processing can enhance the randomness of the input data, so as to avoid the over-fitting phenomenon of the final target neural network as much as possible. Therefore, the use of the target neural network can better perform image classification.
  • each of the above-mentioned multiple building units includes an input node and multiple intermediate nodes, the connection between the nodes of each building unit constitutes an edge, and the input of each intermediate node of the multiple intermediate nodes is The sum of the products of the corresponding multiple inputs and their corresponding edge weight parameters, where each input corresponds to an edge weight parameter, and the edge weight parameter corresponding to each input is used to indicate each input to each intermediate node. Weights.
  • the target neural network is constructed in the process of setting edge weight parameters, the importance of different edges can be measured, and during the optimization process, the corresponding edge with a relatively large edge weight parameter value can be selected, and the edge weight can be discarded.
  • the edge of the parameter value is relatively small, so that the finally constructed target neural network maintains a certain stability.
  • each of the above-mentioned multiple building units includes an input node and a plurality of intermediate nodes, and each intermediate node of the multiple intermediate nodes corresponds to a level, wherein the intermediate node of the first level is connected to the input node , The intermediate node of the i-th level is connected to the input node, and the intermediate node of the i-th level is connected to the intermediate nodes of the previous i-1 levels, and i is an integer greater than 1.
  • FIG. 19 is a schematic diagram of the hardware structure of the neural network structure search device provided by an embodiment of the present application.
  • the neural network structure search device 3000 shown in FIG. 19 includes a memory 3001, a processor 3002, a communication interface 3003, and a bus 3004. Among them, the memory 3001, the processor 3002, and the communication interface 3003 implement communication connections between each other through the bus 3004.
  • the memory 3001 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).
  • the memory 3001 may store a program. When the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 is configured to execute each step of the neural network structure search method of the embodiment of the present application.
  • the processor 3002 may adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more
  • the integrated circuit is used to execute related programs to implement the neural network structure search method of the method embodiment of the present application.
  • the processor 3002 may also be an integrated circuit chip with signal processing capabilities.
  • the various steps of the neural network structure search method of the present application can be completed by hardware integrated logic circuits in the processor 3002 or instructions in the form of software.
  • the above-mentioned processor 3002 may also be a general-purpose processor, a digital signal processing (digital signal processing, DSP), an application specific integrated circuit (ASIC), an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, Discrete gates or transistor logic devices, discrete hardware components.
  • DSP digital signal processing
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers.
  • the storage medium is located in the memory 3001, the processor 3002 reads the information in the memory 3001, and combines its hardware to complete the functions required by the units included in the neural network structure search device, or execute the neural network structure of the method embodiment of the application Search method.
  • the communication interface 3003 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 3000 and other devices or communication networks. For example, the information of the neural network to be constructed and the training data needed in the process of constructing the neural network can be obtained through the communication interface 3003.
  • a transceiver device such as but not limited to a transceiver to implement communication between the device 3000 and other devices or communication networks. For example, the information of the neural network to be constructed and the training data needed in the process of constructing the neural network can be obtained through the communication interface 3003.
  • the bus 3004 may include a path for transferring information between various components of the device 3000 (for example, the memory 3001, the processor 3002, and the communication interface 3003).
  • FIG. 20 is a schematic diagram of the hardware structure of an image processing apparatus according to an embodiment of the present application.
  • the image processing apparatus 4000 shown in FIG. 20 includes a memory 4001, a processor 4002, a communication interface 4003, and a bus 4004.
  • the memory 4001, the processor 4002, and the communication interface 4003 implement communication connections between each other through the bus 4004.
  • the memory 4001 may be ROM, static storage device and RAM.
  • the memory 4001 may store a program. When the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 and the communication interface 4003 are used to execute each step of the image processing method of the embodiment of the present application.
  • the processor 4002 may adopt a general-purpose CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits to execute related programs to realize the functions required by the units in the image processing apparatus of the embodiment of the present application. Or execute the image processing method in the method embodiment of this application.
  • the processor 4002 may also be an integrated circuit chip with signal processing capability.
  • each step of the image processing method of the embodiment of the present application can be completed by an integrated logic circuit of hardware in the processor 4002 or instructions in the form of software.
  • the aforementioned processor 4002 may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component.
  • the methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a mature storage medium in the field, such as random memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers.
  • the storage medium is located in the memory 4001, and the processor 4002 reads the information in the memory 4001, and combines its hardware to complete the functions required by the units included in the image processing apparatus of the embodiment of the application, or perform the image processing of the method embodiment of the application. method.
  • the communication interface 4003 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 4000 and other devices or a communication network.
  • a transceiver device such as but not limited to a transceiver to implement communication between the device 4000 and other devices or a communication network.
  • the image to be processed can be acquired through the communication interface 4003.
  • the bus 4004 may include a path for transferring information between various components of the device 4000 (for example, the memory 4001, the processor 4002, and the communication interface 4003).
  • FIG. 21 is a schematic diagram of the hardware structure of the neural network training device of an embodiment of the present application. Similar to the aforementioned device 3000 and device 4000, the neural network training device 5000 shown in FIG. 21 includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. Among them, the memory 5001, the processor 5002, and the communication interface 5003 implement communication connections between each other through the bus 5004.
  • the neural network After the neural network is constructed by the neural network structure search device shown in FIG. 19, the neural network can be trained by the neural network training device 5000 shown in FIG. 21, and the trained neural network can be used to execute this application Example image processing method.
  • the device shown in FIG. 21 can obtain training data and the neural network to be trained from the outside through the communication interface 5003, and then the processor trains the neural network to be trained according to the training data.
  • the device 3000, device 4000, and device 5000 only show a memory, a processor, and a communication interface, in a specific implementation process, those skilled in the art should understand that the device 3000, device 4000, and device 5000 may also Including other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 3000, the device 4000, and the device 5000 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the device 3000, the device 4000, and the device 5000 may also only include the components necessary to implement the embodiments of the present application, and not necessarily include all the components shown in FIGS. 19, 20, and 21.
  • the disclosed system, device, and method may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • each unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of this application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program code .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Multimedia (AREA)
  • Neurology (AREA)
  • Image Analysis (AREA)

Abstract

人工智能领域中计算机视觉领域的一种神经网络结构的搜索方法、图像处理方法及装置。其中，该神经网络结构的搜索方法包括：确定搜索空间和多个构建单元（1001），堆叠多个构建单元，以得到搜索网络（1002）；在搜索空间内对搜索网络中的构建单元的网络架构进行优化，得到优化后的构建单元（1003）；根据优化后的构建单元搭建目标神经网络（1004）。其中，在每个构建单元中，每个节点的输出特征图的部分通道经过待选操作处理，得到处理后的特征图，该处理后的特征图与剩余特征图拼接后输入到下一个节点。由于特征图只有部分通道送入待选操作处理，因此，可以降低搜索开销。

Description

神经网络结构的搜索方法、图像处理方法和装置
本申请要求于2019年07月12日提交中国专利局、申请号为201910627480.7、申请名称为“神经网络结构的搜索方法、图像处理方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能领域，并且更具体地，涉及一种神经网络结构的搜索方法、图像处理方法和装置。
背景技术
人工智能(artificial intelligence,AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。换句话说，人工智能是计算机科学的一个分支，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式作出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策的功能。人工智能领域的研究包括机器人，自然语言处理，计算机视觉，决策与推理，人机交互，推荐与搜索，AI基础理论等。
随着人工智能技术的快速发展，神经网络(例如，深度神经网络)近年来在图像、视频以及语音等多种媒体信号的处理与分析中取得了很大的成就。一个性能优良的神经网络往往拥有精妙的网络结构，而这需要具有高超技能和丰富经验的人类专家花费大量精力进行构建。为了更好地构建神经网络，人们提出了通过神经网络结构搜索(neural architecture search，NAS)的方法来搭建神经网络，通过自动化地搜索神经网络结构，从而得到性能优异的神经网络结构。
传统方案常采用可微分的神经网络结构搜索方法来搭建神经网络。在传统方案中，一般是根据一定数量的构建单元搭建成搜索网络，然后在搜索空间内对搜索网络中构建单元的各个节点之间的连接关系进行优化，以得到优化后的构建单元，最后再根据优化后的构建单元搭建目标神经网络。但是，传统方案在对构建单元进行优化的过程中，每个节点的输出特征图要经过所有的可选操作进行处理，需要处理的数据量较多，搜索开销很大。
发明内容
本申请提供一种神经网络结构的搜索方法、图像处理方法、装置、计算机可读存储介质和芯片，以降低搜索开销。
第一方面，提供了一种神经网络结构的搜索方法，该方法包括：确定搜索空间和多个构建单元；堆叠多个构建单元，得到搜索网络；在搜索空间内对搜索网络中的构建单元的网络结构进行优化，得到优化后的构建单元；根据优化后的构建单元搭建目标神经网络。
其中，上述构建单元是由多个节点之间通过神经网络的基本操作连接得到的网络结构，上述构建单元是一种用于构建神经网络的基础模块，利用构建单元可以搭建神经网络。
在上述多个构建单元中的每个构建单元中，每个节点的输出特征图的部分通道经过待选操作处理，得到处理后的特征图，该每个节点的下一个节点的输入特征图是处理后的特征图与该每个节点剩余的未经过待选操作处理的特征图拼接后得到的特征图。
上述每个节点的下一个节点可以是指，该每个节点存在相连关系，并且该每个节点的输出特征图的部分通道经过待选操作处理后得到的特征图输入的节点。
具体地，在上述多个构建单元的每个构建单元中，每个节点的输出特征图有c个通道，该每个节点的输出特征图的c/K个通道经过待选操作处理，得到处理后的特征图，该每个节点的下一个节点的输入特征图是处理后的特征图与剩余特征图拼接后得到的特征图。其中，处理后的特征图有c/K个通道，剩余特征图有c(K-1)/K个通道，该剩余特征图为该每个节点输出的未经待选操作处理的特征图，c和K均为大于1的整数，上述待选操作包括所述搜索空间中的所有操作。
也就是说，在上述多个构建单元的每个构建单元中，每个节点的输出特征图只有部分通道经过了待选操作处理，而处理后的特征图与剩余特征图拼接后输入到该每个节点的下一个节点，作为下一个节点的输入特征图。
例如，一个构建单元中包含第一节点和第二节点，第二节点为第一节点的下一个节点，第一节点输出的特征图有c个通道，第一节点输出的特征图的c/K个通道经过待选操作处理，得到处理后的特征图，第二个节点的输入特征图是处理后的特征图与剩余特征图拼接得到的特征图，该剩余特征图为第一节点输出的未经过待选操作处理的特征图。
可选地，上述搜索空间是根据待构建的目标神经网络的应用需求确定的。
具体地，上述搜索空间可以是根据目标神经网络的处理数据的类型确定的。
当上述目标神经网络是用于处理图像数据的神经网络时，上述搜索空间包含的操作种类和数量要与图像数据的处理相适应。
例如，当目标神经网络是用于处理图像数据的神经网络时，上述搜索空间可以包含卷积操作，池化操作，跳连接(skip-connect)操作等等。
当上述目标神经网络是用于处理语音数据的神经网络时，上述搜索空间包含的操作种类和数量要与语音数据的处理相适应。
例如，当目标神经网络是用于处理语音数据的神经网络时，上述搜索空间可以包含激活函数(如ReLU、Tanh)等等。
可选地，上述搜索空间是根据目标神经网络的应用需求和执行神经网络结构搜索的设备的显存资源条件确定的。
上述执行神经网络结构搜索的设备的显存资源条件可以是指执行神经网络结构搜索的设备的显存资源大小。
上述搜索空间包含的操作种类和数量可以根据目标神经网络的应用需求和执行神经网络结构搜索的设备的显存资源条件来综合确定。
具体地，可以先根据目标神经网络的应用需求确定搜索空间包含的操作种类和数量，然后再结合执行神经网络结构搜索的设备的显存资源条件来调整搜索空间包含的操作种类和数量，以确定搜索空间最终包含的操作种类和数量。
例如，在根据目标神经网络的应用需求确定搜索空间包含的操作种类和数量之后，如果执行神经网络结构搜索的设备的显存资源较少，那么，可以将搜索空间中一些不太重要的操作删掉；而如果执行神经网络结构搜索的设备的显存资源较为充足时，可以保持搜索空间包含的操作种类和数量，或者增加搜索空间包含操作的种类和数量。
上述显存资源可以替换为缓存资源，缓存资源是用于构建神经网络的设备在优化过程中用于存放运算数据的内存或者存储单元。该缓存资源具体可以包括显存资源。
可选地，上述构建单元的数量是根据执行神经网络结构搜索的设备的显存资源条件确定的。
具体地，当执行神经网络结构搜索的设备的显存资源较少时，构建单元的数量可以少一些，而当执行神经网络结构搜索的设备的显存资源比较充足时，构建单元的数量可以多一些。
可选地，上述构建单元的数量是根据待构建的目标神经网络的应用需求和执行神经网络结构搜索的设备的显存资源条件确定的。
具体地，可以先根据目标神经网络的应用需求确定构建单元的初始数量，然后再根据执行神经网络结构搜索的设备的显存资源进一步调整构建单元的初始数量，从而确定构建单元的最终数量。
例如，在根据目标神经网络的应用需求确定构建单元的初始数量之后，如果执行目标神经网络结构搜索的设备的显存资源较少，那么，可以进一步减少构建单元的数量；而如果执行目标神经网络结构搜索的设备的显存资源比较充足，可以保持构建单元的初始数量不变，此时，该构建单元的初始数量就是构建单元的最终数量。
本申请中，在神经网络结构的搜索过程中，由于构建单元中的每个节点的输出特征图只有部分通道的特征图进行待选择操作处理，能够降低待选操作处理的特征图的通道数量，进而减少搜索时占用的显存，减少搜索开销。
此外，本申请中通过选择每个节点的输出特征图中部分通道进行待选操作处理，减少了待选操作处理的数据量，可以在一定程度上降低最终搭建的目标网络出现过拟合的可能性。
进一步的，由于本申请能够减少搜索过程中占用的显存，因此，本申请能够在同样显存资源的情况下，增加网络搜索过程处理的每批数据的数据量，实现对更复杂的神经网络结构的搜索。
可选地，上述堆叠多个构建单元，以得到搜索网络，包括：按照预设的堆叠方式将所述多个构建单元依次堆叠起来，以得到搜索网络，其中，在该搜索网络中，位于搜索网络前面的构建单元的输出是位于搜索网络的后面的构建单元的输入。
上述预设的堆叠方式可以包括在什么位置堆放什么类型的构建单元以及堆叠的数量等信息。
结合第一方面，在第一方面的某些实现方式中，上述每个节点的下一个节点的输入特征图是处理后的特征图与剩余特征图拼接后得到的特征图，包括：每个节点的下一个节点的输入特征图是处理后的特征图与剩余的特征图经过拼接和通道顺序调换后得到的特征图。
上述调换通道顺序可以是将拼接后的特征图的通道顺序重新进行调整。
在得到处理后的特征图之后，可以将处理后的特征图与剩余的特征图进行拼接，得到拼接后的特征图，并将拼接后的特征图的通道顺序进行调换，然后将通道顺序调换后得到的特征图输入到下一个节点。
或者，在得到上述处理后的特征图之后，还可以先进行通道顺序的调换，再将通道顺序调换后得到的特征图拼接在一起，然后将拼接后得到的特征图输入到下一个节点进行处理。
本申请中，通过将拼接后的特征图的通道的顺序进行调换，能够将拼接后的特征图的通道顺序打乱，使得下一个节点可以随机选择部分通道的特征图进行处理，可以增强输入数据的随机性，从而尽可能的避免最终得到的目标神经网络出现过拟合的现象。
结合第一方面，在第一方面的某些实现方式中，上述多个构建单元中的每个构建单元包括输入节点和多个中间节点，每个构建单元的节点之间的连接构成边，该多个中间节点中的每个中间节点的输入为对应的多个输入特征图分别与各自对应的边权重参数的乘积的和，其中，每个输入对应一个边权重参数，每个输入对应的边权重参数用于指示每个输入到所述每个中间节点时的权重。
上述输入节点的数量可以是一个也可以是多个。
本申请中，通过设置边权重参数，能够衡量不同边的重要程度，并在优化的过程中可以选择出对应的边权重参数值比较大的边，舍弃边权重参数值比较小的边，从而使得最终构建得到的目标神经网络保持一定的稳定性。
结合第一方面，在第一方面的某些实现方式中，上述在搜索空间内对搜索网络中的构建单元的网络结构进行优化，以得到优化后的构建单元，包括：在搜索空间内，对搜索网络中的构建单元的网络结构参数进行调整，得到优化后的构建单元。
其中，上述构建单元的网络结构参数包括待选操作权重参数和边权重参数，优化后的构建单元保留部分边权重参数对应的边，优化后的构建单元的保留部分待选操作权重参数对应的操作。
对于优化后的构建单元来说，可以保留部分参数值较大的边权重参数值对应的边，保留部分参数值较大的待选操作权重参数对应的操作。
结合第一方面，在第一方面的某些实现方式中，上述多个构建单元中的每个构建单元包括输入节点和多个中间节点，多个中间节点中的每个中间节点对应一个层级。
其中，第一层级的中间节点与输入节点相连，第i层级的中间节点与输入节点相连，第i层级的中间节点与前i-1个层级的中间节点相连，i为大于1的整数。
上述输入节点的数量可以是一个也可以是多个，当输入节点的数量为多个时，第一层级的中间节点与每个输入节点都相连。
结合第一方面，在第一方面的某些实现方式中，K是根据执行上述方法的设备的显存资源的大小确定的。
上述K的数值可以设置为2的倍数，例如，2，4，6，8等等。
具体地，当执行上述方法的设备的显存资源较多时，K可以设置成一个较小的数值，当执行上述方法的设备的显存资源较少时，K可以设置成一个较大的数值。
例如，当执行上述方法的设备的显存资源较多时，K的取值为4，当执行上述方法的设备的显存资源较少时，K的取值为8。
还可以结合执行上述方法的设备的显存资源的大小和目标神经网络的性能综合确定K的数值。
具体地，可以先根据执行上述方法的设备的显存资源的大小设置K的数值，然后再根据目标神经网络在该K值的情况下的性能，对K的数值进行调整。
例如，根据执行上述方法的设备的显存资源的大小，可以将K设置成6或8，但是目标神经网络在K＝8时的性能不够理想，而在目标神经网络在K＝6时的性能符合要求，那么，就可以将K设置为6。
上述目标神经网络在K取不同数值时的性能可以根据测试结果得到。
第二方面，提供了一种图像处理方法，该方法包括：获取待处理图像；根据目标神经网络对待处理图像进行分类，得到待处理图像的分类结果。
其中，目标神经网络由多个优化后的构建单元搭建而成，多个优化后的构建单元是通过对搜索网络中的多个构建单元的网络结构进行优化得到的，在多个构建单元的每个构建单元中，每个节点的输出特征图的c/K个通道经过待选操作处理，得到处理后的特征图，每个节点的下一个节点的输入特征图是处理后的特征图与剩余特征图拼接后得到的特征图，每个节点的输出特征图有c个通道，处理后的特征图有c/K个通道，剩余特征图有c(K-1)/K个通道，剩余特征图为每个节点输出的未经待选操作处理的特征图，待选操作包括搜索空间中的所有操作，c和K均为大于1的整数。
本申请中，由于目标神经网络在构建过程中，每个构建单元中的每个节点的输出特征图的部分通道经过待选操作处理，可以减少最终得到的目标神经网络发生过拟合的现象，因此，利用该目标神经网络能够更好地进行图像分类。
应理解，在第二方面的方法中，目标神经网络是由优化后的构建单元搭建而成，并且该目标神经网络可以是经过训练数据(包括训练图像，以及训练图像的分类结果)进行训练后的神经网络。在目标神经网络对待处理图像进行处理时，每个优化后的构建单元中的每个节点输出的特征图的全部通道都会经过待选操作处理，经过待选操作处理后的特征图再输入到下一个节点，这是利用目标神经网络进行图像处理与神经网络结构搜索处理以得到目标神经网络的不同之处。
结合第二方面，在第二方面的某些实现方式中，每个节点的下一个节点的输入特征图是处理后的特征图与剩余特征图拼接后得到的特征图，包括：每个节点的下一个节点的输入特征图是处理后的特征图与剩余的特征图经过拼接和通道顺序调换后得到的特征图。
本申请中，由于目标神经网络在构建过程中，通过将拼接后的特征图的通道的顺序进行调换，能够将拼接后的特征图的通道顺序打乱，使得下一个节点可以随机选择部分通道的特征图进行处理，可以增强输入数据的随机性，从而尽可能的避免最终得到的目标神经网络出现过拟合的现象，因此，利用该目标神经网络能够更好地进行图像分类。
结合第二方面，在第二方面的某些实现方式中，多个构建单元中的每个构建单元包括输入节点和多个中间节点，每个构建单元的节点之间的连接构成边，多个中间节点中的每个中间节点的输入为对应的多个输入分别与各自对应的边权重参数的乘积的和，其中，每个输入对应一个边权重参数，每个输入对应的边权重参数用于指示每个输入到每个中间节点时的权重。
本申请中，由于目标神经网络在构建过程中，通过设置边权重参数，能够衡量不同边 的重要程度，并在优化的过程中可以选择出对应的边权重参数值比较大的边，舍弃边权重参数值比较小的边，从而使得最终构建得到的目标神经网络保持一定的稳定性。
结合第二方面，在第二方面的某些实现方式中，多个构建单元中的每个构建单元包括输入节点和多个中间节点，多个中间节点中的每个中间节点对应一个层级，其中，第一层级的中间节点与输入节点相连，第i层级的中间节点与输入节点相连，第i层级的中间节点与前i-1个层级的中间节点相连，i为大于1的整数。
第三方面，提供了一种图像处理方法，该方法包括：获取待处理图像；根据目标神经网络对待处理图像进行处理，得到待处理图像的处理结果。
上述对图像进行处理，可以是指对图像进行识别，分类，检测等等。
第四方面，提供了一种图像处理方法，该方法包括：获取道路画面；根据目标神经网络对道路画面进行卷积处理，得到道路画面的多个卷积特征图；根据目标神经网络对道路画面的多个卷积特征图进行反卷积处理，获得该道路画面的语义分割结果。
其中，上述目标神经网络是根据第一方面中的任意一种实现方式构建得到的目标神经网络。
第五方面，提供了一种图像处理方法，该方法包括：获取人脸图像；根据目标神经网络对人脸图像进行卷积处理，得到人脸图像的卷积特征图；将人脸图像的卷积特征图与身份证件图像的卷积特征图进行对比，得到人脸图像的验证结果。
上述身份证件图像可的卷积特征图可以是预先获取的，并存储在相应的数据库中。例如，预先对身份证件图像进行卷积处理，将得到的卷积特征图存储到数据库中。
另外，上述目标神经网络是根据第一方面中的任意一种实现方式构建得到的目标神经网络。
应理解，在上述第一方面中对相关内容的扩展、限定、解释和说明也适用于第二方面、第三方面、第四方面和第五方面中相同的内容。
第六方面，提供了一种神经网络结构搜索装置，该装置包括：存储器，用于存储程序；处理器，用于执行所述存储器存储的程序，当所述存储器存储的程序被执行时，所述处理器用于执行第一方面中的任意一种实现方式中的方法。
第七方面，提供了一种图像处理装置，该装置包括：存储器，用于存储程序；处理器，用于执行所述存储器存储的程序，当所述存储器存储的程序被执行时，所述处理器用于执行第二方面至第五方面中的任意一种实现方式中的方法。
第八方面，提供一种计算机可读介质，该计算机可读介质存储用于设备执行的程序代码，该程序代码包括用于执行第一方面至第五方面中的任意一种实现方式中的方法。
第九方面，提供一种包含指令的计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述第一方面至第五方面中的任意一种实现方式中的方法。
第十方面，提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行上述第一方面至第五方面中的任意一种实现方式中的方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行第一方面至第五方面中的任意一种实现方式中的方法。
附图说明
图1是本申请实施例提供的一种人工智能主体框架示意图；
图2为本申请实施例提供的一种具体应用的示意图；
图3为本申请实施例提供的一种具体应用的示意图；
图4为本申请实施例提供的系统架构的结构示意图；
图5为本申请实施例提供的一种卷积神经网络的结构示意图；
图6为本申请实施例提供的一种卷积神经网络的结构示意图；
图7为本申请实施例提供的一种芯片的硬件结构示意图；
图8为本申请实施例提供的一种系统架构的示意图；
图9是本申请实施例的神经网络结构的搜索方法的示意性流程图；
图10是本申请实施例的构建单元的示意图；
图11是特征图的处理过程的示意图；
图12是特征图的通道顺序调换的示意图；
图13是特征图的通道顺序调换的示意图；
图14是不同的特征图叠加后输入节点的示意图；
图15是搜索网络的结构的示意图；
图16是本申请实施例的神经网络结构的搜索方法的示意图；
图17是本申请实施例的神经网络结构的搜索方法的示意图；
图18是本申请实施例的图像处理方法的示意性流程图；
图19是本申请实施例的神经网络结构搜索装置的示意性框图；
图20是本申请实施例的图像处理装置的示意性框图；
图21是本申请实施例的神经网络训练装置的示意性框图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行描述。
图1示出一种人工智能主体框架示意图，该主体框架描述了人工智能系统总体工作流程，适用于通用的人工智能领域需求。
下面从“智能信息链”(水平轴)和“信息技术(information technology，IT)价值链”(垂直轴)两个维度对上述人工智能主题框架进行详细的阐述。
“智能信息链”反映从数据的获取到处理的一列过程。举例来说，可以是智能信息感知、智能信息表示与形成、智能推理、智能决策、智能执行与输出的一般过程。在这个过程中，数据经历了“数据—信息—知识—智慧”的凝练过程。
“IT价值链”从人智能的底层基础设施、信息(提供和处理技术实现)到系统的产业生态过程，反映人工智能为信息技术产业带来的价值。
(1)基础设施：
基础设施为人工智能系统提供计算能力支持，实现与外部世界的沟通，并通过基础平台实现支撑。
基础设施可以通过传感器与外部沟通，基础设施的计算能力可以由智能芯片提供。
这里的智能芯片可以是中央处理器(central processing unit，CPU)、神经网络处理器(neural-network processing unit，NPU)、图形处理器(graphics processing unit，GPU)、专门应用的集成电路(application specific integrated circuit，ASIC)以及现场可编程门阵列(field programmable gate array，FPGA)等硬件加速芯片。
基础设施的基础平台可以包括分布式计算框架及网络等相关的平台保障和支持，可以包括云存储和计算、互联互通网络等。
例如，对于基础设施来说，可以通过传感器和外部沟通获取数据，然后将这些数据提供给基础平台提供的分布式计算系统中的智能芯片进行计算。
(2)数据：
基础设施的上一层的数据用于表示人工智能领域的数据来源。该数据涉及到图形、图像、语音、文本，还涉及到传统设备的物联网数据，包括已有系统的业务数据以及力、位移、液位、温度、湿度等感知数据。
(3)数据处理：
上述数据处理通常包括数据训练，机器学习，深度学习，搜索，推理，决策等处理方式。
其中，机器学习和深度学习可以对数据进行符号化和形式化的智能信息建模、抽取、预处理、训练等。
推理是指在计算机或智能系统中，模拟人类的智能推理方式，依据推理控制策略，利用形式化的信息进行机器思维和求解问题的过程，典型的功能是搜索与匹配。
决策是指智能信息经过推理后进行决策的过程，通常提供分类、排序、预测等功能。
(4)通用能力：
对数据经过上面提到的数据处理后，进一步基于数据处理的结果可以形成一些通用的能力，比如可以是算法或者一个通用系统，例如，翻译，文本的分析，计算机视觉的处理，语音识别，图像的识别等等。
(5)智能产品及行业应用：
智能产品及行业应用指人工智能系统在各领域的产品和应用，是对人工智能整体解决方案的封装，将智能信息决策产品化、实现落地应用，其应用领域主要包括：智能制造、智能交通、智能家居、智能医疗、智能安防、自动驾驶，平安城市，智能终端等。
本申请实施例可以应用在人工智能中的很多领域，例如，智能制造、智能交通、智能家居、智能医疗、智能安防、自动驾驶，平安城市等领域。
具体地，本申请实施例可以具体应用在图像分类、图像检索、图像语义分割、图像超分辨率和自然语言处理等需要使用(深度)神经网络的领域。
下面对相册图片分类和自动驾驶这两种应用场景进行简单的介绍。
相册图片分类：
具体地，当用户在终端设备(例如，手机)或者云盘上存储了大量的图片时，通过对相册中图像进行识别可以方便用户或者系统对相册进行分类管理，提升用户体验。
利用本申请实施例的神经网络结构搜索方法能够搜索得到适用于相册分类的神经网络结构，然后再根据训练图片库中的训练图片对神经网络进行训练，就可以得到相册分类神经网络。接下来就可以利用该相册分类神经网络对图片进行分类，从而为不同的类别的 图片打上标签，便于用户查看和查找。另外，这些图片的分类标签也可以提供给相册管理系统进行分类管理，节省用户的管理时间，提高相册管理的效率，提升用户体验。
例如，如图2所示，可以通过神经网络结构搜索系统(对应于本申请实施例的神经网络结构搜索方法)构建得到适用于相册分类的神经网络。在构建该神经网络时，可以利用训练图片库的对搜索网络中的构建单元的网络结构进行优化，得到优化后的构建单元，然后再利用该优化后的构建单元来搭建神经网络。在获得适用于相册分类的神经网络之后，可以再根据训练图片对该神经网络进行训练，得到相册分类神经网络。接下来，就可以利用相册分类神经网络对待处理图片进行分类。如图2所示，相册分类神经网络对输入的图片进行处理，得到图片的类别为郁金香。
自动驾驶场景下的物体识别：
自动驾驶中有大量的传感器数据需要处理，深度神经网络凭借着其强大的能力在自动驾驶中发挥着重要的作用。然而手工设计相应的数据处理网络费时费力。因此，通过采用本申请实施例的神经网络结构搜索方法，能够构建得到适用于自动驾驶场景下进行数据处理的神经网络，接下来，通过自动驾驶场景下的数据对该神经网络进行训练，能够得到传感器数据处理网络，最后就可以利用该传感器处理网络对输入的道路画面进行处理，从而识别出道路画面中的不同物体。
如图3所示，神经网络结构搜索系统能够根据车辆检测任务构建出一个神经网络，在构建该神经网络时，可以利用传感器数据对搜索网络中的构建单元的网络结构进行优化，得到优化后的构建单元，然后再利用该优化后的构建单元来搭建神经网络。在得到了神经网络之后，就可以根据传感器数据对该神经网络进行训练，得到传感器数据处理网络。接下来，就可以利用该传感器数据处理网络对传感器数据进行处理。如图3所示，传感器数据处理网络对输入的道路画面进行处理，能够识别出道路画面中的车辆(如图3右下角矩形框部分所示)。
由于本申请实施例涉及大量神经网络的应用，为了便于理解，下面先对本申请实施例可能涉及的神经网络的相关术语和概念进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以x s和截距1为输入的运算单元，该运算单元的输出可以为：
Figure PCTCN2020089403-appb-000001
其中，s＝1、2、……n，n为大于1的自然数，W s为x s的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入，激活函数可以是sigmoid函数。神经网络是将多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(2)深度神经网络
深度神经网络(deep neural network，DNN)，也称多层神经网络，可以理解为具有多层隐含层的神经网络。按照不同层的位置对DNN进行划分，DNN内部的神经网络可以 分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定与第i+1层的任意一个神经元相连。
虽然DNN看起来很复杂，但是就每一层的工作来说，其实并不复杂，简单来说就是如下线性关系表达式：
Figure PCTCN2020089403-appb-000002
其中，
Figure PCTCN2020089403-appb-000003
是输入向量，
Figure PCTCN2020089403-appb-000004
是输出向量，
Figure PCTCN2020089403-appb-000005
是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量
Figure PCTCN2020089403-appb-000006
经过如此简单的操作得到输出向量
Figure PCTCN2020089403-appb-000007
由于DNN层数多，系数W和偏移向量
Figure PCTCN2020089403-appb-000008
的数量也比较多。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为
Figure PCTCN2020089403-appb-000009
上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。
综上，第L-1层的第k个神经元到第L层的第j个神经元的系数定义为
Figure PCTCN2020089403-appb-000010
需要注意的是，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。训练深度神经网络的也就是学习权重矩阵的过程，其最终目的是得到训练好的深度神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。
(3)卷积神经网络
卷积神经网络(convolutional neuron network，CNN)是一种带有卷积结构的深度神经网络。卷积神经网络包含了一个由卷积层和子采样层构成的特征抽取器，该特征抽取器可以看作是滤波器。卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。在卷积神经网络的卷积层中，一个神经元可以只与部分邻层神经元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。卷积核可以以随机大小的矩阵的形式初始化，在卷积神经网络的训练过程中卷积核可以通过学习得到合理的权重。另外，共享权重带来的直接好处是减少卷积神经网络各层之间的连接，同时又降低了过拟合的风险。
(4)循环神经网络(recurrent neural networks，RNN)是用来处理序列数据的。在传统的神经网络模型中，是从输入层到隐含层再到输出层，层与层之间是全连接的，而对于每一层层内之间的各个节点是无连接的。这种普通的神经网络虽然解决了很多难题，但是却仍然对很多问题无能无力。例如，你要预测句子的下一个单词是什么，一般需要用到前面的单词，因为一个句子中前后单词并不是独立的。RNN之所以称为循环神经网路，即一个序列当前的输出与前面的输出也有关。具体的表现形式为网络会对前面的信息进行记忆并应用于当前输出的计算中，即隐含层本层之间的节点不再无连接而是有连接的，并且隐含层的输入不仅包括输入层的输出还包括上一时刻隐含层的输出。理论上，RNN能够对任何长度的序列数据进行处理。对于RNN的训练和对传统的CNN或DNN的训练一样。
既然已经有了卷积神经网络，为什么还要循环神经网络？原因很简单，在卷积神经网络中，有一个前提假设是：元素之间是相互独立的，输入与输出也是独立的，比如猫和狗。但现实世界中，很多元素都是相互连接的，比如股票随时间的变化，再比如一个人说了：我喜欢旅游，其中最喜欢的地方是云南，以后有机会一定要去。这里填空，人类应该都知 道是填“云南”。因为人类会根据上下文的内容进行推断，但如何让机器做到这一步？RNN就应运而生了。RNN旨在让机器像人一样拥有记忆的能力。因此，RNN的输出就需要依赖当前的输入信息和历史的记忆信息。
(5)损失函数
在训练深度神经网络的过程中，因为希望深度神经网络的输出尽可能的接近真正想要预测的值，所以可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重向量(当然，在第一次更新之前通常会有初始化的过程，即为深度神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重向量让它预测低一些，不断地调整，直到深度神经网络能够预测出真正想要的目标值或与真正想要的目标值非常接近的值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么深度神经网络的训练就变成了尽可能缩小这个loss的过程。
(6)反向传播算法
神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的神经网络模型中参数的大小，使得神经网络模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的神经网络模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的神经网络模型的参数，例如权重矩阵。
如图4所示，本申请实施例提供了一种系统架构100。在图4中，数据采集设备160用于采集训练数据。针对本申请实施例的图像处理方法来说，训练数据可以包括训练图像以及训练图像对应的分类结果，其中，训练图像的结果可以是人工预先标注的结果。
在采集到训练数据之后，数据采集设备160将这些训练数据存入数据库130，训练设备120基于数据库130中维护的训练数据训练得到目标模型/规则101。
下面对训练设备120基于训练数据得到目标模型/规则101进行描述，训练设备120对输入的原始图像进行处理，将输出的图像与原始图像进行对比，直到训练设备120输出的图像与原始图像的差值小于一定的阈值，从而完成目标模型/规则101的训练。
上述目标模型/规则101能够用于实现本申请实施例的图像处理方法。本申请实施例中的目标模型/规则101具体可以为神经网络。需要说明的是，在实际的应用中，所述数据库130中维护的训练数据不一定都来自于数据采集设备160的采集，也有可能是从其他设备接收得到的。另外需要说明的是，训练设备120也不一定完全基于数据库130维护的训练数据进行目标模型/规则101的训练，也有可能从云端或其他地方获取训练数据进行模型训练，上述描述不应该作为对本申请实施例的限定。
根据训练设备120训练得到的目标模型/规则101可以应用于不同的系统或设备中，如应用于图4所示的执行设备110，所述执行设备110可以是终端，如手机终端，平板电脑，笔记本电脑，增强现实(augmented reality，AR)AR/虚拟现实(virtual reality，VR)，车载终端等，还可以是服务器或者云端等。在图4中，执行设备110配置输入/输出(input/output，I/O)接口112，用于与外部设备进行数据交互，用户可以通过客户设备140 向I/O接口112输入数据，所述输入数据在本申请实施例中可以包括：客户设备输入的待处理图像。
预处理模块113和预处理模块114用于根据I/O接口112接收到的输入数据(如待处理图像)进行预处理，在本申请实施例中，也可以没有预处理模块113和预处理模块114(也可以只有其中的一个预处理模块)，而直接采用计算模块111对输入数据进行处理。
在执行设备110对输入数据进行预处理，或者在执行设备110的计算模块111执行计算等相关的处理过程中，执行设备110可以调用数据存储系统150中的数据、代码等以用于相应的处理，也可以将相应处理得到的数据、指令等存入数据存储系统150中。
最后，I/O接口112将处理结果，如上述得到的去噪处理后的图像返回给客户设备140，从而提供给用户。
值得说明的是，训练设备120可以针对不同的目标或称不同的任务，基于不同的训练数据生成相应的目标模型/规则101，该相应的目标模型/规则101即可以用于实现上述目标或完成上述任务，从而为用户提供所需的结果。
在图4中所示情况下，用户可以手动给定输入数据，该手动给定可以通过I/O接口112提供的界面进行操作。另一种情况下，客户设备140可以自动地向I/O接口112发送输入数据，如果要求客户设备140自动发送输入数据需要获得用户的授权，则用户可以在客户设备140中设置相应权限。用户可以在客户设备140查看执行设备110输出的结果，具体的呈现形式可以是显示、声音、动作等具体方式。客户设备140也可以作为数据采集端，采集如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果作为新的样本数据，并存入数据库130。当然，也可以不经过客户设备140进行采集，而是由I/O接口112直接将如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果，作为新的样本数据存入数据库130。
值得注意的是，图4仅是本申请实施例提供的一种系统架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制，例如，在图4中，数据存储系统150相对执行设备110是外部存储器，在其它情况下，也可以将数据存储系统150置于执行设备110中。
如图4所示，根据训练设备120训练得到目标模型/规则101，该目标模型/规则101在本申请实施例中可以是本申请中的神经网络，具体的，本申请实施例提供的神经网络可以CNN，深度卷积神经网络(deep convolutional neural networks,DCNN)，循环神经网络(recurrent neural network，RNNS)等等。
由于CNN是一种非常常见的神经网络，下面结合图5重点对CNN的结构进行详细的介绍。如上文的基础概念介绍所述，卷积神经网络是一种带有卷积结构的深度神经网络，是一种深度学习(deep learning)架构，深度学习架构是指通过机器学习的算法，在不同的抽象层级上进行多个层次的学习。作为一种深度学习架构，CNN是一种前馈(feed-forward)人工神经网络，该前馈人工神经网络中的各个神经元可以对输入其中的图像作出响应。
本申请实施例的图像处理方法具体采用的神经网络的结构可以如图5所示。在图5中，卷积神经网络(CNN)200可以包括输入层210，卷积层/池化层220(其中池化层为可选的)，以及神经网络层230。其中，输入层210可以获取待处理图像，并将获取到的待处 理图像交由卷积层/池化层220以及后面的神经网络层230进行处理，可以得到图像的处理结果。下面对图5中的CNN 200中内部的层结构进行详细的介绍。
卷积层/池化层220：
卷积层：
如图5所示卷积层/池化层220可以包括如示例221-226层，举例来说：在一种实现中，221层为卷积层，222层为池化层，223层为卷积层，224层为池化层，225为卷积层，226为池化层；在另一种实现方式中，221、222为卷积层，223为池化层，224、225为卷积层，226为池化层。即卷积层的输出可以作为随后的池化层的输入，也可以作为另一个卷积层的输入以继续进行卷积操作。
下面将以卷积层221为例，介绍一层卷积层的内部工作原理。
卷积层221可以包括很多个卷积算子，卷积算子也称为核，其在图像处理中的作用相当于一个从输入图像矩阵中提取特定信息的过滤器，卷积算子本质上可以是一个权重矩阵，这个权重矩阵通常被预先定义，在对图像进行卷积操作的过程中，权重矩阵通常在输入图像上沿着水平方向一个像素接着一个像素(或两个像素接着两个像素……这取决于步长stride的取值)的进行处理，从而完成从图像中提取特定特征的工作。该权重矩阵的大小应该与图像的大小相关，需要注意的是，权重矩阵的纵深维度(depth dimension)和输入图像的纵深维度是相同的，在进行卷积运算的过程中，权重矩阵会延伸到输入图像的整个深度。因此，和一个单一的权重矩阵进行卷积会产生一个单一纵深维度的卷积化输出，但是大多数情况下不使用单一权重矩阵，而是应用多个尺寸(行×列)相同的权重矩阵，即多个同型矩阵。每个权重矩阵的输出被堆叠起来形成卷积图像的纵深维度，这里的维度可以理解为由上面所述的“多个”来决定。不同的权重矩阵可以用来提取图像中不同的特征，例如一个权重矩阵用来提取图像边缘信息，另一个权重矩阵用来提取图像的特定颜色，又一个权重矩阵用来对图像中不需要的噪点进行模糊化等。该多个权重矩阵尺寸(行×列)相同，经过该多个尺寸相同的权重矩阵提取后的卷积特征图的尺寸也相同，再将提取到的多个尺寸相同的卷积特征图合并形成卷积运算的输出。
这些权重矩阵中的权重值在实际应用中需要经过大量的训练得到，通过训练得到的权重值形成的各个权重矩阵可以用来从输入图像中提取信息，从而使得卷积神经网络200进行正确的预测。
当卷积神经网络200有多个卷积层的时候，初始的卷积层(例如221)往往提取较多的一般特征，该一般特征也可以称之为低级别的特征；随着卷积神经网络200深度的加深，越往后的卷积层(例如226)提取到的特征越来越复杂，比如高级别的语义之类的特征，语义越高的特征越适用于待解决的问题。
池化层：
由于常常需要减少训练参数的数量，因此卷积层之后常常需要周期性的引入池化层，在如图5中220所示例的221-226各层，可以是一层卷积层后面跟一层池化层，也可以是多层卷积层后面接一层或多层池化层。在图像处理过程中，池化层的唯一目的就是减少图像的空间大小。池化层可以包括平均池化算子和/或最大池化算子，以用于对输入图像进行采样得到较小尺寸的图像。平均池化算子可以在特定范围内对图像中的像素值进行计算产生平均值作为平均池化的结果。最大池化算子可以在特定范围内取该范围内值最大的像 素作为最大池化的结果。另外，就像卷积层中用权重矩阵的大小应该与图像尺寸相关一样，池化层中的运算符也应该与图像的大小相关。通过池化层处理后输出的图像尺寸可以小于输入池化层的图像的尺寸，池化层输出的图像中每个像素点表示输入池化层的图像的对应子区域的平均值或最大值。
神经网络层230：
在经过卷积层/池化层220的处理后，卷积神经网络200还不足以输出所需要的输出信息。因为如前所述，卷积层/池化层220只会提取特征，并减少输入图像带来的参数。然而为了生成最终的输出信息(所需要的类信息或其他相关信息)，卷积神经网络200需要利用神经网络层230来生成一个或者一组所需要的类的数量的输出。因此，在神经网络层230中可以包括多层隐含层(如图5所示的231、232至23n)以及输出层240，该多层隐含层中所包含的参数可以根据具体的任务类型的相关训练数据进行预先训练得到，例如该任务类型可以包括图像识别，图像分类，图像超分辨率重建等等。
在神经网络层230中的多层隐含层之后，也就是整个卷积神经网络200的最后层为输出层240，该输出层240具有类似分类交叉熵的损失函数，具体用于计算预测误差，一旦整个卷积神经网络200的前向传播(如图5由210至240方向的传播为前向传播)完成，反向传播(如图5由240至210方向的传播为反向传播)就会开始更新前面提到的各层的权重值以及偏差，以减少卷积神经网络200的损失，及卷积神经网络200通过输出层输出的结果和理想结果之间的误差。
本申请实施例的图像处理方法具体采用的神经网络的结构可以如图6所示。在图6中，卷积神经网络(CNN)200可以包括输入层110，卷积层/池化层120(其中池化层为可选的)，以及神经网络层130。与图5相比，图6中的卷积层/池化层120中的多个卷积层/池化层并行，将分别提取的特征均输入给全神经网络层130进行处理。
需要说明的是，图5和图6所示的卷积神经网络仅作为一种本申请实施例的图像处理方法的两种可能的卷积神经网络的示例，在具体的应用中，本申请实施例的图像处理方法所采用的卷积神经网络还可以以其他网络模型的形式存在。
另外，采用本申请实施例的神经网络结构的搜索方法得到的卷积神经网络的结构可以如图5和图6中的卷积神经网络结构所示。
图7为本申请实施例提供的一种芯片的硬件结构，该芯片包括神经网络处理器50。该芯片可以被设置在如图1所示的执行设备110中，用以完成计算模块111的计算工作。该芯片也可以被设置在如图1所示的训练设备120中，用以完成训练设备120的训练工作并输出目标模型/规则101。如图5或图6所示的卷积神经网络中各层的算法均可在如图7所示的芯片中得以实现。
神经网络处理器NPU 50NPU作为协处理器挂载到主中央处理器(central processing unit，CPU)(host CPU)上，由主CPU分配任务。NPU的核心部分为运算电路50，控制器504控制运算电路503提取存储器(权重存储器或输入存储器)中的数据并进行运算。
在一些实现中，运算电路503内部包括多个处理单元(process engine,PE)。在一些实现中，运算电路503是二维脉动阵列。运算电路503还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路503是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路从权重存储器502中取矩阵B相应的数据，并缓存在运算电路中每一个PE上。运算电路从输入存储器501中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器(accumulator)508中。
向量计算单元507可以对运算电路的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。例如，向量计算单元507可以用于神经网络中非卷积/非FC层的网络计算，如池化(pooling)，批归一化(batch normalization)，局部响应归一化(local response normalization)等。
在一些实现种，向量计算单元能507将经处理的输出的向量存储到统一缓存器506。例如，向量计算单元507可以将非线性函数应用到运算电路503的输出，例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元507生成归一化的值、合并值，或二者均有。在一些实现中，处理过的输出的向量能够用作到运算电路503的激活输入，例如用于在神经网络中的后续层中的使用。
统一存储器506用于存放输入数据以及输出数据。
权重数据直接通过存储单元访问控制器505(direct memory access controller，DMAC)将外部存储器中的输入数据搬运到输入存储器501和/或统一存储器506、将外部存储器中的权重数据存入权重存储器502，以及将统一存储器506中的数据存入外部存储器。
总线接口单元(bus interface unit，BIU)510，用于通过总线实现主CPU、DMAC和取指存储器509之间进行交互。
与控制器504连接的取指存储器(instruction fetch buffer)509，用于存储控制器504使用的指令；
控制器504，用于调用指存储器509中缓存的指令，实现控制该运算加速器的工作过程。
入口：可以根据实际发明说明这里的数据是说明数据，比如探测到车辆速度？障碍物距离等
一般地，统一存储器506，输入存储器501，权重存储器502以及取指存储器509均为片上(On-Chip)存储器，外部存储器为该NPU外部的存储器，该外部存储器可以为双倍数据率同步动态随机存储器(double data rate synchronous dynamic random access memory，简称DDR SDRAM)、高带宽存储器(high bandwidth memory，HBM)或其他可读可写的存储器。
其中，图5或图6所示的卷积神经网络中各层的运算可以由运算电路303或向量计算单元307执行。
上文中介绍的图4中的执行设备110能够执行本申请实施例的图像处理方法的各个步骤，图5和图6所示的CNN模型和图7所示的芯片也可以用于执行本申请实施例的图像处理方法的各个步骤。下面结合附图对本申请实施例的图像处理方法和本申请实施例的图像处理方法进行详细的介绍。
如图8所示，本申请实施例提供了一种系统架构300。该系统架构包括本地设备301、本地设备302以及执行设备210和数据存储系统250，其中，本地设备301和本地设备302通过通信网络与执行设备210连接。
执行设备210可以由一个或多个服务器实现。可选的，执行设备210可以与其它计算设备配合使用，例如：数据存储器、路由器、负载均衡器等设备。执行设备210可以布置在一个物理站点上，或者分布在多个物理站点上。执行设备210可以使用数据存储系统250中的数据，或者调用数据存储系统250中的程序代码来实现本申请实施例的搜索神经网络结构的方法。
具体地，执行设备210可以执行以下过程：确定搜索空间和多个构建单元；堆叠所述多个构建单元，以得到搜索网络，所述搜索网络是用于搜索神经网络结构的神经网络；在所述搜索空间内对所述搜索网络中的构建单元的网络结构进行优化，以得到优化后的构建单元，其中，在优化过程中搜索空间逐渐减小，构建单元数量逐渐增加，搜索空间的减小和构建单元数量的增加使得所述优化过程中产生的显存消耗在预设范围内；根据所述优化后的构建单元搭建所述目标神经网络。
通过上述过程执行设备210能够搭建成一个目标神经网络，该目标神经网络可以用于图像分类或者进行图像处理等等。
用户可以操作各自的用户设备(例如本地设备301和本地设备302)与执行设备210进行交互。每个本地设备可以表示任何计算设备，例如个人计算机、计算机工作站、智能手机、平板电脑、智能摄像头、智能汽车或其他类型蜂窝电话、媒体消费设备、可穿戴设备、机顶盒、游戏机等。
每个用户的本地设备可以通过任何通信机制/通信标准的通信网络与执行设备210进行交互，通信网络可以是广域网、局域网、点对点连接等方式，或它们的任意组合。
在一种实现方式中，本地设备301、本地设备302从执行设备210获取到目标神经网络的相关参数，将目标神经网络部署在本地设备301、本地设备302上，利用该目标神经网络进行图像分类或者图像处理等等。
在另一种实现中，执行设备210上可以直接部署目标神经网络，执行设备210通过从本地设备301和本地设备302获取待处理图像，并根据目标神经网络对待处理图像进行分类或者其他类型的图像处理。
上述执行设备210也可以称为云端设备，此时执行设备210一般部署在云端。
下面先结合图9对本申请实施例的神经网络结构的搜索方法进行详细的介绍。图9所示的方法可以由神经网络构搜索置来执行，该神经网络结构搜索装置可以是电脑、服务器、云端设备等运算能力足以实现神经网络结构的搜索的设备。
图9所示的方法包括步骤1001至1004，下面分别对这些步骤进行详细的描述。
1001、确定搜索空间和多个构建单元。
可选地，上述搜索空间是根据待构建的目标神经网络的应用需求确定的。
也就是说，可以根据目标神经网络的应用需求来确定上述搜索空间。具体地，可以根据目标神经网络需要处理的数据的数据类型来确定上述搜索空间。
一般来说，搜索空间中包含的操作种类和数量要与目标神经网络的应用需求相匹配。当上述目标神经网络用于处理图像数据时，上述搜索空间包含的操作种类和数量要与图像数据的处理相适应，当上述目标神经网络用于处理语音数据时，上述搜索空间包含的操作种类和数量要与语音数据的处理相适应。
例如，当目标神经网络是用于处理图像数据的神经网络时，上述搜索空间可以包含卷 积操作，池化操作，跳连接(skip-connect)操作等等。
再如，当目标神经网络是用于处理语音数据的神经网络时，上述搜索空间可以包含激活函数(如ReLU、Tanh)等等。
可选地，上述搜索空间是根据目标神经网络的应用需求和执行神经网络结构搜索的设备的显存资源条件确定的。
其中，执行神经网络结构搜索的设备的显存资源条件可以是指执行神经网络结构搜索的设备的显存资源大小。
也就是说，在本申请中，可以根据目标神经网络的应用需求和执行神经网络结构搜索的设备的显存资源条件来综合确定上述搜索空间。
具体地，可以先根据目标神经网络的应用需求确定搜索空间包含的操作种类和数量，然后再根据执行神经网络结构搜索的设备的显存资源条件对搜索空间包含的操作种类和数量进行调整，以确定搜索空间最终包含的操作种类和数量。
例如，在根据目标神经网络的应用需求确定搜索空间包含的操作种类和数量之后，如果执行神经网络结构搜索的设备的显存资源较少，那么，可以将搜索空间中一些不太重要的操作删掉。而如果执行神经网络结构搜索的设备的显存资源较为充足时，可以保持搜索空间包含的操作种类和数量，或者增加搜索空间包含的操作种类和数量。
可选地，上述构建单元的数量是根据执行神经网络结构搜索的设备的显存资源条件来确定的。
具体地，可以根据执行神经网络结构搜索的设备的显存资源的大小来确定构建单元的数量。当执行神经网络结构搜索的设备的显存资源的较为充足时，构建单元的数量也可以比较多，当执行神经网络结构搜索的设备的显存资源比较有限时，构建单元的数量可以设置的小一些。
上述构建单元的数量也可以根据经验来设置，例如，可以根据经验确定一般需要堆叠多少数量的构建单元形成搜索网络。
上述多个构建单元中的每个构建单元可以是由多个节点之间通过神经网络的基本操作连接得到的网络结构，构建单元是用于构建神经网络的基础模块，利用构建单元可以搭建神经网络。
下面结合图10对本申请实施例的构建单元的结构进行简单的介绍，应理解，图10所示的构建单元只是一种可能的构建单元，图10所示的构建单元并不对本申请的构建单元的结构造成任何限定。
如图10所示，构建单元可以包括节点c_{k-2}和c_{k-1}以及节点0，节点1和节点2，其中，节点c_{k-2}和c_{k-1}为输入节点，节点0和1是中间节点，节点2为输出节点。在图10所示的构建单元可以接收节点c_{k-2}和c_{k-1}输出的数据，并由节点0和1分别对输入的数据进行处理，其中，节点0输出的数据还会输入到节点1中进行处理，节点0和节点1输出的数据会送入到节点2中进行处理，节点2最终输出该构建单元处理完的数据。
在上述图10中，各个节点表示的可以都是特征图。图10中的粗箭头表示一个或者多个基本操作，汇入同一个中间节点的基本操作运算结果在该中间节点处相加，图10中的细箭头表示通道维度的特征图连接，输出节点2输出的特征图由2个中间节点(节点0和 节点1)的输出按照顺序在特征图通道维度连接而成。
应理解，图10中的粗箭头和细箭头所对应的操作应是特定情况下涉及的操作，这里的相加和通道维度连接在此处都是为卷积神经网络而设计的，在其他情况下，构建单元的节点之间所对应的操作也可以是其他类型的运算或操作。
上述搜索空间包含的可以是预先设定好的卷积神经网络中的基础运算或者基础运算的组合，这些基础运算或者基础运算的组合可以统称为基本操作。
上述搜索空间可以包含以下8种基本操作：
(1)池化核大小为3×3的均值池化(avg_pool_3x3)；
(2)池化核大小为3×3的最大值池化(max_pool_3x3)；
(3)卷积核大小为3×3的分离卷积(sep_conv_3x3)；
(4)卷积核大小为5×5的分离卷积(sep_conv_5x5)；
(5)卷积核大小为3×3且空洞率为2的空洞卷积(dil_conv_3x3)；
(6)卷积核大小为5×5且空洞率为2的空洞卷积(dil_conv_5x5)；
(7)跳连接操作；
(8)置零操作(Zero，相应位置所有神经元置零)。
1002、堆叠多个构建单元，以得到搜索网络。
上述步骤1002中的搜索网络可以是用于搜索神经网络结构的神经网络。
可选地，上述堆叠多个构建单元，以得到搜索网络，包括：按照预设的堆叠方式将所述多个构建单元依次堆叠起来，以得到搜索网络，其中，在该搜索网络中，位于搜索网络前面的构建单元的输出是位于搜索网络的后面的构建单元的输入。
上述预设的堆叠方式可以是在什么位置堆放什么类型的构建单元，以及每种类型的构建单元堆叠的数量等等。
1003、在搜索空间内对搜索网络中的构建单元的网络结构进行优化，以得到优化后的构建单元。
在上述多个构建单元的每个构建单元中，每个节点的输出特征图的部分通道经过待选操作处理，得到处理后的特征图。每个节点的下一个节点的输入特征图是处理后的特征图与该每个节点剩余的未经过待选操作处理的特征图拼接后得到的特征图。
上述每个节点的下一个节点可以是指，该每个节点存在相连关系，并且该每个节点的输出特征图的部分通道经过待选操作处理后得到的特征图输入的节点。
具体地，在上述每个构建单元中，每个节点的输出特征图有c个通道，该每个节点的输出特征图的c/K个通道经过待选操作处理，得到处理后的特征图，该每个节点的下一个节点的输入特征图是处理后的特征图与剩余特征图拼接后得到的特征图。
其中，处理后的特征图有c/K个通道，剩余特征图有c(K-1)/K个通道，该剩余特征图为该每个节点输出的未经待选操作处理的特征图，c和K均为大于1的整数，上述待选操作包括所述搜索空间中的所有操作。
在本申请方案中，在多个构建单元的每个构建单元中，每个节点的输出特征图只有部分通道送入到待选操作进行处理，而处理后的特征图与剩余特征图拼接后输入到该每个节点的下一个节点，作为下一个节点的输入特征图。
例如，一个构建单元中包含第一节点和第二节点，第二节点为第一节点的下一个节点， 第一节点输出的特征图有c个通道，第一节点输出的特征图的c/K个通道经过待选操作处理，得到处理后的特征图，第二个节点的输入特征图是处理后的特征图与剩余特征图拼接得到的特征图，该剩余特征图为第一节点输出的未经过待选操作处理的特征图。
下面结合图11对上述特征图的处理过程进行说明。
如图11，节点0输出第一特征图，第一特征图中的部分特征图进行待选操作处理后，得到第二特征图，第二特征图可以在经过通道顺序调换后输入到节点1中继续进行处理。
具体地，如图11所示，第一特征图的宽为w，高为h，通道数为c。按照通道数可以将第一特征图分为特征图A和特征图B，特征图A的通道数为c/K，特征图B的通道数为c(K-1)/K。其中，特征图A经过待选操作进行处理，这里的待选操作包括操作1至操作8，经过这8个操作处理后的输出分别与各自操作的权重参数(W1至W8)加权求和后，得到特征图A’。
上述特征图B并未经过待选操作，在得到特征图A’之后，可以将特征图A’与特征图B在通道的维度上进行拼接，得到第二特征图。
在上述过程中，仅选择第一特征图的c/K通道进行待选操作处理，能够减少计算的复杂度，可以减少搜索开销。
上述K可以是2或者2的倍数。例如，K可以是2,4,6,8等等。
一般来说，当K的数值越大时，相应的搜索开销越小，但是随着K的数值的增大，最终搜索得到的目标神经网络的性能可能会受到一定的影响。
可选地，上述K的数值根据执行图9所示的方法的设备的显存资源的大小来确定。
具体地，当执行上述方法的设备的显存资源较多时，K可以设置成一个较小的数值，当执行上述方法的设备的显存资源较少时，K可以设置成一个较大的数值。
例如，当执行上述方法的设备的显存资源较多时，K的取值为4，当执行上述方法的设备的显存资源较少时，K的取值为8。
还可以结合执行上述方法的设备的显存资源的大小和目标神经网络的性能综合确定K的数值。
具体地，可以先根据执行上述方法的设备的显存资源的大小设置K的数值，然后再根据目标神经网络在该K值的情况下的性能，对K的数值进行调整。
例如，根据执行上述方法的设备的显存资源的大小，可以将K设置成6或8，但是目标神经网络在K＝8时的性能不够理想，而在目标神经网络在K＝6时的性能符合要求，那么，就可以将K设置为6。
上述目标神经网络在K取不同数值时的性能可以根据测试结果得到。
例如，上述K＝4，如图10所示，节点0输出的特征图包括c个通道，那么，节点0输出的c个通道的特征图中，可以有c/4个通道经过待选操作处理后得到处理后的特征图，而节点0输出的特征图中剩余的未被待选操作处理的特征图有3c/4个通道，那么，处理后的特征图(通道数为c/4)与剩余的特征图(通道数为3c/4)拼接后得到拼接后的特征图，该拼接后的特征图的通道数为c，该拼接后的特征图可以输入到节点1中。
1004、根据优化后的构建单元搭建目标神经网络。
本申请中，在神经网络结构的搜索过程中，由于构建单元中的每个节点的输出特征图只有部分通道的特征图进行待选择操作处理，能够降低待选操作处理的特征图的通道数 量，进而减少搜索时占用的显存，减少搜索开销。
另外，本申请中通过选择每个节点的输出特征图中部分通道进行待选操作处理，减少了待选操作处理的数据量，可以在一定程度上降低最终搭建的目标网络出现过拟合的可能性。
进一步的，由于本申请能够减少搜索过程中占用的显存，因此，本申请能够在同样显存资源的情况下，增加网络搜索过程处理的每批数据的数据量，实现对更复杂的神经网络结构的搜索。
在本申请中，为了使得拼接得到的特征图在输入到下一个节点之后，能够随机选择该拼接后的特征图中的部分通道的特征图进行处理，可以将拼接后的特征图调换顺序后再输入到下一个节点。
可选地，作为一个实施例，每个节点的下一个节点的输入特征图是处理后的特征图与剩余特征图拼接后得到的特征图，包括：每个节点的下一个节点的输入特征图是处理后的特征图与剩余的特征图经过拼接和通道顺序调换后得到的特征图。上述调换通道顺序可以是将拼接后的特征图的通道拼接顺序重新进行调整。
上述调换通道顺序可以是将拼接后的特征图的通道拼接顺序重新进行调整。
在得到处理后的特征图之后，可以将处理后的特征图与剩余的特征图进行拼接，得到拼接后的特征图，然后再对拼接后的特征图的通道顺序进行调换，然后将通道顺序调换后得到的特征图输入到下一个节点。
或者，在得到上述处理后的特征图之后，还可以先进行通道顺序的调换，然后再将顺序调换后得到的特征图拼接在一起，将拼接后得到的特征图输入到下一个节点进行处理。
下面结合图12和图13对特征图的通道顺序调换的过程进行说明。
例如，如图12所示，可以将处理后的特征图的c/K个通道进行移位，将该处理后的特征图的c/K个通道从特征图的一端移动到另一端，从而得到拼接和通道顺序调换后的特征图。具体地，在图12中，特征图A’与特征图B可以调换顺序。
再如，如图13所示，还可以拼接后的特征图中的c/K个通道插入到拼接后的特征图中的c(K-1)/K通道中，从而得到拼接和通道顺序调换后的特征图，具体地，在图13中，可以将特征图B拆分为特征图B1和特征图B2，特征图A’插入到特征图B1和特征图B2之间。
另外，除了图12和图13所示的通道顺序调换方式之外，还可以将处理后的特征图按照通道分成多个子特征图，然后将子特征图分布在剩余特征图的不同的通道中间，从而得到拼接和通道顺序调换后的特征图。
本申请中，通过将拼接后的特征图的通道的顺序进行调换，能够将拼接后的特征图的通道顺序打乱，使得下一个节点可以随机选择部分通道的特征图进行处理，可以增强输入数据的随机性，从而尽可能的避免最终得到的目标神经网络出现过拟合的现象。
在本申请中，为了增加最终搜索到的目标神经网络的稳定性，还可以为构建单元的节点之间的边设置边权重参数。并在对搜索网络中的构建单元进行优化的过程中，保留参数值较大的边权重参数对应的边，舍弃参数值较小的边权重参数较小对应的边，从而使得最终构建得到的目标神经网络保持一定的稳定性。
可选地，上述多个构建单元中的每个构建单元包括输入节点和多个中间节点，每个构 建单元的节点之间的连接构成边，该多个中间节点中的每个中间节点的输入为对应的多个输入特征图分别与各自对应的边权重参数的乘积的和，其中，每个输入对应一个边权重参数，每个输入对应的边权重参数用于指示每个输入到每个中间节点时的权重。
例如，如图14所示，假设节点3为中间节点，节点0、节点1和节点2的输出特征图中的部分通道处理后与剩余的特征图拼接后输入到节点3，这时可以为每条边设置不同的权重，将0与3之间的边权重参数设置为E1，将1与3之间的边权重参数设置为E2，将2与3之间的边权重参数设置为E3，在叠加时每条边上输出的特征图分别与各自的权重相乘后进行叠加，得到节点3的输入特征图。
如图14所示，3条边输出的特征图分别为F1、F2和F3，那么，可以根据公式(1)得到节点3的输入特征图。
F＝E1*F1+E1*F1+E1*F1       (1)
在公式(1)中，F表示节点3的输入特征图，F1、F2和F3分别为3条边的输出特征图，分别为3条边对应的边权重参数。
本申请中，通过设置边权重参数，能够衡量不同边的重要程度，并在优化的过程中可以选择出对应的边权重参数值比较大的边，舍弃边权重参数值比较小的边，从而使得最终构建得到的目标神经网络保持一定的稳定性。
上述步骤1003中对构建单元的网络结构进行优化，以得到优化后的构建单元，具体包括：在搜索空间内，对搜索网络中的构建单元的网络结构参数进行调整，得到优化后的构建单元。
其中，优化后的构建单元保留部分边权重参数对应的边，优化后的构建单元的保留部分待选操作权重参数对应的操作。
在对构建单元的网络结构进行优化时，可以选择训练数据和验证数据，其中，训练数据用于训练卷积参数，验证数据训练网络结构参数。在训练过程中可以先按照训练数据计算损失函数，调整卷积参数，再依据验证数据计算损失函数调整网络结构参数，这样不断交替迭代训练，完成训练后就得到了优化后的构建单元。
可选地，上述多个构建单元中的每个构建单元包括输入节点和多个中间节点，多个中间节点中的每个中间节点对应一个层级，其中，第一层级的中间节点与所述输入节点相连，第i层级的中间节点与所述输入节点相连，第i层级的中间节点与前i-1个层级的中间节点相连，i为大于1的整数。
上述搜索网络可以包含多种类型的构建单元，下面简单介绍一下搜索网络包含的常见的构建单元。
可选地，上述搜索网络中的构建单元包括第一类构建单元。
其中，第一类构建单元是输入特征图的数量(具体可以是通道数)和大小分别与输出特征图的数量和大小相同的构建单元。
例如，某个第一类构建单元的输入的是大小为C×D1×D2(C为通道数，D1和D2分别是宽和高)的特征图，经过该第一类构建单元处理后输出的特征图的大小仍然是C×D1×D2。
上述第一类构建单元具体可以是普通单元(normal cell)
可选地，上述搜索网络中的构建单元包括第二类构建单元。
其中，第二类构建单元的输出特征图的分辨率是输入特征图的1/M，第二类构建单元的输出特图的数量是输入特征图的数量的M倍，M为大于1的正整数。
上述M的取值一般可以是2、4、6和8等数值。
例如，某个第二类构建单元的输入是1个大小为C×D1×D2(C为通道数，D1和D2分别是宽和高，C1和C2的乘积可以表示特征图的分辨率)的特征图，那么，经过该第二类构建单元处理后，得到的1个大小为
Figure PCTCN2020089403-appb-000011
的特征图。
上述第二类构建单元具体可以是下采样单元(redution cell)。
当搜索网络由上述第一类构建单元和第二类构建单元组成时，搜索网络的结构可以如图15所示。
如图15，搜索网络由5个构建单元依次堆叠而成，其中，位于搜索网络最前端和最后端的是第一类构建单元，每两个第一构建单元之间存在一个第二类构建单元。
图15中的搜索网络中的第一个构建单元能够对输入的图像进行处理，第一类构建单元对图像进行处理后，将处理得到的特征图输入到第二类构建单元进行处理，这样依次向后传输，直到搜索网络中的最后一个第一类构建单元输出特征图。
搜索网络的最后一个第一类构建单元输出的特征图送入到分类器中进行处理，由分类器根据特征图对图像进行分类。
为了更好地理解本申请实施例的神经网络结构搜索方法，下面结合图16对本申请实施例的神经网络结构搜索方法的整体过程进行简单的介绍。
如图16所示，可以根据待构建的神经网络的任务需求(也就是待构建的神经网络需要处理任务的任务类型)来确定构建何种类型的神经网络。接下来，再根据该神经网络处理的任务需求，确定搜索空间的大小和构建单元的数量，并对构建单元进行堆叠，得到搜索网络。在得到搜索网络之后就可以对搜索网络中的构建单元的网络结构进行优化(可以采用训练数据进行优化)了，对构建单元的网络结构进行优化的过程中可以采用通道采样结构搜索，在进行通道采样结构搜索时主要的改进包括通道采样(就是每个节点的输出特征图只有部分通道经过待选操作处理，并且处理后的特征图与未处理的特征图经过拼接后输入到下一个节点)和边正则化(在构建单元中，每个中间节点的输入为对应的多个输入特征图分别与各自对应的边权重参数的乘积的和)。经过优化后的构建单元可以构成神经网络并输出。图16中虚线框内的优化过程相当于上文中的步骤1003的优化过程。
本申请实施例的神经网络结构的搜索方法可以由神经网络结构搜索系统来执行，图17示出了神经网络结构搜索系统执行本申请实施例的神经网络结构搜索方法的过程。下面对图17所示的内容进行详细介绍。
图17所示的神经网络结构搜索系统中，操作仓库101可以包含预先设定好的卷积神经网络中的基本操作。
该操作仓库101可以包含以下8种基本操作：
(1)池化核大小为3×3的均值池化(avg_pool_3x3)；
(2)池化核大小为3×3的最大值池化(max_pool_3x3)；
(3)卷积核大小为3×3的分离卷积(sep_conv_3x3)；
(4)卷积核大小为5×5的分离卷积(sep_conv_5x5)；
(5)卷积核大小为3×3且空洞率为2的空洞卷积(dil_conv_3x3)；
(6)卷积核大小为5×5且空洞率为2的空洞卷积(dil_conv_5x5)；
(7)跳连接操作；
(8)置零操作(Zero，相应位置所有神经元置零)。
通道采样结构搜索模块102用于对搜索网络的构建单元的网络结构进行优化，在优化过程中，在每个构建单元中，每个节点的输出特征图只有部分通道经过待选操作处理，并且经过处理后的特征图与未经处理的特征图经过拼接后作为下一个节点的输入特征图。
构建单元103是优化后的构建单元，能够用于搭建目标神经网络。
具体地，可以根据目标任务确定操作仓库101(相当于上文中的搜索空间)的大小和初始数量的构建单元，然后根据初始数量的构建单元堆叠得到搜索网络。接下来，可以对构建单元进行优化，得到优化后的构建单元。在优化的过程中，经过通道采样1022实现每个节点的输出特征图只有部分通道进行待选操作处理，而经过边正则化1023能够实现构建单元中相连的节点对应的有边权重参数。最终优化得到的构建单元为构建单元103，构建单元103的数量一般为多个，根据该多个构建单元103就可以搭建目标神经网络了。
在图17中，对构建单元1021的优化过程相当于图9所示的方法中的步骤1003中的优化过程。具体优化过程可参见步骤1003的相关描述。
为了说明本申请实施例的神经网络结构搜索方法的效果，下面对本申请实施例的神经网络结构搜索方法和现有方案进行对比。表1示出了在相似约束条件下本申请实施例的神经网络结构搜索方法和现有方案在图像分类数据集上的分类准确率和搜索开销。
在表1中，NASNet-A、AmoebaNet-B、ENAS、PNAS、DARTS(2ND)和SNAS分别表示传统方案的网络结构，表1示出了传统方案和本申请在数据集CIFAR10下的搜索开销。由表1可知，本申请方案与传统方案相比，搜索开销大大减少。
表1
网络结构 搜索开销(GDs)
NASNet-A 1800
AmoebaNet-B 3150
ENAS 0.5
PNAS 225
DARTS(2ND) 4
SNAS 1.5
本申请 0.1
在表2中，ProxylessNAS为现有方案的一种网络架构，表2示出了现有方案和本申请方案在数据集ImageNet下的搜索开销。
表2
网络结构 搜索开销(GDs)
ProxylessNAS 8.3
本申请 3.8
由表2可知，在数据集ImageNet下本申请方案相对于传统方案，搜索开销也大大减少。
下面结合表3对本申请方案以及传统方案的分类效果进行说明。
在表3中，CIFAR10和ImageNet为不同的数据集，ImageNetTop1和ImageNetTop5是子指标，是指在ImageNet数据集中前1个或者前5个结果中出现正确结果的比例(准确率)。NASNet-A、AmoebaNet-B、ENAS、PNAS、DARTS(2ND)和SNAS分别表示不同的网络结构，CIFR10、ImageNetTop1、ImageNetTop5所在的列下面的数据表示分类准确率。
表3
Figure PCTCN2020089403-appb-000012
由表3可知，本申请与现有方案相比，在数据集CIFR10和ImageNet下图像分类的准确率还有所提高。
下面结合数据集CIFAR10对本申请实施例的神经网络结构的搜索方法的效果进行详细的介绍。如表4可知，神经网络结构搜索方法不采用通道采样也不采用边正则化(相当于是传统方案)时，分类的准确率为97％，搜索开销为0.4GDs。而当神经网络结构搜索方法仅采用边正则化而不采用通道采样时，分类的准确率为97.18，准确率有所提高，而搜索开销仍为0.4GDs，没有发生变化。而当神经网络结构搜索方法既采用了通道采样又采用了边正则化时，分类的准确率为97.43，分类准确率有所提高，搜索开销为0.1GDs，搜索开销也大大减少。
表4
通道采样 边正则化 CIFAR10 搜索开销
× × 97.00 0.4GDs
× √ 97.18 0.4GDs
√ √ 97.43 0.1GDs
上文结合附图对本申请实施例的神经网络结构的搜索方法进行了详细的介绍，本申请实施例的神经网络结构的搜索方法构建得到的神经网络可以用于图像处理(例如，图像分类)等，下面对这些具体应用进行介绍。
图18是本申请实施例的图像处理方法的示意性流程图。应理解，上文中对图9所示的方法的相关内容限定、解释和扩展同样适用于图18所示的方法，下面在介绍图18所示的方法时适当省略重复的描述。图18所示的方法包括：
2001、获取待处理图像；
2002、根据目标神经网络对待处理图像进行分类，得到待处理图像的分类结果。
其中，目标神经网络由多个优化后的构建单元搭建而成，多个优化后的构建单元是通过对搜索网络中的多个构建单元的网络结构进行优化得到的，在多个构建单元的每个构建单元中，每个节点的输出特征图的c/K个通道经过待选操作处理，得到处理后的特征图， 每个节点的下一个节点的输入特征图是处理后的特征图与剩余特征图拼接后得到的特征图，每个节点的输出特征图有c个通道，处理后的特征图有c/K个通道，剩余特征图有c(K-1)/K个通道，剩余特征图为每个节点输出的未经待选操作处理的特征图，待选操作包括搜索空间中的所有操作，c和K均为大于1的整数。
应理解，在图18所示的方法中，目标神经网络是由优化后的构建单元搭建而成，并且该目标神经网络可以是经过训练数据(包括训练图像，以及训练图像的分类结果)进行训练后的神经网络。在目标神经网络对待处理图像进行处理时，每个优化后的构建单元中的每个节点输出的特征图的全部通道都会经过待选操作处理，经过待选操作处理后的特征图再输入到下一个节点，这是利用目标神经网络进行图像处理与神经网络结构搜索处理以得到目标神经网络的不同之处。
本申请中，由于目标神经网络在构建过程中，每个构建单元中的每个节点的输出特征图的部分通道经过待选操作处理，可以减少最终得到的目标神经网络发生过拟合的现象，因此，利用该目标神经网络能够更好地进行图像分类。
可选地，上述每个节点的下一个节点的输入特征图是处理后的特征图与剩余特征图拼接后得到的特征图，包括：每个节点的下一个节点的输入特征图是处理后的特征图与剩余的特征图经过拼接和通道顺序调换后得到的特征图。
本申请中，由于目标神经网络在构建过程中，通过将拼接后的特征图的通道的顺序进行调换，能够将拼接后的特征图的通道顺序打乱，使得下一个节点可以随机选择部分通道的特征图进行处理，可以增强输入数据的随机性，从而尽可能的避免最终得到的目标神经网络出现过拟合的现象，因此，利用该目标神经网络能够更好地进行图像分类。
可选地，上述多个构建单元中的每个构建单元包括输入节点和多个中间节点，每个构建单元的节点之间的连接构成边，多个中间节点中的每个中间节点的输入为对应的多个输入分别与各自对应的边权重参数的乘积的和，其中，每个输入对应一个边权重参数，每个输入对应的边权重参数用于指示每个输入到每个中间节点时的权重。
本申请中，由于目标神经网络在构建过程中，通过设置边权重参数，能够衡量不同边的重要程度，并在优化的过程中可以选择出对应的边权重参数值比较大的边，舍弃边权重参数值比较小的边，从而使得最终构建得到的目标神经网络保持一定的稳定性。
可选地，上述多个构建单元中的每个构建单元包括输入节点和多个中间节点，多个中间节点中的每个中间节点对应一个层级，其中，第一层级的中间节点与输入节点相连，第i层级的中间节点与输入节点相连，第i层级的中间节点与前i-1个层级的中间节点相连，i为大于1的整数。
图19是本申请实施例提供的神经网络结构搜索装置的硬件结构示意图。图19所示的神经网络结构搜索装置3000(该装置3000具体可以是一种计算机设备)包括存储器3001、处理器3002、通信接口3003以及总线3004。其中，存储器3001、处理器3002、通信接口3003通过总线3004实现彼此之间的通信连接。
存储器3001可以是只读存储器(read only memory，ROM)，静态存储设备，动态存储设备或者随机存取存储器(random access memory，RAM)。存储器3001可以存储程序，当存储器3001中存储的程序被处理器3002执行时，处理器3002用于执行本申请实施例的神经网络结构的搜索方法的各个步骤。
处理器3002可以采用通用的中央处理器(central processing unit，CPU)，微处理器，应用专用集成电路(application specific integrated circuit，ASIC)，图形处理器(graphics processing unit，GPU)或者一个或多个集成电路，用于执行相关程序，以实现本申请方法实施例的神经网络结构的搜索方法。
处理器3002还可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，本申请的神经网络结构的搜索方法的各个步骤可以通过处理器3002中的硬件的集成逻辑电路或者软件形式的指令完成。
上述处理器3002还可以是通用处理器、数字信号处理器(digital signal processing，DSP)、专用集成电路(ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器3001，处理器3002读取存储器3001中的信息，结合其硬件完成本神经网络结构搜索装置中包括的单元所需执行的功能，或者执行本申请方法实施例的神经网络结构的搜索方法。
通信接口3003使用例如但不限于收发器一类的收发装置，来实现装置3000与其他设备或通信网络之间的通信。例如，可以通过通信接口3003获取待构建的神经网络的信息以及构建神经网络过程中需要的训练数据。
总线3004可包括在装置3000各个部件(例如，存储器3001、处理器3002、通信接口3003)之间传送信息的通路。
图20是本申请实施例的图像处理装置的硬件结构示意图。图20所示的图像处理装置4000包括存储器4001、处理器4002、通信接口4003以及总线4004。其中，存储器4001、处理器4002、通信接口4003通过总线4004实现彼此之间的通信连接。
存储器4001可以是ROM，静态存储设备和RAM。存储器4001可以存储程序，当存储器4001中存储的程序被处理器4002执行时，处理器4002和通信接口4003用于执行本申请实施例的图像处理方法的各个步骤。
处理器4002可以采用通用的，CPU，微处理器，ASIC，GPU或者一个或多个集成电路，用于执行相关程序，以实现本申请实施例的图像处理装置中的单元所需执行的功能，或者执行本申请方法实施例的图像处理方法。
处理器4002还可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，本申请实施例的图像处理方法的各个步骤可以通过处理器4002中的硬件的集成逻辑电路或者软件形式的指令完成。
上述处理器4002还可以是通用处理器、DSP、ASIC、FPGA或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存 储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器4001，处理器4002读取存储器4001中的信息，结合其硬件完成本申请实施例的图像处理装置中包括的单元所需执行的功能，或者执行本申请方法实施例的图像处理方法。
通信接口4003使用例如但不限于收发器一类的收发装置，来实现装置4000与其他设备或通信网络之间的通信。例如，可以通过通信接口4003获取待处理图像。
总线4004可包括在装置4000各个部件(例如，存储器4001、处理器4002、通信接口4003)之间传送信息的通路。
图21是本申请实施例的神经网络训练装置的硬件结构示意图。与上述装置3000和装置4000类似，图21所示的神经网络训练装置5000包括存储器5001、处理器5002、通信接口5003以及总线5004。其中，存储器5001、处理器5002、通信接口5003通过总线5004实现彼此之间的通信连接。
在通过图19所示的神经网络结构搜索装置构建得到了神经网络之后，可以通过图21所示的神经网络训练装置5000对该神经网络进行训练，训练得到的神经网络就可以用于执行本申请实施例的图像处理方法了。
具体地，图21所示的装置可以通过通信接口5003从外界获取训练数据以及待训练的神经网络，然后由处理器根据训练数据对待训练的神经网络进行训练。
应注意，尽管上述装置3000、装置4000和装置5000仅仅示出了存储器、处理器、通信接口，但是在具体实现过程中，本领域的技术人员应当理解，装置3000、装置4000和装置5000还可以包括实现正常运行所必须的其他器件。同时，根据具体需要，本领域的技术人员应当理解，装置3000、装置4000和装置5000还可包括实现其他附加功能的硬件器件。此外，本领域的技术人员应当理解，装置3000、装置4000和装置5000也可仅仅包括实现本申请实施例所必须的器件，而不必包括图19、图20和图21中所示的全部器件。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络 单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (14)

  1. 一种神经网络结构的搜索方法，其特征在于，包括：
    确定搜索空间和多个构建单元，所述多个构建单元中的每个构建单元是由多个节点之间通过神经网络的基本操作连接得到的网络结构；
    堆叠所述多个构建单元，得到搜索网络，在所述多个构建单元的每个构建单元中，每个节点的输出特征图的c/K个通道经过待选操作处理，得到处理后的特征图，所述每个节点的下一个节点的输入特征图是所述处理后的特征图与剩余特征图拼接后得到的特征图，所述每个节点的输出特征图有c个通道，所述处理后的特征图有c/K个通道，所述剩余特征图有c(K-1)/K个通道，所述剩余特征图为所述每个节点输出的未经所述待选操作处理的特征图，所述待选操作包括所述搜索空间中的所有操作，c和K均为大于1的整数；
    在所述搜索空间内对所述搜索网络中的构建单元的网络结构进行优化，得到优化后的构建单元；
    根据所述优化后的构建单元搭建所述目标神经网络。
  2. 如权利要求1所述的方法，其特征在于，所述每个节点的下一个节点的输入特征图是所述处理后的特征图与剩余特征图拼接后得到的特征图，包括：
    所述每个节点的下一个节点的输入特征图是所述处理后的特征图与剩余的特征图经过拼接和通道顺序调换后得到的特征图。
  3. 如权利要求1或2所述的方法，其特征在于，所述多个构建单元中的每个构建单元包括输入节点和多个中间节点，每个构建单元的节点之间的连接构成边，所述多个中间节点中的每个中间节点的输入为对应的多个输入分别与各自对应的边权重参数的乘积的和，其中，每个输入对应一个边权重参数，每个输入对应的边权重参数用于指示每个输入到所述每个中间节点时的权重。
  4. 如权利要求3所述的方法，其特征在于，在所述搜索空间内对所述搜索网络中的构建单元的网络结构进行优化，以得到优化后的构建单元，包括：
    在所述搜索空间内，对所述搜索网络中的构建单元的网络结构参数进行调整，得到优化后的构建单元，所述网络结构参数包括待选操作权重参数和边权重参数，所述优化后的构建单元保留部分边权重参数对应的边，优化后的构建单元的保留部分待选操作权重参数对应的操作。
  5. 如权利要求1-4中任一项所述的方法，其特征在于，所述多个构建单元中的每个构建单元包括输入节点和多个中间节点，所述多个中间节点中的每个中间节点对应一个层级，其中，第一层级的中间节点与所述输入节点相连，第i层级的中间节点与所述输入节点相连，所述第i层级的中间节点与前i-1个层级的中间节点相连，i为大于1的整数。
  6. 如权利要求1-5中任一项所述的方法，其特征在于，K是根据执行所述方法的设备的显存资源的大小确定的。
  7. 一种图像处理方法，其特征在于，包括：
    获取待处理图像；
    根据目标神经网络对所述待处理图像进行分类，得到所述待处理图像的分类结果；
    其中，所述目标神经网络由多个优化后的构建单元搭建而成，所述多个优化后的构建单元是通过对搜索网络中的多个构建单元的网络结构进行优化得到的，在所述多个构建单元的每个构建单元中，每个节点的输出特征图的c/K个通道经过待选操作处理，得到处理后的特征图，所述每个节点的下一个节点的输入特征图是所述处理后的特征图与剩余特征图拼接后得到的特征图，所述每个节点的输出特征图有c个通道，所述处理后的特征图有c/K个通道，所述剩余特征图有c(K-1)/K个通道，所述剩余特征图为所述每个节点输出的未经所述待选操作处理的特征图，所述待选操作包括所述搜索空间中的所有操作，c和K均为大于1的整数。
  8. 如权利要求7所述的方法，其特征在于，所述每个节点的下一个节点的输入特征图是所述处理后的特征图与剩余特征图拼接后得到的特征图，包括：
    所述每个节点的下一个节点的输入特征图是所述处理后的特征图与剩余的特征图经过拼接和通道顺序调换后得到的特征图。
  9. 如权利要求7或8所述的方法，其特征在于，所述多个构建单元中的每个构建单元包括输入节点和多个中间节点，每个构建单元的节点之间的连接构成边，所述多个中间节点中的每个中间节点的输入为对应的多个输入分别与各自对应的边权重参数的乘积的和，其中，每个输入对应一个边权重参数，每个输入对应的边权重参数用于指示每个输入到所述每个中间节点时的权重。
  10. 如权利要求7-9中任一项所述的方法，其特征在于，所述多个构建单元中的每个构建单元包括输入节点和多个中间节点，所述多个中间节点中的每个中间节点对应一个层级，其中，第一层级的中间节点与所述输入节点相连，第i层级的中间节点与所述输入节点相连，所述第i层级的中间节点与前i-1个层级的中间节点相连，i为大于1的整数。
  11. 一种神经网络结构搜索装置，其特征在于，包括：
    存储器，用于存储程序；
    处理器，用于执行所述存储器存储的程序，当所述存储器存储的程序被执行时，所述处理器用于执行以下过程：
    确定搜索空间和多个构建单元，所述多个构建单元中的每个构建单元是由多个节点之间通过神经网络的基本操作连接得到的网络结构；
    堆叠所述多个构建单元，得到搜索网络，在所述多个构建单元的每个构建单元中，每个节点输出特征图的c/K个通道经过待选操作处理，得到处理后的特征图，所述每个节点的下一个节点的输入特征图是所述处理后的特征图与剩余特征图拼接后得到的特征图，所述每个节点的输出特征图有c个通道，所述处理后的特征图有c/K个通道，所述剩余特征图有c(K-1)/K个通道，所述剩余特征图为所述每个节点输出的未经所述待选操作处理的特征图，所述待选操作包括所述搜索空间中的所有操作，c和K均为大于1的整数；
    在所述搜索空间内对所述搜索网络中的构建单元的网络结构进行优化，得到优化后的构建单元；
    根据所述优化后的构建单元搭建所述目标神经网络。
  12. 一种图像处理装置，其特征在于，包括：
    存储器，用于存储程序；
    处理器，用于执行所述存储器存储的程序，当所述存储器存储的程序被执行时，所述 处理器用于执行以下过程：
    获取待处理图像；
    根据目标神经网络对所述待处理图像进行分类，得到所述待处理图像的分类结果；
    其中，所述目标神经网络由多个优化后的构建单元搭建而成，所述多个优化后的构建单元是通过对搜索网络中的多个构建单元的网络结构进行优化得到的，在所述多个构建单元的每个构建单元中，每个节点的输出特征图的c/K个通道经过待选操作处理，得到处理后的特征图，所述每个节点的下一个节点的输入特征图是所述处理后的特征图与剩余特征图拼接后得到的特征图，所述每个节点的输出特征图有c个通道，所述处理后的特征图有c/K个通道，所述剩余特征图有c(K-1)/K个通道，所述剩余特征图为所述每个节点输出的未经所述待选操作处理的特征图，所述待选操作包括所述搜索空间中的所有操作，c和K均为大于1的整数。
  13. 一种计算机可读存储介质，其特征在于，所述计算机可读介质存储用于设备执行的程序代码，该程序代码包括用于执行如权利要求1-6或者7-10中任一项所述的方法。
  14. 一种芯片，其特征在于，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，以执行如权利要求1-6或者7-10中任一项所述的方法。
PCT/CN2020/089403 2019-07-12 2020-05-09 神经网络结构的搜索方法、图像处理方法和装置 WO2021008206A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/573,220 US12026938B2 (en) 2019-07-12 2022-01-11 Neural architecture search method and image processing method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910627480.7 2019-07-12
CN201910627480.7A CN112215332B (zh) 2019-07-12 2019-07-12 神经网络结构的搜索方法、图像处理方法和装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/573,220 Continuation US12026938B2 (en) 2019-07-12 2022-01-11 Neural architecture search method and image processing method and apparatus

Publications (1)

Publication Number Publication Date
WO2021008206A1 true WO2021008206A1 (zh) 2021-01-21

Family

ID=74048241

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/089403 WO2021008206A1 (zh) 2019-07-12 2020-05-09 神经网络结构的搜索方法、图像处理方法和装置

Country Status (2)

Country Link
CN (1) CN112215332B (zh)
WO (1) WO2021008206A1 (zh)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111241985B (zh) * 2020-01-08 2022-09-09 腾讯科技（深圳）有限公司 一种视频内容识别方法、装置、存储介质、以及电子设备
CN112861951B (zh) * 2021-02-01 2024-03-26 上海依图网络科技有限公司 一种图像神经网络参数的确定方法及电子设备
CN113052812B (zh) * 2021-03-22 2022-06-24 山西三友和智慧信息技术股份有限公司 一种基于AmoebaNet的MRI前列腺癌检测方法
CN115700771A (zh) * 2021-07-31 2023-02-07 华为技术有限公司 编解码方法及装置
CN113902094A (zh) * 2021-09-16 2022-01-07 昆明理工大学 面向语言模型的双单元搜索空间的结构搜索方法
CN113986816B (zh) * 2021-12-09 2023-05-02 北京奕斯伟计算技术股份有限公司 可重构计算芯片
CN115760777B (zh) * 2022-11-21 2024-04-30 脉得智能科技(无锡)有限公司 基于神经网络结构搜索的桥本氏甲状腺炎诊断系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090132449A1 (en) * 2006-05-22 2009-05-21 Fujitsu Limited Neural network learning device, method, and program
CN109284820A (zh) * 2018-10-26 2019-01-29 北京图森未来科技有限公司 一种深度神经网络的结构搜索方法及装置
CN109919304A (zh) * 2019-03-04 2019-06-21 腾讯科技（深圳）有限公司 神经网络搜索方法、装置、可读存储介质和计算机设备
CN109978142A (zh) * 2019-03-29 2019-07-05 腾讯科技（深圳）有限公司 神经网络模型的压缩方法和装置

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018156942A1 (en) * 2017-02-23 2018-08-30 Google Llc Optimizing neural network architectures
CN109344888A (zh) * 2018-09-19 2019-02-15 广东工业大学 一种基于卷积神经网络的图像识别方法、装置及设备
CN109615073B (zh) * 2018-12-03 2021-06-04 郑州云海信息技术有限公司 一种神经网络模型的构建方法、设备以及存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090132449A1 (en) * 2006-05-22 2009-05-21 Fujitsu Limited Neural network learning device, method, and program
CN109284820A (zh) * 2018-10-26 2019-01-29 北京图森未来科技有限公司 一种深度神经网络的结构搜索方法及装置
CN109919304A (zh) * 2019-03-04 2019-06-21 腾讯科技（深圳）有限公司 神经网络搜索方法、装置、可读存储介质和计算机设备
CN109978142A (zh) * 2019-03-29 2019-07-05 腾讯科技（深圳）有限公司 神经网络模型的压缩方法和装置

Also Published As

Publication number Publication date
CN112215332A (zh) 2021-01-12
US20220130142A1 (en) 2022-04-28
CN112215332B (zh) 2024-05-14

Similar Documents

Publication Publication Date Title
WO2020221200A1 (zh) 神经网络的构建方法、图像处理方法及装置
WO2021120719A1 (zh) 神经网络模型更新方法、图像处理方法及装置
WO2021008206A1 (zh) 神经网络结构的搜索方法、图像处理方法和装置
WO2021057056A1 (zh) 神经网络架构搜索方法、图像处理方法、装置和存储介质
WO2020216227A9 (zh) 图像分类方法、数据处理方法和装置
WO2021190451A1 (zh) 训练图像处理模型的方法和装置
US20220092351A1 (en) Image classification method, neural network training method, and apparatus
WO2022042713A1 (zh) 一种用于计算设备的深度学习训练方法和装置
WO2022083536A1 (zh) 一种神经网络构建方法以及装置
WO2022001805A1 (zh) 一种神经网络蒸馏方法及装置
WO2021022521A1 (zh) 数据处理的方法、训练神经网络模型的方法及设备
WO2021218517A1 (zh) 获取神经网络模型的方法、图像处理方法及装置
WO2021018163A1 (zh) 神经网络的搜索方法及装置
WO2022052601A1 (zh) 神经网络模型的训练方法、图像处理方法及装置
WO2021164750A1 (zh) 一种卷积层量化方法及其装置
US20220157041A1 (en) Image classification method and apparatus
CN110222718B (zh) 图像处理的方法及装置
WO2022007867A1 (zh) 神经网络的构建方法和装置
WO2023231794A1 (zh) 一种神经网络参数量化方法和装置
WO2021227787A1 (zh) 训练神经网络预测器的方法、图像处理方法及装置
WO2021018251A1 (zh) 图像分类方法及装置
WO2021136058A1 (zh) 一种处理视频的方法及装置
WO2024067884A1 (zh) 一种数据处理方法及相关装置
WO2022156475A1 (zh) 神经网络模型的训练方法、数据处理方法及装置
US20220222934A1 (en) Neural network construction method and apparatus, and image processing method and apparatus

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20840240

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20840240

Country of ref document: EP

Kind code of ref document: A1