WO2021018245A1 - 图像分类方法及装置 - Google Patents

图像分类方法及装置 Download PDF

Info

Publication number
WO2021018245A1
WO2021018245A1 PCT/CN2020/105830 CN2020105830W WO2021018245A1 WO 2021018245 A1 WO2021018245 A1 WO 2021018245A1 CN 2020105830 W CN2020105830 W CN 2020105830W WO 2021018245 A1 WO2021018245 A1 WO 2021018245A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature map
channels
convolution
image
output feature
Prior art date
Application number
PCT/CN2020/105830
Other languages
English (en)
French (fr)
Inventor
韩凯
王云鹤
舒晗
许春景
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP20848554.0A priority Critical patent/EP4006776A4/en
Publication of WO2021018245A1 publication Critical patent/WO2021018245A1/zh
Priority to US17/587,689 priority patent/US20220157041A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
    • G06F18/2193—Validation; Performance evaluation; Active pattern learning techniques based on specific statistical tests
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • This application relates to the field of artificial intelligence, and more specifically, to an image classification method and device.
  • Computer vision is an inseparable part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is about how to use cameras/video cameras and computers to obtain What we need is the knowledge of the data and information of the subject. Vividly speaking, it is to install eyes (camera/camcorder) and brain (algorithm) on the computer to replace the human eye to identify, track and measure the target, so that the computer can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multi-dimensional data.
  • computer vision is to use various imaging systems to replace the visual organs to obtain input information, and then the computer replaces the brain to complete the processing and interpretation of the input information.
  • the ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.
  • Image classification is the basis of various image processing applications. Computer vision often involves the problem of how to classify the acquired images. With the rapid development of artificial intelligence technology, deep learning-based convolutional neural networks (convolutional neural networks, CNN) have been more and more widely used in image classification processing. However, the amount of parameters and calculations included in the convolutional neural network are too large.
  • An image classification method and device which helps to reduce the amount of calculation and parameter amount of image classification processing.
  • an image classification method includes: acquiring an input feature map of an image to be processed; performing convolution processing on the input feature map according to M convolution kernels of a neural network to obtain candidates for M channels Output feature map, M is a positive integer; matrix transformation is performed on the M channels of the candidate output feature map according to N matrices to obtain output feature maps of N channels, where each matrix of the N matrices The number of channels is less than M, N is greater than M, and N is a positive integer; the image to be processed is classified according to the output feature map to obtain a classification result of the image to be processed.
  • the M convolution kernels may be standard convolution kernels in existing convolutional neural networks.
  • the number of channels of the standard convolution kernel is the same as the number of channels of the input feature map.
  • the number of channels of the input feature map of the image to be processed is C
  • the existing convolution The number of channels of the standard convolution kernel in the neural network is also C, that is, the number of channels of the standard convolution kernel is the same as the number of channels of the input feature map, where C is a positive integer.
  • a small number of standard convolution kernels are used to perform convolution processing on the image to be processed to obtain a small number of candidate feature maps, and perform matrix transformation on these small number of candidate feature maps to obtain the The required output feature map.
  • the number of standard convolution kernels is less than the number of standard convolution kernels in the existing convolutional neural network.
  • the number of channels of the matrix used in the matrix transformation is also smaller than that of the standard convolution kernel. Therefore, it helps to reduce the calculation amount and parameter amount of the neural network model, thereby reducing the calculation amount and parameter amount of image classification processing.
  • the number of channels in each of the N matrices may be 1, or the number of channels in each of the N matrices may also be greater than 1.
  • the N matrices include M groups of convolution kernels, and the M groups of convolution kernels respectively correspond to the M channels of the candidate output feature map;
  • the matrix transformation of the M candidate output feature maps according to the N matrices to obtain the output feature maps of the N channels includes: according to each of the M sets of convolution kernels, the candidate The corresponding channels in the M channels of the output feature map are convolved to obtain the output feature maps of the N channels.
  • the corresponding channel among the M channels of the candidate output feature map is output according to each group of the convolution kernels in the M groups of convolution kernels.
  • Performing convolution to obtain the output feature maps of the N channels includes: performing convolution on the corresponding channels in the M channels of the candidate output feature map according to each set of convolution kernels in the M sets of convolution kernels Deep convolution to obtain output feature maps of the N channels.
  • the convolution kernels of each group of the M groups of convolution kernels are the same as the convolution kernels of other groups of the M groups of convolution kernels.
  • the classifying the image to be processed according to the output feature map to obtain a classification result of the image to be processed includes: outputting the candidate The feature map and the output feature map are feature stitched to obtain a feature stitched feature map.
  • the number of channels of the feature stitched feature map is M+N; the image to be processed is classified according to the feature stitched feature map to obtain the Describe the classification result of the image to be processed.
  • the aforementioned feature splicing may mean that the candidate output feature map and the output feature map form a new feature map in the depth direction, that is, the aforementioned feature splicing feature map.
  • the number of channels of the candidate output feature map is M
  • the number of channels of the output feature map is N
  • the candidate output feature map and the output feature map can be feature spliced to obtain a channel number of M+N Feature splicing feature map.
  • Feature splicing can be through identity feature mapping, which can introduce more details (or features) into the output feature map.
  • identity mapping does not introduce additional parameters or calculations, so it can be added without adding parameters. Improve the effect of image classification in the case of the amount of calculation and the amount of calculation.
  • an image classification method includes: obtaining an input feature map of an image to be processed; performing convolution processing on the input feature map according to M first convolution kernels of a neural network to obtain M channels M is a positive integer; matrix transformation is performed on the M first candidate output feature maps according to K first matrices to obtain first output feature maps of K channels, wherein, K The number of channels in each of the first matrices is less than M, K is greater than M, and K is a positive integer; according to the P second convolution kernels of the neural network, the first output feature map is convolved to obtain P The second candidate output feature maps of the channel, and P is a positive integer; matrix transformation is performed on the P second candidate output feature maps according to the N second matrices to obtain the second output feature maps of N channels, where the The number of channels in each of the N second matrices is less than P, N is greater than P, and N is a positive integer; the image to be processed is classified according to the second output feature map
  • the M first convolution kernels may be standard convolution kernels in existing convolutional neural networks.
  • the number of channels of the standard convolution kernel is the same as the number of channels of the input feature map.
  • the number of channels of the input feature map of the image to be processed is C
  • the existing convolution The number of channels of the standard convolution kernel in the neural network is also C, that is, the number of channels of the standard convolution kernel is the same as the number of channels of the input feature map, where C is a positive integer.
  • the P second convolution kernels may also be standard convolution kernels in existing convolutional neural networks.
  • the convolution processing in the existing convolutional neural network is realized through a small number of standard convolution kernels and matrix transformations, which can effectively reduce the redundancy between the output feature maps and reduce the calculation amount of the neural network model. And the amount of parameters, thereby reducing the amount of calculation and the amount of parameters for image classification processing. Therefore, the image classification method in the embodiments of the present application can improve without increasing the amount of parameters and calculations (or reducing the amount of parameters and calculations). The effect of image classification.
  • the number of channels in each of the K first matrices may be 1, or the number of channels in each of the K first matrices may also be greater than 1.
  • the number of channels in each of the N second matrices may be 1, or the number of channels in each of the N second matrices may also be greater than 1.
  • the first output feature map is subjected to convolution processing according to P second convolution kernels of the neural network to obtain second candidate outputs of P channels
  • the feature map includes: performing deep convolution on the first output feature map to obtain a deep convolution feature map; performing convolution processing on the deep convolution feature map according to the P second convolution kernels to obtain the The second candidate output feature map.
  • the performing depth convolution on the first output feature map to obtain a deep convolution feature map includes: performing a step on the first output feature map. Depth convolution with an amplitude greater than 1, to obtain the depth convolution feature map.
  • the classifying the image to be processed according to the second output feature map to obtain a classification result of the image to be processed includes: The input feature map and the second output feature map are residually connected to obtain a residual connection feature map; the image to be processed is classified according to the residual connection feature map to obtain a classification result of the image to be processed.
  • an image classification device including: an acquisition unit for acquiring an input feature map of an image to be processed; a convolution unit for convolving the input feature map according to M convolution kernels of a neural network Product processing to obtain candidate output feature maps of M channels, where M is a positive integer; the matrix transformation unit is used to perform matrix transformation on the M channels of the candidate output feature maps according to N matrices to obtain output features of N channels Figure, wherein the number of channels in each of the N matrices is less than M, N is greater than M, and N is a positive integer; the classification unit is used to classify the image to be processed according to the output feature map to obtain The classification result of the image to be processed.
  • the M convolution kernels may be standard convolution kernels in existing convolutional neural networks.
  • the number of channels of the standard convolution kernel is the same as the number of channels of the input feature map.
  • the number of channels of the input feature map of the image to be processed is C
  • the existing convolution The number of channels of the standard convolution kernel in the neural network is also C, that is, the number of channels of the standard convolution kernel is the same as the number of channels of the input feature map, where C is a positive integer.
  • a small number of standard convolution kernels are used to perform convolution processing on the image to be processed to obtain a small number of candidate feature maps, and perform matrix transformation on these small number of candidate feature maps to obtain the The required output feature map.
  • the number of standard convolution kernels is less than the number of standard convolution kernels in the existing convolutional neural network.
  • the number of channels of the matrix used in the matrix transformation is also smaller than that of the standard convolution kernel. Therefore, it helps to reduce the calculation amount and parameter amount of the neural network model, thereby reducing the calculation amount and parameter amount of image classification processing.
  • the number of channels in each of the N matrices may be 1, or the number of channels in each of the N matrices may also be greater than 1.
  • the N matrices include M sets of convolution kernels, and the M sets of convolution kernels respectively correspond to the M channels of the candidate output feature map;
  • the matrix transformation unit is specifically configured to: according to each group of convolution kernels in the M groups of convolution kernels, convolve the corresponding channels in the M channels of the candidate output feature map to obtain the N channels The output feature map.
  • the matrix transformation unit is specifically configured to: according to each of the M groups of convolution kernels, output a feature map of the candidate Corresponding channels in the M channels are subjected to deep convolution to obtain output feature maps of the N channels.
  • the convolution kernels of each group of the M groups of convolution kernels are the same as the convolution kernels of other groups of the M groups of convolution kernels.
  • the classification unit is specifically configured to: perform feature splicing on the candidate output feature map and the output feature map to obtain a feature splicing feature map, the feature The number of channels of the spliced feature map is M+N; the image to be processed is classified according to the feature spliced feature map to obtain a classification result of the image to be processed.
  • the aforementioned feature splicing may mean that the candidate output feature map and the output feature map form a new feature map in the depth direction, that is, the aforementioned feature splicing feature map.
  • the number of channels of the candidate output feature map is M
  • the number of channels of the output feature map is N
  • the candidate output feature map and the output feature map can be feature spliced to obtain a channel number of M+N Feature splicing feature map.
  • Feature splicing can be through identity feature mapping, which can introduce more details (or features) into the output feature map.
  • identity mapping does not introduce additional parameters or calculations, so it can be added without adding parameters. Improve the effect of image classification in the case of the amount of calculation and the amount of calculation.
  • an image classification device including: an acquisition unit for acquiring an input feature map of an image to be processed; a first convolution unit for verifying the input according to M first convolution checks of a neural network The feature map is subjected to convolution processing to obtain the first candidate output feature maps of M channels, where M is a positive integer; the first matrix transformation unit is used to perform the M first candidate output feature maps according to the K first matrices Matrix transformation is used to obtain first output feature maps of K channels, where the number of channels of each matrix in the K first matrices is less than M, K is greater than M, and K is a positive integer; the second convolution unit uses Performing convolution processing on the first output feature map according to the P second convolution kernels of the neural network to obtain the second candidate output feature maps of P channels, where P is a positive integer; the second matrix transformation unit is used for N second matrices perform matrix transformation on the P second candidate output feature maps to obtain second output feature maps of N channels, wherein the number of channels of each
  • the M first convolution kernels may be standard convolution kernels in existing convolutional neural networks.
  • the number of channels of the standard convolution kernel is the same as the number of channels of the input feature map.
  • the number of channels of the input feature map of the image to be processed is C
  • the existing convolution The number of channels of the standard convolution kernel in the neural network is also C, that is, the number of channels of the standard convolution kernel is the same as the number of channels of the input feature map, where C is a positive integer.
  • the P second convolution kernels may also be standard convolution kernels in existing convolutional neural networks.
  • the convolution processing in the existing convolutional neural network is realized through a small number of standard convolution kernels and matrix transformations, which can effectively reduce the redundancy between the output feature maps and reduce the calculation amount of the neural network model. And the amount of parameters, thereby reducing the amount of calculation and the amount of parameters for image classification processing. Therefore, the image classification method in the embodiments of the present application can improve without increasing the amount of parameters and calculations (or reducing the amount of parameters and calculations). The effect of image classification.
  • the number of channels in each of the K first matrices may be 1, or the number of channels in each of the K first matrices may also be greater than 1.
  • the number of channels in each of the N second matrices may be 1, or the number of channels in each of the N second matrices may also be greater than 1.
  • the image classification device further includes a depth convolution unit, configured to: perform depth convolution on the first output feature map to obtain a depth convolution feature map
  • the second convolution unit is specifically configured to: perform convolution processing on the deep convolution feature map according to the P second convolution kernels to obtain the second candidate output feature map.
  • the depth convolution unit is specifically configured to: perform a depth convolution on the first output feature map with a stride greater than 1 to obtain the depth convolution Feature map.
  • the image classification device further includes a residual connection unit, configured to: perform residual connection between the input feature map and the second output feature map, Obtain a residual connection feature map; the classification unit is specifically configured to classify the image to be processed according to the residual connection feature map to obtain a classification result of the image to be processed.
  • an image classification device in a fifth aspect, includes: a memory for storing a program; a processor for executing the program stored in the memory. When the program stored in the memory is executed, the processing The device is used to execute the method in any one of the foregoing first aspect or second aspect.
  • the processor in the fifth aspect mentioned above can be either a central processing unit (CPU), or a combination of a CPU and a neural network computing processor.
  • the neural network computing processor here can include a graphics processing unit (graphics processing unit). unit, GPU), neural-network processing unit (NPU), tensor processing unit (TPU), and so on.
  • graphics processing unit graphics processing unit
  • NPU neural-network processing unit
  • TPU tensor processing unit
  • TPU is an artificial intelligence accelerator application specific integrated circuit fully customized by Google for machine learning.
  • a computer-readable medium stores program code for device execution, and the program code includes a method for executing the method in any one of the first aspect or the second aspect .
  • a computer program product containing instructions is provided.
  • the computer program product runs on a computer, the computer executes the method in any one of the foregoing first aspect or second aspect.
  • a chip in an eighth aspect, includes a processor and a data interface.
  • the processor reads instructions stored in a memory through the data interface, and executes any one of the first aspect or the second aspect. The method in the implementation mode.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute the instructions stored in the memory.
  • the processor is configured to execute the method in any one of the implementation manners of the first aspect or the second aspect.
  • the aforementioned chip may specifically be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).
  • FPGA field-programmable gate array
  • ASIC application-specific integrated circuit
  • an electronic device in a ninth aspect, includes the image classification device in any one of the foregoing third aspect or fourth aspect.
  • the electronic device may specifically be a terminal device or a server.
  • a small number of standard convolution kernels are used to perform convolution processing on the image to be processed to obtain a small number of candidate feature maps, and perform matrix transformation on these small number of candidate feature maps to obtain the The required output feature map.
  • the number of standard convolution kernels is less than the number of standard convolution kernels in the existing convolutional neural network.
  • the number of channels of the matrix used in the matrix transformation is also smaller than that of the standard convolution kernel. Therefore, it helps to reduce the calculation amount and parameter amount of the neural network model, thereby reducing the calculation amount and parameter amount of image classification processing.
  • Fig. 1 is a schematic structural diagram of a system architecture provided by an embodiment of the present application.
  • Fig. 2 is a schematic diagram of image classification according to a convolutional neural network model provided by an embodiment of the present application.
  • Fig. 3 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application.
  • Fig. 4 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • Fig. 5 is a schematic flowchart of an image classification method provided by an embodiment of the present application.
  • Fig. 6 is a schematic block diagram of convolution processing provided by an embodiment of the present application.
  • Fig. 7 is a schematic block diagram of a feature amplification unit provided by an embodiment of the present application.
  • FIG. 8 is a schematic flowchart of an image classification method provided by another embodiment of the present application.
  • Fig. 9 is a schematic block diagram of a spindle module provided by an embodiment of the present application.
  • Fig. 10 is a schematic block diagram of a spindle module provided by another embodiment of the present application.
  • Fig. 11 is a schematic block diagram of a neural network provided by an embodiment of the present application.
  • FIG. 12 is a schematic diagram of the hardware structure of an image classification device according to an embodiment of the present application.
  • FIG. 13 is a schematic diagram of the hardware structure of a neural network training device according to an embodiment of the present application.
  • the image classification method provided by the embodiments of the present application can be applied to image retrieval, album management, safe city, human-computer interaction, and other scenes that require image classification or image recognition.
  • the images in the embodiments of this application may be static images (or called static pictures) or dynamic images (or called dynamic pictures).
  • the images in this application may be videos or dynamic pictures, or The images in can also be static pictures or photos.
  • static images or dynamic images are collectively referred to as images.
  • the image classification method of the embodiment of the present application can be specifically applied to album classification and photo recognition scenes, and these two scenes are described in detail below.
  • the image classification method of the embodiment of the present application can facilitate the user to classify and manage different object categories, thereby facilitating the user's search, saving the user's management time, and improving the efficiency of album management.
  • the neural network provided in this application can be used to first extract the picture features of the pictures in the album, and then perform the analysis on the pictures in the album according to the extracted picture features. Categorize to obtain the classification result of the picture, and then classify the pictures in the album according to the classification result of the picture, and obtain the album arranged according to the picture category.
  • pictures belonging to the same category may be arranged in a row or a row. For example, in the final album, the pictures in the first row belong to airplanes, and the pictures in the second row belong to cars.
  • the user can use the image classification method of the embodiment of the present application to process the captured photo, and can automatically recognize the category of the object being photographed, for example, it can automatically recognize that the object being photographed is a flower, an animal, etc.
  • the image classification method of the embodiment of the application can be used to identify the object obtained by taking a photo, and the category to which the object belongs can be identified.
  • the photo obtained by the user includes a shared bicycle, and the image classification method of the embodiment of the application is used
  • the shared bicycle can be recognized, and the object is recognized as a bicycle, and further, bicycle related information can be displayed.
  • album classification and photo identification described above are only two specific scenarios applied by the image classification method in the embodiment of this application, and the image classification method in the embodiment of this application is not limited to the above two scenarios when applied.
  • the image classification method of the application embodiment can be applied to any scene that requires image classification or image recognition.
  • the image classification method in the embodiments of this application uses a new neural network model, which can also be similarly applied to other fields using neural networks, such as face recognition, speech recognition, target detection, machine translation, and Semantic segmentation, etc.
  • the embodiments of this application involve a large number of related applications of neural networks.
  • a neural network can be composed of neural units, which can refer to an arithmetic unit that takes x s and intercept 1 as inputs.
  • the output of the arithmetic unit can be as shown in formula (1-1):
  • s 1, 2,...n, n is a natural number greater than 1
  • W s is the weight of x s
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field.
  • the local receptive field can be a region composed of several neural units.
  • Deep neural network also known as multi-layer neural network
  • DNN can be understood as a neural network with multiple hidden layers.
  • DNN is divided according to the positions of different layers.
  • the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer.
  • the first layer is the input layer
  • the last layer is the output layer
  • the number of layers in the middle are all hidden layers.
  • the layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer.
  • DNN looks complicated, it is not complicated in terms of the work of each layer. In simple terms, it is the following linear relationship expression: among them, Is the input vector, Is the output vector, Is the offset vector, W is the weight matrix (also called coefficient), and ⁇ () is the activation function.
  • Each layer is just the input vector After such a simple operation, the output vector is obtained Due to the large number of DNN layers, the coefficient W and the offset vector The number is also relatively large.
  • the definition of these parameters in the DNN is as follows: Take the coefficient W as an example: Suppose that in a three-layer DNN, the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.
  • the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as
  • the input layer has no W parameter.
  • more hidden layers make the network more capable of portraying complex situations in the real world. Theoretically speaking, a model with more parameters is more complex and has a greater "capacity", which means it can complete more complex learning tasks.
  • Training a deep neural network is also a process of learning a weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by vectors W of many layers).
  • Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with convolutional structure.
  • the convolutional neural network contains a feature extractor composed of a convolution layer and a sub-sampling layer.
  • the feature extractor can be regarded as a filter.
  • the convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network.
  • a neuron can be connected to only part of the neighboring neurons.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels.
  • Sharing weight can be understood as the way to extract image information has nothing to do with location.
  • the convolution kernel can be initialized in the form of a matrix of random size. During the training of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.
  • RNN Recurrent Neural Networks
  • RNN can process sequence data of any length.
  • the training of RNN is the same as the training of traditional CNN or DNN.
  • the neural network can use an error back propagation (BP) algorithm to modify the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forwarding the input signal to the output will cause error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss is converged.
  • the backpropagation algorithm is a backpropagation motion dominated by error loss, and aims to obtain the optimal neural network model parameters, such as the weight matrix.
  • the pixel value of the image can be a red-green-blue (RGB) color value, and the pixel value can be a long integer representing the color.
  • the pixel value is 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. In each color component, the smaller the value, the lower the brightness, and the larger the value, the higher the brightness.
  • the pixel values can be grayscale values.
  • an embodiment of the present application provides a system architecture 100.
  • a data collection device 160 is used to collect training data.
  • the training data may include training images and classification results corresponding to the training images, wherein the classification results of the training images may be manually pre-labeled results.
  • the data collection device 160 stores the training data in the database 130, and the training device 120 trains to obtain the target model/rule 101 based on the training data maintained in the database 130.
  • the training device 120 processes the input original image and compares the output image with the original image until the output image of the training device 120 differs from the original image. The difference is less than a certain threshold, thereby completing the training of the target model/rule 101.
  • the above-mentioned target model/rule 101 can be used to implement the image classification method of the embodiment of the present application, that is, the image to be processed is input into the target model/rule 101 after relevant preprocessing to obtain the classification result of the image.
  • the target model/rule 101 in the embodiment of the present application may specifically be a neural network.
  • the training data maintained in the database 130 may not all come from the collection of the data collection device 160, and may also be received from other devices.
  • the training device 120 does not necessarily perform the training of the target model/rule 101 completely based on the training data maintained by the database 130. It may also obtain training data from the cloud or other places for model training.
  • the above description should not be used as a reference to this application. Limitations of Examples.
  • the target model/rule 101 trained according to the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG. 1.
  • the execution device 110 may be a terminal, such as a mobile phone terminal, a tablet computer, Notebook computers, augmented reality (AR)/virtual reality (VR), vehicle-mounted terminals, etc., can also be servers or cloud devices.
  • the execution device 110 is configured with an input/output (input/output, I/O) interface 112 for data interaction with external devices.
  • the user can input data to the I/O interface 112 through the client device 140.
  • the input data in this embodiment of the application may include: the image to be processed input by the client device.
  • the preprocessing module 113 and the preprocessing module 114 are used for preprocessing according to the input data (such as the image to be processed) received by the I/O interface 112.
  • the preprocessing module 113 and the preprocessing module may not be provided 114 (there may only be one preprocessing module), and the calculation module 111 is directly used to process the input data.
  • the execution device 110 may call data, codes, etc. in the data storage system 150 for corresponding processing .
  • the data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 150.
  • the I/O interface 112 returns the processing result, such as the classification result of the to-be-processed image obtained as described above, to the client device 140 to provide it to the user.
  • the training device 120 can generate corresponding target models/rules 101 based on different training data for different goals or tasks, and the corresponding target models/rules 101 can be used to achieve the above goals or complete The above tasks provide the user with the desired result.
  • the user can manually set input data, and the manual setting can be operated through the interface provided by the I/O interface 112.
  • the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send the input data and the user's authorization is required, the user can set the corresponding authority in the client device 140.
  • the user can view the result output by the execution device 110 on the client device 140, and the specific presentation form may be a specific manner such as display, sound, and action.
  • the client device 140 can also be used as a data collection terminal to collect the input data of the input I/O interface 112 and the output result of the output I/O interface 112 as new sample data, and store it in the database 130 as shown in the figure.
  • the I/O interface 112 directly uses the input data input to the I/O interface 112 and the output result of the output I/O interface 112 as a new sample as shown in the figure.
  • the data is stored in the database 130.
  • Fig. 1 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data The storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.
  • the target model/rule 101 is obtained by training according to the training device 120, and the target model/rule 101 may be the neural network in this application in the embodiment of this application.
  • the neural network provided in the embodiment of this application Can be CNN, deep convolutional neural networks (deep convolutional neural networks, DCNN), recurrent neural networks (recurrent neural network, RNNS) and so on.
  • CNN is a very common neural network
  • the structure of CNN will be introduced in detail below in conjunction with Figure 2.
  • a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture.
  • a deep learning architecture refers to a machine learning algorithm. Multi-level learning is carried out on the abstract level of
  • CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the input image.
  • a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer/pooling layer 220 (the pooling layer is optional), and a neural network layer 230.
  • CNN convolutional neural network
  • the convolutional layer/pooling layer 220 as shown in FIG. 2 may include layers 221-226 as shown in Examples.
  • layer 221 is a convolutional layer
  • layer 222 is a pooling layer
  • layer 223 is Convolutional layer
  • 224 is a pooling layer
  • 225 is a convolutional layer
  • 226 is a pooling layer
  • 221 and 222 are convolutional layers
  • 223 is a pooling layer
  • 224 and 225 are convolutions.
  • the accumulation layer, 226 is the pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.
  • the convolution layer 221 can include many convolution operators.
  • the convolution operator is also called a kernel. Its function in image processing is equivalent to a filter that extracts specific information from the input image matrix.
  • the convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. In the process of convolution on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. ...It depends on the value of stride) to complete the work of extracting specific features from the image.
  • the size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix and the depth dimension of the input image are the same.
  • the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension convolution output, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row ⁇ column) are applied. That is, multiple homogeneous matrices.
  • the output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" mentioned above.
  • Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to eliminate unwanted noise in the image.
  • the multiple weight matrices have the same size (row ⁇ column), and the feature maps extracted by the multiple weight matrices of the same size have the same size, and then the multiple extracted feature maps of the same size are combined to form a convolution operation. Output.
  • weight values in these weight matrices need to be obtained through a lot of training in practical applications.
  • Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions. .
  • the initial convolutional layer (such as 221) often extracts more general features, which can also be called low-level features; with the convolutional neural network
  • the features extracted by the subsequent convolutional layers (for example, 226) become more and more complex, such as high-level semantic features, and features with higher semantics are more suitable for the problem to be solved.
  • the pooling layer can be a convolutional layer followed by a layer
  • the pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers.
  • the pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain a smaller size image.
  • the average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling.
  • the maximum pooling operator can take the pixel with the largest value within a specific range as the result of the maximum pooling.
  • the operators in the pooling layer should also be related to the image size.
  • the size of the image output after processing by the pooling layer can be smaller than the size of the image of the input pooling layer, and each pixel in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.
  • the convolutional neural network 200 After processing by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 200 needs to use the neural network layer 230 to generate one or a group of required classes of output. Therefore, the neural network layer 230 may include multiple hidden layers (231, 232 to 23n as shown in FIG. 2) and an output layer 240. The parameters contained in the multiple hidden layers can be based on specific task types. The relevant training data of the, for example, the task type can include image recognition, image classification, image super-resolution reconstruction and so on.
  • the output layer 240 After the multiple hidden layers in the neural network layer 230, that is, the final layer of the entire convolutional neural network 200 is the output layer 240.
  • the output layer 240 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error.
  • the convolutional neural network 200 shown in FIG. 2 is only used as an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.
  • the convolutional neural network 200 shown in FIG. 2 may be used to process the image to be processed to obtain the classification result of the image to be processed.
  • the image to be processed is processed by the input layer 210, the convolutional layer/pooling layer 220, and the neural network layer 230 to output the classification result of the image to be processed.
  • FIG. 3 is a chip hardware structure provided by an embodiment of the application, and the chip includes a neural network processor 50.
  • the chip may be set in the execution device 110 as shown in FIG. 1 to complete the calculation work of the calculation module 111.
  • the chip can also be set in the training device 120 as shown in FIG. 1 to complete the training work of the training device 120 and output the target model/rule 101.
  • the algorithms of each layer in the convolutional neural network as shown in Figure 2 can be implemented in the chip as shown in Figure 3.
  • the neural network processor NPU 50 is mounted on the host CPU (host CPU) as a coprocessor, and the host CPU distributes tasks.
  • the core part of the NPU is the arithmetic circuit 503.
  • the controller 504 controls the arithmetic circuit 503 to extract data from the memory (weight memory or input memory) and perform calculations.
  • the arithmetic circuit 503 includes multiple processing units (process engines, PE). In some implementations, the arithmetic circuit 503 is a two-dimensional systolic array. The arithmetic circuit 503 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.
  • the arithmetic circuit 503 fetches the data corresponding to matrix B from the weight memory 502 and caches it on each PE in the arithmetic circuit 503.
  • the arithmetic circuit 503 takes the matrix A data and the matrix B from the input memory 501 to perform matrix operations, and the partial result or final result of the obtained matrix is stored in an accumulator 508.
  • the vector calculation unit 507 may perform further processing on the output of the arithmetic circuit 503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and so on.
  • the vector calculation unit 507 can be used for network calculations in the non-convolutional/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .
  • the vector calculation unit 507 can store the processed output vector to the unified buffer 506.
  • the vector calculation unit 507 may apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate the activation value.
  • the vector calculation unit 507 generates a normalized value, a combined value, or both.
  • the processed output vector can be used as an activation input to the arithmetic circuit 503, for example for use in a subsequent layer in a neural network.
  • the unified memory 506 is used to store input data and output data.
  • the weight data directly transfers the input data in the external memory to the input memory 501 and/or the unified memory 506 through the storage unit access controller 505 (direct memory access controller, DMAC), and stores the weight data in the external memory into the weight memory 502, And the data in the unified memory 506 is stored in the external memory.
  • DMAC direct memory access controller
  • the bus interface unit (BIU) 510 is used to implement interaction between the main CPU, the DMAC, and the fetch memory 509 through the bus.
  • An instruction fetch buffer 509 connected to the controller 504 is used to store instructions used by the controller 504;
  • the controller 504 is configured to call the instructions cached in the memory 509 to control the working process of the computing accelerator.
  • the unified memory 506, the input memory 501, the weight memory 502, and the instruction fetch memory 509 are all on-chip (On-Chip) memories, and the external memory is a memory external to the NPU.
  • the external memory can be a double data rate synchronous dynamic random access memory. Memory (double data rate synchronous dynamic random access memory, referred to as DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.
  • DDR SDRAM double data rate synchronous dynamic random access memory
  • HBM high bandwidth memory
  • each layer in the convolutional neural network shown in FIG. 2 can be executed by the arithmetic circuit 503 or the vector calculation unit 307.
  • the execution device 110 in FIG. 1 introduced above can execute each step of the image classification method of the embodiment of the present application.
  • the CNN model shown in FIG. 2 and the chip shown in FIG. 3 can also be used to execute the image of the embodiment of the present application.
  • the various steps of the classification method. The image classification method of the embodiment of the present application will be described in detail below with reference to the accompanying drawings.
  • the image classification method provided in the embodiments of the present application can be executed on a server, can also be executed on the cloud, or can be executed on a terminal device.
  • a terminal device as an example, as shown in FIG. 4, the technical solution of the embodiment of the present invention can be applied to a terminal device.
  • the image classification method in the embodiment of the present application can classify an input image to obtain a classification result of the input image.
  • the terminal device may be mobile or fixed.
  • the terminal device may be a mobile phone with image processing function, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC). ), personal digital assistant (PDA), personal computer (PC), camera, video camera, smart watch, wearable device (WD) or self-driving vehicle, etc., embodiments of the present invention There is no restriction on this.
  • the classification of images is the basis of various image processing applications, and computer vision often involves the problem of how to classify the acquired images.
  • the high-precision convolutional neural network has a large amount of parameters and calculations, and the memory and computing resources of the terminal equipment are very limited, and do not have strong computing power and caching capabilities, resulting in high-precision volumes
  • Product neural networks are difficult to deploy on terminal equipment.
  • the embodiment of the application proposes an image classification method.
  • the required output feature map can be obtained through a small number of standard convolution kernels less than the number of standard convolution kernels in the existing convolutional neural network. This method is helpful In order to reduce the amount of calculation and parameters of image classification processing.
  • FIG. 5 shows a schematic flowchart of an image classification method 500 provided by an embodiment of the present application.
  • the method may be executed by a device capable of image classification.
  • the method may be executed by the terminal device in FIG. 4.
  • the image to be processed may be an image captured by the terminal device through a camera, or the image to be processed may also be an image obtained from inside the terminal device (for example, , The image stored in the album of the terminal device, or the image obtained by the terminal device from the cloud).
  • the input feature map of the image to be processed may be a feature map obtained after processing by other layers in the convolutional neural network.
  • the other layer in the convolutional neural network mentioned here refers to a layer in the convolutional neural network.
  • the other layer can be the input layer, convolutional layer, pooling layer or One of the fully connected layers.
  • S520 Perform convolution processing on the input feature map according to the M convolution kernels of the neural network to obtain candidate output feature maps of M channels, where M is a positive integer.
  • the M convolution kernels may be standard convolution kernels in existing convolutional neural networks.
  • the number of channels of the standard convolution kernel is the same as the number of channels of the input feature map.
  • the number of channels of the input feature map of the image to be processed is C
  • the existing convolutional neural is also C, that is, the number of channels of the standard convolution kernel is the same as the number of channels of the input feature map, where C is a positive integer.
  • S530 Perform matrix transformation on the M channels of the candidate output feature map according to the N matrices to obtain output feature maps of the N channels.
  • the number of channels of each of the N matrices is less than M, N is greater than M, and N is a positive integer.
  • S520 and S530 are the feature in feature (FiF) in the embodiment of the present application, and the feature in feature can be used to replace the convolutional layer in the existing convolutional neural network model.
  • the feature amplification unit uses a small number of standard convolution kernels for convolution processing to obtain a small number of candidate feature maps, and performs matrix transformation on these small number of candidate feature maps to obtain the required output feature maps, which can reduce the number of output feature maps. The redundancy of this helps reduce the amount of calculations and parameters for image classification processing.
  • the feature amplification unit in S520 and S530 will be described in detail below in conjunction with FIG. 6 and FIG. 7.
  • Figure 6 shows the convolution processing of a convolutional layer in the existing convolutional neural network.
  • the input feature map of the convolutional layer includes C channels
  • the output feature map of the convolutional layer includes N channels.
  • N standard convolution kernels are required in the convolutional layer.
  • Each convolution kernel in the standard convolution kernel includes C channels.
  • the input feature maps of the above C channels may refer to one input feature map, and the number of channels of the input feature map is C; or the input feature maps of the above C channels may also refer to C input feature maps.
  • each input feature map is two-dimensional (that is, the number of channels is 1).
  • this application is uniformly described as the input feature maps of C channels.
  • the descriptions of other feature maps in the embodiments of the present application can be understood similarly, and will not be repeated here.
  • candidate output feature maps of M channels are obtained.
  • the candidate output feature maps of M channels can be considered as a candidate output feature map including M channels, or the number of M channels is 1.
  • FIG. 7 shows the feature amplification processing of the feature amplification unit in the embodiment of the present application.
  • the feature amplification unit can be used to replace the convolutional layer in the existing convolutional neural network (for example, the convolutional layer shown in FIG. 6).
  • the convolution processing can be performed through M standard convolution kernels to obtain Candidate feature maps of M channels (such as S520 above), each of the M standard convolution kernels includes C channels; then N matrices are used to perform matrix transformation on the M channels of the candidate output feature map, Obtain the output feature maps of N channels (such as S530 above); wherein, the number of channels of each matrix in the N matrices can be less than M, N can be greater than M, and N is a positive integer.
  • the feature amplification unit performs convolution processing based on a small number of standard convolution kernels, and performs matrix variables on each channel of the obtained small number of candidate feature maps, which can effectively reduce the redundancy between output feature maps. .
  • the number of standard convolution kernels in the feature amplification unit is less than that of the existing convolutional neural network, and the number of channels of the matrix used in the matrix transformation is also less than that of the standard convolution kernel, so it helps to reduce the neural network model The calculation amount and parameter amount of the image classification process can be reduced.
  • the N matrices in the above matrix transformation may be N matrices with a channel number of 1, or the N matrices in the above matrix transformation may also be matrices with N channels greater than 1, for example , The number of channels of the N matrices is 2.
  • the feature maps are matrix-transformed according to N matrices, and usually the convolution kernel can also be considered as a matrix. Therefore, the N matrices can also be considered as N convolution kernels.
  • the N matrices can be understood as the N convolution kernels, and the N convolution kernels can also refer to the matrix transformation of the candidate output feature maps of the M channels. The N matrices.
  • the convolution kernels in S520 are standard convolution kernels in existing convolutional neural networks.
  • the number of channels of the standard convolution kernel is the same as the number of channels of the input feature map for convolution, and the number of channels of each convolution kernel in the above N convolution kernels in this application can be less than the number of input features for convolution
  • the number of channels of the graph is M (that is, the candidate output feature graphs of the M channels).
  • the number of channels of the N convolution kernels may be 1.
  • the calculation amount of the feature amplification unit can be reduced.
  • the amount of parameters when the feature amplification unit is used to replace the convolutional layer in the existing convolutional neural network (for example, the convolutional layer shown in Figure 6), it helps to reduce the amount of calculation and the amount of parameters of the neural network model , Thereby reducing the amount of calculation and parameters of image classification processing.
  • the number of channels of the N matrices in the foregoing matrix transformation may be 1.
  • the N matrices may be divided into M groups of convolution kernels, and the M groups of convolution kernels may respectively correspond to the M channels of the candidate output feature map.
  • the M groups of convolution kernels may have a one-to-one correspondence with the M channels of the candidate output feature map.
  • the first group of the M groups of convolution kernels may correspond to the first channel of the M channels of the candidate output feature map
  • the second group of the M groups of convolution kernels may correspond to the The second channel of the M channels of the candidate output feature map corresponds to, ...
  • the M-th group in the M groups of convolution kernels may correspond to the M-th channel of the M channels of the candidate output feature map .
  • performing matrix transformation on the M candidate output feature maps according to N matrices to obtain output feature maps of N channels may include: according to each set of convolution kernels in the M sets of convolution kernels.
  • the product kernel is used to convolve the corresponding channels among the M channels of the candidate output feature map to obtain the output feature maps of the N channels.
  • the first group of the M groups of convolution kernels can be used to convolve the first channel of the M channels of the candidate output feature map, ..., the M groups of convolution can be used
  • the M-th group in the kernel performs convolution on the M-th channel among the M channels of the candidate output feature map.
  • the first group of the M groups of convolution kernels includes S convolution kernels (the number of channels of the convolution kernel is 1), as shown in Fig. 7 ⁇ 1,1 to ⁇ 1,s , then
  • S convolution kernels can be used to convolve the first channel of the M channels of the candidate output feature map to obtain S output feature maps (or it can also be considered as an output feature with S channels. Figure).
  • the number of floating point operations (FLOPs) r S of the feature amplification unit can be approximated by the following formula ( 1) means:
  • S is the number of convolutions included in each group of the M groups of convolution kernels
  • C is the number of channels of the input feature map of the feature amplification unit.
  • the compression ratio r C of the parameter quantity of the feature amplification unit can be approximately expressed by the following formula (3):
  • S is the number of convolutions included in each group of the M groups of convolution kernels
  • C is the number of channels of the input feature map of the feature amplification unit.
  • Outputting the feature map may include: performing depthwise convolution on the corresponding channels in the M channels of the candidate output feature map according to each set of convolution kernels in the M sets of convolution kernels, to obtain the result The output characteristic diagram of the N channels.
  • the depth convolution can refer to the prior art, which will not be repeated here.
  • the convolution kernels of each group of the M groups of convolution kernels are the same as the convolution kernels of other groups of the M groups of convolution kernels.
  • the M channels of the candidate output feature map may be multiplexed with the same convolution kernel for convolution.
  • the first group of the M groups of convolution kernels includes S convolution kernels, such as ⁇ 1,1 to ⁇ 1,s as shown in FIG. 7. These S convolution kernels can be used in sequence, and the The first channel of the M channels of the candidate output feature map is convolved; the S convolution kernels included in the second group of the M groups of convolution kernels can also be ⁇ 1,1 to ⁇ 1,1 to shown in FIG. 7 ⁇ 1,s , the S convolution kernels can be used in sequence to perform convolution on the second channel of the M channels of the candidate output feature map.
  • the S convolution kernels included in other groups in the M groups of convolution kernels may also be ⁇ 1,1 to ⁇ 1,s shown in FIG. 7, which will not be repeated here.
  • the calculation amount of the feature amplification unit can be reduced.
  • the amount of parameters reduces the amount of calculations and parameters for image classification processing.
  • the number of channels of the N matrices in the foregoing matrix transformation may be greater than one.
  • the N matrices can be divided into M/2 groups of convolution kernels, and the M/2 groups of convolution kernels can be respectively compared with the candidate Two of the M channels of the output characteristic map correspond one-to-one.
  • the number of channels of the N convolution kernels is 2, if the input feature maps (that is, the candidate output feature maps of the M channels) are to be compared according to the N convolution kernels ) Performs matrix transformation, the number of channels of each of the N convolution kernels needs to be the same (or the same) as the number of channels of the input feature map.
  • the candidate output feature maps of the M channels are equivalent to being divided into M/2 feature maps with a number of channels of 2, and are convolved with the N convolution kernels. Accordingly, the N convolutions
  • the kernel is also divided into M/2 groups of convolution kernels. It should be understood that the “division” mentioned here is only an explanatory description for easy understanding, and there may be no division operation in practice.
  • the first group of the M/2 group of convolution kernels may correspond to the first channel and the second channel of the M channels of the candidate output feature map
  • the M/2 group of convolution kernels The second group in can correspond to the third channel and the fourth channel in the M channels of the candidate output feature map
  • the M/2th group in the M/2 group of convolution kernels can be Corresponding to the M-1th channel and the Mth channel among the M channels of the candidate output feature map.
  • the corresponding channels in the M channels of the candidate output feature map may be convolved to obtain the N channels Output feature map.
  • the first group of the M/2 groups of convolution kernels can be used to convolve the first channel and the second channel of the M channels of the candidate output feature map,..., you can use
  • the M/2th group in the M/2 group of convolution kernels performs convolution on the M-1th channel and the Mth channel among the M channels of the candidate output feature map.
  • the convolution kernels of each group in the M/2 group of convolution kernels are the same as the convolution kernels of other groups in the M/2 group of convolution kernels.
  • the M channels of the candidate output feature map may be multiplexed with the same convolution kernel for convolution.
  • the case that the number of channels of the N matrices in the matrix transformation is greater than 2 is similar to the embodiment in which the number of channels of the N matrices in the matrix transformation is equal to 2, and will not be repeated here.
  • the calculation amount and the parameter amount of the feature amplification unit can be reduced, Reduce the amount of calculations and parameters for image classification processing.
  • the number of channels of the N matrices (that is, the aforementioned N convolution kernels) in the matrix transformation may be equal to the number of channels M of the candidate output feature map.
  • the classifying the image to be processed according to the output feature map to obtain the classification result of the image to be processed may include: characterizing the candidate output feature map and the output feature map Splicing to obtain a feature stitching feature map, the number of channels of the feature stitching feature map is M+N; classifying the image to be processed according to the feature stitching feature map to obtain a classification result of the image to be processed.
  • the feature splicing mentioned above means that the candidate output feature map and the output feature map form a new feature map in the depth direction, that is, the feature splicing feature map described above.
  • the number of channels of the candidate output feature map is M
  • the number of channels of the output feature map is N
  • the candidate output feature map and the output feature map can be feature spliced to obtain a channel number of M+N Feature splicing feature map.
  • Feature splicing can be through identity feature mapping, which can introduce more details (or features) into the output feature map.
  • identity mapping does not introduce additional parameters or calculations, so it can be added without adding parameters. Improve the effect of image classification in the case of the amount of calculation and the amount of calculation.
  • S540 Classify the image to be processed according to the output feature map to obtain a classification result of the image to be processed.
  • FIG. 8 shows a schematic flowchart of an image classification method 800 provided by another embodiment of the present application.
  • the method may be executed by an image classification apparatus, for example, the method may be executed by the terminal device in FIG. 4.
  • the image to be processed may be an image captured by the terminal device through a camera, or the image to be processed may also be an image obtained from inside the terminal device (for example, , The image stored in the album of the terminal device, or the image obtained by the terminal device from the cloud).
  • the input feature map of the image to be processed may be a feature map obtained after processing by other layers in the convolutional neural network.
  • the other layer in the convolutional neural network mentioned here refers to a layer in the convolutional neural network.
  • the other layer can be the input layer, convolutional layer, pooling layer or One of the fully connected layers.
  • S820 Perform convolution processing on the input feature map according to the M first convolution kernels of the neural network to obtain first candidate output feature maps of M channels, where M is a positive integer.
  • the M first convolution kernels may be standard convolution kernels in existing convolutional neural networks.
  • S830 Perform matrix transformation on the M first candidate output feature maps according to the K first matrices to obtain first output feature maps of K channels.
  • the number of channels of each of the K first matrices is less than M, K is greater than M, and K is a positive integer.
  • the foregoing S820 and S830 may be a feature in feature (FiF) in the method 500 in FIG. 5.
  • F feature in feature
  • S820 and S830 may also be referred to as the first feature amplification unit in the following embodiments.
  • S840 Perform convolution processing on the first output feature map according to P second convolution kernels of the neural network to obtain second candidate output feature maps of P channels, where P is a positive integer.
  • the P second convolution kernels may be standard convolution kernels in existing convolutional neural networks.
  • the performing convolution processing on the first output feature map according to the P second convolution kernels of the neural network to obtain the second candidate output feature maps of the P channels may include: Perform deep convolution on the feature map to obtain a deep convolution feature map; perform convolution processing on the deep convolution feature map according to the P second convolution kernels to obtain the second candidate output feature map.
  • the performing depth convolution on the first output feature map to obtain a depth convolution feature map may include: performing a depth convolution on the first output feature map with a step greater than 1 to obtain the Deep convolution feature map.
  • S850 Perform matrix transformation on the P second candidate output feature maps according to the N second matrices to obtain second output feature maps of N channels.
  • the number of channels of each of the N second matrices is less than P, N is greater than P, and N is a positive integer.
  • the foregoing S840 and S850 may also be a feature in feature (FiF) in the method 500 in FIG. 5.
  • F feature in feature
  • S840 and S850 may also be referred to as second feature amplification units in the following embodiments.
  • S860 Classify the image to be processed according to the second output feature map to obtain a classification result of the image to be processed.
  • the classifying the image to be processed according to the second output feature map to obtain the classification result of the image to be processed may include: comparing the input feature map and the second output feature map Perform residual connection to obtain a residual connection feature map; classify the image to be processed according to the residual connection feature map to obtain a classification result of the image to be processed.
  • the above S820 to S850 may be the spindle module (spindle block) in the embodiment of this application, and the spindle module may be used to replace a block in the existing convolutional neural network model, for example, the existing A module in the convolutional neural network model can include two convolutional layers.
  • spindle modules in the following embodiments all refer to the spindle modules in the method 800 (S820 to S850) in FIG. 8 described above.
  • the spindle module may include the first feature amplification unit and the second feature amplification unit.
  • the spindle module can be composed of at least two feature amplification units (such as the feature amplification unit shown in FIG. 5), which uses a small number of standard convolution kernels to perform convolution processing to obtain a small number of candidate feature maps, and Performing matrix transformation on these few candidate feature maps to obtain the required output feature maps can reduce the redundancy between the output feature maps and help reduce the amount of calculations and parameters for image classification processing.
  • the spindle modules in S820 to S850 will be described in detail below in conjunction with FIG. 9 and FIG. 10.
  • Figure 9 shows the spindle module with a step length of 1 in the embodiment of the present application.
  • the spindle may be composed of at least two feature amplification units (such as the feature amplification unit shown in FIG. 5).
  • the size (width and height) of the input feature map is the same as the size of the output feature map.
  • the input feature map of the spindle module is the input feature map of the image to be processed
  • the output feature map of the spindle module is the second output feature map of N channels, if the size of the input feature map is A*B , The size of the second output feature map is also A*B.
  • deep convolution may also be performed between the first feature amplification unit and the second feature amplification unit.
  • the step size of the depth convolution may be 1.
  • the spindle module Perform residual connection, that is, perform residual connection between the input feature map and the second output feature map.
  • the input feature map and the second output feature map can be residually connected to obtain a residual connection feature map; accordingly, the image to be processed can be classified according to the residual connection feature map , To obtain the classification result of the image to be processed.
  • Figure 10 shows a spindle module with a step length greater than 1 in the embodiment of the present application.
  • the spindle may be composed of at least two feature amplification units (such as the feature amplification unit shown in FIG. 5).
  • the size (width and height) of the output feature map is smaller than the size of the input feature map.
  • the size (width and height) of the output feature map is half of the size of the input feature map.
  • the input feature map of the spindle module is the input feature map of the image to be processed
  • the output feature map of the spindle module is the second output feature map of N channels, if the size of the input feature map is A*B , The size of the second output feature map is also (A/2)*(B/2).
  • deep convolution may also be performed between the first feature amplification unit and the second feature amplification unit.
  • the step size of the depth convolution may be greater than one.
  • the number of channels of the output characteristic map of the spindle module may be N (that is, the second output characteristic map of N channels), and the number of channels of the output characteristic map of the first characteristic amplification unit may be Is K (that is, the first output characteristic map of K channels), the number of channels K of the output of the first characteristic amplification unit can be greater than N, and accordingly, the number of channels is reduced by the second characteristic amplification unit K is reduced to N.
  • the number of channels can be increased by the first feature amplification unit, and then the number of channels can be reduced by the second feature amplification unit to meet the number of channels output by the spindle module.
  • the number of channels of the output feature map of the spindle module is 100 (ie, the second output feature map of 100 channels), and the number of channels of the output feature map of the first feature amplification unit may be 1000 (ie, 1000 channels).
  • the first output characteristic map of the channel) at this time, the number of channels 1000 output by the first characteristic amplification unit is greater than the number of channels 100 output by the spindle module, and accordingly, the second characteristic amplification unit can be passed through Reduce the number of channels from 1000 to 100.
  • the feature amplification unit in the embodiment of the present application can implement the convolution processing in the existing convolutional neural network through a small number of standard convolution kernels and matrix transformations, which can effectively reduce the redundancy between the output feature maps.
  • the calculation amount and the parameter amount of the neural network model are reduced, thereby reducing the calculation amount and the parameter amount of the image classification processing. Therefore, the spindle module in the embodiment of the present application may not increase the parameter amount and the calculation amount (or reduce the parameter amount and the calculation amount) In the case of improved image classification.
  • Fig. 11 is a schematic block diagram of a neural network provided by an embodiment of the present application.
  • the neural network shown in FIG. 11 can be used to implement the image classification method shown in FIG. 8.
  • the neural network in FIG. 11 may include one or more spindle modules in the method 800 of FIG. 8.
  • the spindle module may be used to replace a block in the existing convolutional neural network model, for example, the existing convolutional neural network.
  • a module in the network model can include two convolutional layers.
  • the spindle module may include at least two feature amplification units shown in FIG. 5.
  • the spindle module may include two feature amplification units as described in method 800 in FIG. 8: a first feature amplification unit and a second feature amplification unit.
  • Feature amplification unit a feature augmentation unit can be used to replace a convolutional layer in the existing convolutional neural network model.
  • the neural network shown in FIG. 11 may also include a convolutional layer, a pooling layer, or a fully connected layer, which is not limited in this application.
  • an embodiment of the present application proposes an efficient neural network model HWNet.
  • HWNet includes multiple spindle modules, where each spindle module includes a feature augmentation module.
  • the network structure of HWNet can refer to the design criteria of existing neural networks. For example, in the design of existing neural networks, as the size of the feature map gradually decreases, the number of channels of the feature map gradually increases.
  • the specific structure of HWNet can be shown in Table 1 below.
  • the first layer of HWNet is a convolutional layer with 16 standard convolution kernels, and then there are 12 spindle modules with gradually increasing channels of input feature maps. These spindle module groups are divided into 5 In each stage, the size of the feature map in each stage is the same.
  • Table 2 is the test experiment data of image classification on the ImageNet dataset for HWNet and several existing neural network models.
  • MobileNet is a convolutional neural network model proposed by Google
  • ShuffleNet is a convolutional neural network model designed for mobile terminal equipment proposed by Megvii Technology
  • IGCV3 is an interleaved low-rank grouped convolution .
  • FIG. 12 is a schematic diagram of the hardware structure of an image classification device according to an embodiment of the present application.
  • the image classification device 4000 shown in FIG. 12 includes a memory 4001, a processor 4002, a communication interface 4003, and a bus 4004. Among them, the memory 4001, the processor 4002, and the communication interface 4003 implement communication connections between each other through the bus 4004.
  • the memory 4001 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).
  • the memory 4001 may store a program. When the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 and the communication interface 4003 are used to execute each step of the image classification method of the embodiment of the present application.
  • the processor 4002 may adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more
  • the integrated circuit is used to execute related programs to realize the functions required by the units in the image classification device of the embodiment of the present application, or to execute the image classification method of the method embodiment of the present application.
  • the processor 4002 may also be an integrated circuit chip with signal processing capability.
  • each step of the image classification method in the embodiment of the present application can be completed by an integrated logic circuit of hardware in the processor 4002 or instructions in the form of software.
  • the above-mentioned processor 4002 may also be a general-purpose processor, a digital signal processing (digital signal processing, DSP), an ASIC, a ready-made programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic Devices, discrete hardware components.
  • the aforementioned general-purpose processor may be a microprocessor or the processor may also be any conventional processor.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers.
  • the storage medium is located in the memory 4001, and the processor 4002 reads the information in the memory 4001, and combines its hardware to complete the functions required by the units included in the image classification apparatus of the embodiment of the application, or execute the image classification of the method embodiment of the application method.
  • the communication interface 4003 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 4000 and other devices or a communication network.
  • a transceiver device such as but not limited to a transceiver to implement communication between the device 4000 and other devices or a communication network.
  • the image to be processed can be acquired through the communication interface 4003.
  • the bus 4004 may include a path for transferring information between various components of the device 4000 (for example, the memory 4001, the processor 4002, and the communication interface 4003).
  • FIG. 13 is a schematic diagram of the hardware structure of the neural network training device 5000 according to an embodiment of the present application. Similar to the above device 4000, the neural network training device 5000 shown in FIG. 13 includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. Among them, the memory 5001, the processor 5002, and the communication interface 5003 implement communication connections between each other through the bus 5004.
  • the memory 5001 may store a program.
  • the processor 5002 is configured to execute each step of the training method for training the image classification device of the embodiment of the present application.
  • the processor 5002 may adopt a general-purpose CPU, a microprocessor, an ASIC, a GPU or one or more integrated circuits to execute related programs to implement the training method for training the image classification device of the embodiment of the present application.
  • the processor 5002 may also be an integrated circuit chip with signal processing capabilities.
  • each step of the training method of the image classification device of the embodiment of the present application can be completed by an integrated logic circuit of hardware in the processor 5002 or instructions in the form of software.
  • the image classification device is trained by the neural network training device 5000 shown in FIG. 13, and the image classification device obtained by training can be used to execute the image classification method of the embodiment of the present application. Specifically, training the neural network through the device 5000 can obtain the neural network in the method shown in FIG. 5 or FIG. 8.
  • the device shown in FIG. 13 can obtain training data and the image classification device to be trained from the outside through the communication interface 5003, and then the processor trains the image classification device to be trained according to the training data.
  • the device 4000 and device 5000 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device 4000 and device 5000 may also include those necessary for normal operation. Other devices. At the same time, according to specific needs, those skilled in the art should understand that the device 4000 and the device 5000 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the device 4000 and the device 5000 may also only include the components necessary to implement the embodiments of the present application, and not necessarily include all the components shown in FIG. 12 and FIG. 13.
  • the processor in this embodiment of the application may be a central processing unit (central processing unit, CPU), and the processor may also be other general-purpose processors, digital signal processors (digital signal processors, DSP), and application-specific integrated circuits. (application specific integrated circuit, ASIC), ready-made programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the memory in the embodiments of the present application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.
  • the non-volatile memory can be read-only memory (ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), and electronic Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
  • the volatile memory may be random access memory (RAM), which is used as an external cache.
  • RAM random access memory
  • static random access memory static random access memory
  • DRAM dynamic random access memory
  • DRAM synchronous dynamic random access memory
  • Access memory synchronous DRAM, SDRAM
  • double data rate synchronous dynamic random access memory double data rate SDRAM, DDR SDRAM
  • enhanced synchronous dynamic random access memory enhanced SDRAM, ESDRAM
  • synchronous connection dynamic random access memory Take memory (synchlink DRAM, SLDRAM) and direct memory bus random access memory (direct rambus RAM, DR RAM).
  • the foregoing embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination.
  • the above-mentioned embodiments may be implemented in the form of a computer program product in whole or in part.
  • the computer program product includes one or more computer instructions or computer programs.
  • the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more sets of available media.
  • the usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium.
  • the semiconductor medium may be a solid state drive.
  • At least one refers to one or more, and “multiple” refers to two or more.
  • the following at least one item (a)” or similar expressions refers to any combination of these items, including any combination of a single item (a) or plural items (a).
  • at least one item (a) of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple .
  • the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, rather than corresponding to the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • the disclosed system, device, and method may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • each unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of this application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program code .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Mathematical Physics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Computational Linguistics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Neurology (AREA)
  • Image Analysis (AREA)

Abstract

本申请涉及人工智能领域中计算机视觉领域的图像识别技术，提供了一种图像分类方法及装置。涉及人工智能领域，具体涉及计算机视觉领域。该方法包括：获取待处理图像的输入特征图；根据神经网络的M个卷积核对所述输入特征图进行卷积处理，得到M个通道的候选输出特征图，M为正整数；根据N个矩阵对所述候选输出特征图的M个通道进行矩阵变换，得到N个通道的输出特征图，其中，所述N个矩阵中的每个矩阵的通道数小于M，N大于M，N为正整数；根据所述输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。该方法有助于降低图像分类处理的计算量和参数量。

Description

图像分类方法及装置
本申请要求于2019年07月30日提交中国专利局、申请号为201910697287.0、申请名称为“图像分类方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能领域，并且更具体地，涉及一种图像分类方法及装置。
背景技术
计算机视觉是各个应用领域，如制造业、检验、文档分析、医疗诊断，和军事等领域中各种智能/自主系统中不可分割的一部分，它是一门关于如何运用照相机/摄像机和计算机来获取我们所需的，被拍摄对象的数据与信息的学问。形象地说，就是给计算机安装上眼睛(照相机/摄像机)和大脑(算法)用来代替人眼对目标进行识别、跟踪和测量等，从而使计算机能够感知环境。因为感知可以看作是从感官信号中提取信息，所以计算机视觉也可以看作是研究如何使人工系统从图像或多维数据中“感知”的科学。总的来说，计算机视觉就是用各种成象系统代替视觉器官获取输入信息，再由计算机来代替大脑对这些输入信息完成处理和解释。计算机视觉的最终研究目标就是使计算机能像人那样通过视觉观察和理解世界，具有自主适应环境的能力。
图像分类是各类图像处理应用的基础，计算机视觉常常会涉及到如何对获取到的图像进行分类的问题。随着人工智能技术的快速发展，基于深度学习的卷积神经网络(convolutional neural networks，CNN)在图像分类处理中得到了越来越广泛的应用。但是，卷积神经网络所包含的参数量和计算量都过大。
因此，如何降低神经网络的运算开销成为一个亟需解决的问题。
发明内容
一种图像分类方法及装置，该方法有助于降低图像分类处理的计算量和参数量。
第一方面，提供了一种图像分类方法，该方法包括：获取待处理图像的输入特征图；根据神经网络的M个卷积核对所述输入特征图进行卷积处理，得到M个通道的候选输出特征图，M为正整数；根据N个矩阵对所述候选输出特征图的M个通道进行矩阵变换，得到N个通道的输出特征图，其中，所述N个矩阵中的每个矩阵的通道数小于M，N大于M，N为正整数；根据所述输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
可选地，所述M个卷积核可以为现有卷积神经网络中的标准卷积核。
需要说明的是，在本申请实施例中，标准卷积核的通道数与输入特征图的通道数相同，例如，若待处理图像的输入特征图的通道数为C，那么，现有卷积神经网络中的标准卷积 核的通道数也为C，即标准卷积核的通道数与输入特征图的通道数相同，其中，C为正整数。
在本申请实施例中，通过少量标准卷积核(即M个卷积核)对待处理图像进行卷积处理，得到少量的候选特征图，并对这些少量的候选特征图进行矩阵变换以得到所需的输出特征图，其中，标准卷积核的个数少于现有卷积神经网络中的标准卷积核的个数，同时，矩阵变换中使用的矩阵的通道数也小于标准卷积核，因此，有助于减少神经网络模型的计算量和参数量，从而降低图像分类处理的计算量和参数量。
可选地，所述N个矩阵中每个矩阵的通道数可以为1，或者，所述N个矩阵中每个矩阵的通道数也可以大于1。
结合第一方面，在第一方面的某些实现方式中，所述N个矩阵包括M组卷积核，所述M组卷积核分别与所述候选输出特征图的M个通道对应；所述根据N个矩阵对所述M个候选输出特征图进行矩阵变换，得到N个通道的输出特征图，包括：根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行卷积，得到所述N个通道的输出特征图。
结合第一方面，在第一方面的某些实现方式中，所述根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行卷积，得到所述N个通道的输出特征图，包括：根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行深度卷积，得到所述N个通道的输出特征图。
结合第一方面，在第一方面的某些实现方式中，所述M组卷积核中的每一组的卷积核与所述M组卷积核中的其他组的卷积核相同。
结合第一方面，在第一方面的某些实现方式中，所述根据所述输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果，包括：对所述候选输出特征图和所述输出特征图进行特征拼接，得到特征拼接特征图，所述特征拼接特征图的通道数为M+N；根据所述特征拼接特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
可选地，上述特征拼接可以是指所述候选输出特征图和所述输出特征图在深度方向上组成新的特征图，即上述特征拼接特征图。
例如，所述候选输出特征图的通道数为M，所述输出特征图的通道数为N，所述候选输出特征图和所述输出特征图可以进行特征拼接，得到一个通道数为M+N的特征拼接特征图。
特征拼接可以通过恒等特征映射的方式，可以将更多的细节(或特征)引入输出特征图，同时，这种恒等映射并不会引入额外的参数或者计算量，因此可以在不增加参数量和计算量的情况下，提升图像分类的效果。
第二方面，提供了一种图像分类方法，该方法包括：获取待处理图像的输入特征图；根据神经网络的M个第一卷积核对所述输入特征图进行卷积处理，得到M个通道的第一候选输出特征图，M为正整数；根据K个第一矩阵对所述M个第一候选输出特征图进行矩阵变换，得到K个通道的第一输出特征图，其中，所述K个第一矩阵中的每个矩阵的通道数小于M，K大于M，K为正整数；根据神经网络的P个第二卷积核对所述第一输出特征图进行卷积处理，得到P个通道的第二候选输出特征图，P为正整数；根据N个第二矩阵对所述P个第二候选输出特征图进行矩阵变换，得到N个通道的第二输出特征图， 其中，所述N个第二矩阵中的每个矩阵的通道数小于P，N大于P，N为正整数；根据所述第二输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
可选地，所述M个第一卷积核可以为现有卷积神经网络中的标准卷积核。
需要说明的是，在本申请实施例中，标准卷积核的通道数与输入特征图的通道数相同，例如，若待处理图像的输入特征图的通道数为C，那么，现有卷积神经网络中的标准卷积核的通道数也为C，即标准卷积核的通道数与输入特征图的通道数相同，其中，C为正整数。
类似地，所述P个第二卷积核也可以为现有卷积神经网络中的标准卷积核。
在本申请实施例中，通过少量标准卷积核及矩阵变换实现现有卷积神经网络中的卷积处理，可以有效减少各输出特征图之间的冗余性，减少神经网络模型的计算量和参数量，从而降低图像分类处理的计算量和参数量，因此，本申请实施例中的图像分类方法可以在不增加参数量和计算量(或者减少参数量和计算量)的情况下，提升图像分类的效果。
可选地，所述K个第一矩阵中每个矩阵的通道数可以为1，或者，所述K个第一矩阵中每个矩阵的通道数也可以大于1。
可选地，所述N个第二矩阵中每个矩阵的通道数可以为1，或者，所述N个第二矩阵中每个矩阵的通道数也可以大于1。
结合第二方面，在第二方面的某些实现方式中，所述根据神经网络的P个第二卷积核对所述第一输出特征图进行卷积处理，得到P个通道的第二候选输出特征图，包括：对所述第一输出特征图进行深度卷积，得到深度卷积特征图；根据所述P个第二卷积核对所述深度卷积特征图进行卷积处理，得到所述第二候选输出特征图。
结合第二方面，在第二方面的某些实现方式中，所述对所述第一输出特征图进行深度卷积，得到深度卷积特征图，包括：对所述第一输出特征图进行步幅大于1的深度卷积，得到所述深度卷积特征图。
结合第二方面，在第二方面的某些实现方式中，所述根据所述第二输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果，包括：对所述输入特征图和所述第二输出特征图进行残差连接，得到残差连接特征图；根据所述残差连接特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
在本申请实施例中，通过残差连接可以将更多的细节(或特征)引入输出特征图，而残差连接并不会引入额外的参数或者计算量，因此，可以在不增加参数量和计算量的情况下，提升图像分类的效果。
第三方面，提供了一种图像分类装置，包括：获取单元，用于获取待处理图像的输入特征图；卷积单元，用于根据神经网络的M个卷积核对所述输入特征图进行卷积处理，得到M个通道的候选输出特征图，M为正整数；矩阵变换单元，用于根据N个矩阵对所述候选输出特征图的M个通道进行矩阵变换，得到N个通道的输出特征图，其中，所述N个矩阵中的每个矩阵的通道数小于M，N大于M，N为正整数；分类单元，用于根据所述输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
可选地，所述M个卷积核可以为现有卷积神经网络中的标准卷积核。
需要说明的是，在本申请实施例中，标准卷积核的通道数与输入特征图的通道数相同，例如，若待处理图像的输入特征图的通道数为C，那么，现有卷积神经网络中的标准卷积 核的通道数也为C，即标准卷积核的通道数与输入特征图的通道数相同，其中，C为正整数。
在本申请实施例中，通过少量标准卷积核(即M个卷积核)对待处理图像进行卷积处理，得到少量的候选特征图，并对这些少量的候选特征图进行矩阵变换以得到所需的输出特征图，其中，标准卷积核的个数少于现有卷积神经网络中的标准卷积核的个数，同时，矩阵变换中使用的矩阵的通道数也小于标准卷积核，因此，有助于减少神经网络模型的计算量和参数量，从而降低图像分类处理的计算量和参数量。
可选地，所述N个矩阵中每个矩阵的通道数可以为1，或者，所述N个矩阵中每个矩阵的通道数也可以大于1。
结合第三方面，在第三方面的某些实现方式中，所述N个矩阵包括M组卷积核，所述M组卷积核分别与所述候选输出特征图的M个通道对应；所述矩阵变换单元具体用于：根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行卷积，得到所述N个通道的输出特征图。
结合第三方面，在第三方面的某些实现方式中，所述矩阵变换单元具体用于：根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行深度卷积，得到所述N个通道的输出特征图。
结合第三方面，在第三方面的某些实现方式中，所述M组卷积核中的每一组的卷积核与所述M组卷积核中的其他组的卷积核相同。
结合第三方面，在第三方面的某些实现方式中，所述分类单元具体用于：对所述候选输出特征图和所述输出特征图进行特征拼接，得到特征拼接特征图，所述特征拼接特征图的通道数为M+N；根据所述特征拼接特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
可选地，上述特征拼接可以是指所述候选输出特征图和所述输出特征图在深度方向上组成新的特征图，即上述特征拼接特征图。
例如，所述候选输出特征图的通道数为M，所述输出特征图的通道数为N，所述候选输出特征图和所述输出特征图可以进行特征拼接，得到一个通道数为M+N的特征拼接特征图。
特征拼接可以通过恒等特征映射的方式，可以将更多的细节(或特征)引入输出特征图，同时，这种恒等映射并不会引入额外的参数或者计算量，因此可以在不增加参数量和计算量的情况下，提升图像分类的效果。
第四方面，提供了一种图像分类装置，包括：获取单元，用于获取待处理图像的输入特征图；第一卷积单元，用于根据神经网络的M个第一卷积核对所述输入特征图进行卷积处理，得到M个通道的第一候选输出特征图，M为正整数；第一矩阵变换单元，用于根据K个第一矩阵对所述M个第一候选输出特征图进行矩阵变换，得到K个通道的第一输出特征图，其中，所述K个第一矩阵中的每个矩阵的通道数小于M，K大于M，K为正整数；第二卷积单元，用于根据神经网络的P个第二卷积核对所述第一输出特征图进行卷积处理，得到P个通道的第二候选输出特征图，P为正整数；第二矩阵变换单元，用于根据N个第二矩阵对所述P个第二候选输出特征图进行矩阵变换，得到N个通道的第二输出特征图，其中，所述N个第二矩阵中的每个矩阵的通道数小于P，N大于P，N为正 整数；分类单元，用于根据所述第二输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
可选地，所述M个第一卷积核可以为现有卷积神经网络中的标准卷积核。
需要说明的是，在本申请实施例中，标准卷积核的通道数与输入特征图的通道数相同，例如，若待处理图像的输入特征图的通道数为C，那么，现有卷积神经网络中的标准卷积核的通道数也为C，即标准卷积核的通道数与输入特征图的通道数相同，其中，C为正整数。
类似地，所述P个第二卷积核也可以为现有卷积神经网络中的标准卷积核。
在本申请实施例中，通过少量标准卷积核及矩阵变换实现现有卷积神经网络中的卷积处理，可以有效减少各输出特征图之间的冗余性，减少神经网络模型的计算量和参数量，从而降低图像分类处理的计算量和参数量，因此，本申请实施例中的图像分类方法可以在不增加参数量和计算量(或者减少参数量和计算量)的情况下，提升图像分类的效果。
可选地，所述K个第一矩阵中每个矩阵的通道数可以为1，或者，所述K个第一矩阵中每个矩阵的通道数也可以大于1。
可选地，所述N个第二矩阵中每个矩阵的通道数可以为1，或者，所述N个第二矩阵中每个矩阵的通道数也可以大于1。
结合第四方面，在第四方面的某些实现方式中，所述图像分类装置还包括深度卷积单元，用于：对所述第一输出特征图进行深度卷积，得到深度卷积特征图；所述第二卷积单元具体用于：根据所述P个第二卷积核对所述深度卷积特征图进行卷积处理，得到所述第二候选输出特征图。
结合第四方面，在第四方面的某些实现方式中，所述深度卷积单元具体用于：对所述第一输出特征图进行步幅大于1的深度卷积，得到所述深度卷积特征图。
结合第四方面，在第四方面的某些实现方式中，所述图像分类装置还包括残差连接单元，用于：对所述输入特征图和所述第二输出特征图进行残差连接，得到残差连接特征图；所述分类单元具体用于：根据所述残差连接特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
在本申请实施例中，通过残差连接可以将更多的细节(或特征)引入输出特征图，而残差连接并不会引入额外的参数或者计算量，因此，可以在不增加参数量和计算量的情况下，提升图像分类的效果。
第五方面，提供了一种图像分类装置，该装置包括：存储器，用于存储程序；处理器，用于执行所述存储器存储的程序，当所述存储器存储的程序被执行时，所述处理器用于执行上述第一方面或第二方面中的任意一种实现方式中的方法。
上述第五方面中的处理器既可以是中央处理器(central processing unit，CPU)，也可以是CPU与神经网络运算处理器的组合，这里的神经网络运算处理器可以包括图形处理器(graphics processing unit，GPU)、神经网络处理器(neural-network processing unit，NPU)和张量处理器(tensor processing unit，TPU)等等。其中，TPU是谷歌(google)为机器学习全定制的人工智能加速器专用集成电路。
第六方面，提供一种计算机可读介质，该计算机可读介质存储用于设备执行的程序代码，该程序代码包括用于执行第一方面或第二方面中的任意一种实现方式中的方法。
第七方面，提供一种包含指令的计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述第一方面或第二方面中的任意一种实现方式中的方法。
第八方面，提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行上述第一方面或第二方面中的任意一种实现方式中的方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行第一方面或第二方面中的任意一种实现方式中的方法。
上述芯片具体可以是现场可编程门阵列(field－programmable gate array，FPGA)或者专用集成电路(application-specific integrated circuit，ASIC)。
第九方面，提供了一种电子设备，该电子设备包括上述第三方面或第四方面中的任意一个方面中的图像分类装置。
当上述电子设备包括上述第三方面或第四方面中的任意一个方面中的图像分类装置时，该电子设备具体可以是终端设备或服务器。
在本申请实施例中，通过少量标准卷积核(即M个卷积核)对待处理图像进行卷积处理，得到少量的候选特征图，并对这些少量的候选特征图进行矩阵变换以得到所需的输出特征图，其中，标准卷积核的个数少于现有卷积神经网络中的标准卷积核的个数，同时，矩阵变换中使用的矩阵的通道数也小于标准卷积核，因此，有助于减少神经网络模型的计算量和参数量，从而降低图像分类处理的计算量和参数量。
附图说明
图1是本申请实施例提供的系统架构的结构示意图。
图2是本申请实施例提供的根据卷积神经网络模型进行图像分类的示意图。
图3是本申请实施例提供的一种芯片硬件结构示意图。
图4是本申请实施例提供的一种应用场景示意图。
图5是本申请一个实施例提供的图像分类方法的示意性流程图。
图6是本申请实施例提供的卷积处理的示意性框图。
图7是本申请实施例提供的特征扩增单元的示意性框图。
图8是本申请另一个实施例提供的图像分类方法的示意性流程图。
图9是本申请一个实施例提供的纺锤模块的示意性框图。
图10是本申请另一个实施例提供的纺锤模块的示意性框图。
图11是本申请实施例提供的神经网络的示意性框图。
图12是本申请实施例的图像分类装置的硬件结构示意图。
图13是本申请实施例的神经网络训练装置的硬件结构示意图。
具体实施方式
下面将结合附图，对本申请中的技术方案进行描述。
本申请实施例提供的图像分类方法能够应用在图片检索、相册管理、平安城市、人机交互以及其他需要进行图像分类或者图像识别的场景。应理解，本申请实施例中的图像可 以为静态图像(或称为静态画面)或动态图像(或称为动态画面)，例如，本申请中的图像可以为视频或动态图片，或者，本申请中的图像也可以为静态图片或照片。为了便于描述，本申请在下述实施例中将静态图像或动态图像统一称为图像。
本申请实施例的图像分类方法可以具体应用到相册分类和拍照识别场景中，下面对这两种场景进行详细的介绍。
相册分类：
用户在手机和云盘上存储了大量图片，按照类别对相册进行分类管理能提高用户的体验。利用本申请实施例的图像分类方法对相册中的图片进行分类，能够得到按照类别进行排列或者存储的相册。本申请实施例的图片分类方法可以方便用户对不同的物体类别进行分类管理，从而方便用户的查找，能够节省用户的管理时间，提高相册管理的效率。
具体地，在采用本申请实施例的图像分类方法进行相册分类时，可以利用本申请提供的神经网络，先提取相册中图片的图片特征，然后再根据提取到的图片特征对相册中的图片进行分类，得到图片的分类结果，接下来，再根据图片的分类结果对相册中的图片进行分类，得到按照图片类别进行排列的相册。其中，在根据图片类别对相册中的图片进行排列时，可以将属于同一类的图片排列在一行或者一行。例如，在最终得到的相册中，第一行的图片都属于飞机，第二行的图片都属于汽车。
拍照识物：
用户在拍照时，可以利用本申请实施例的图像分类方法对拍到的照片进行处理，能够自动识别出被拍物体的类别，例如，可以自动识别出被拍物体是花卉、动物等。进一步地，利用本申请实施例的图像分类方法可以对拍照得到的物体进行识别，识别出该物体所属的类别，例如，用户拍照得到的照片中包括共享单车，利用本申请实施例的图像分类方法能够对共享单车进行识别，识别出该物体属于自行车，进一步地，还可以显示自行车的相关信息。
应理解，上文介绍的相册分类和拍照识物只是本申请实施例的图像分类方法所应用的两个具体场景，本申请实施例的图像分类方法在应用时并不限于上述两个场景，本申请实施例的图像分类方法能够应用到任何需要进行图像分类或者图像识别的场景中。
本申请实施例中的图像分类方法中使用了一种新的神经网络模型，该模型也可以类似地应用于其他使用神经网络的领域，例如，人脸识别、语音识别、目标检测、机器翻译及语义分割等。
本申请实施例涉及了大量神经网络的相关应用，为了更好地理解本申请实施例的方案，下面先对本申请实施例可能涉及的神经网络的相关术语和概念进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以x s和截距1为输入的运算单元，该运算单元的输出可以如公式(1-1)所示：
Figure PCTCN2020105830-appb-000001
其中，s＝1、2、……n，n为大于1的自然数，W s为x s的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入，激活函数可以是sigmoid函数。神经网络是将多个上述单一的神经单元联结在一 起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(2)深度神经网络
深度神经网络(deep neural network，DNN)，也称多层神经网络，可以理解为具有多层隐含层的神经网络。按照不同层的位置对DNN进行划分，DNN内部的神经网络可以分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定与第i+1层的任意一个神经元相连。
虽然DNN看起来很复杂，但是就每一层的工作来说，其实并不复杂，简单来说就是如下线性关系表达式：
Figure PCTCN2020105830-appb-000002
其中，
Figure PCTCN2020105830-appb-000003
是输入向量，
Figure PCTCN2020105830-appb-000004
是输出向量，
Figure PCTCN2020105830-appb-000005
是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量
Figure PCTCN2020105830-appb-000006
经过如此简单的操作得到输出向量
Figure PCTCN2020105830-appb-000007
由于DNN层数多，系数W和偏移向量
Figure PCTCN2020105830-appb-000008
的数量也比较多。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为
Figure PCTCN2020105830-appb-000009
上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。
综上，第L-1层的第k个神经元到第L层的第j个神经元的系数定义为
Figure PCTCN2020105830-appb-000010
需要注意的是，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。训练深度神经网络的也就是学习权重矩阵的过程，其最终目的是得到训练好的深度神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。
(3)卷积神经网络
卷积神经网络(convolutional neuron network，CNN)是一种带有卷积结构的深度神经网络。卷积神经网络包含了一个由卷积层和子采样层构成的特征抽取器，该特征抽取器可以看作是滤波器。卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。在卷积神经网络的卷积层中，一个神经元可以只与部分邻层神经元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。卷积核可以以随机大小的矩阵的形式初始化，在卷积神经网络的训练过程中卷积核可以通过学习得到合理的权重。另外，共享权重带来的直接好处是减少卷积神经网络各层之间的连接，同时又降低了过拟合的风险。
(4)循环神经网络(recurrent neural networks，RNN)是用来处理序列数据的。在传统的神经网络模型中，是从输入层到隐含层再到输出层，层与层之间是全连接的，而对于每一层层内之间的各个节点是无连接的。这种普通的神经网络虽然解决了很多难题，但是却仍然对很多问题无能无力。例如，你要预测句子的下一个单词是什么，一般需要用到前面的单词，因为一个句子中前后单词并不是独立的。RNN之所以称为循环神经网路，即一个序列当前的输出与前面的输出也有关。具体的表现形式为网络会对前面的信息进行记忆并应用于当前输出的计算中，即隐含层本层之间的节点不再无连接而是有连接的，并且隐 含层的输入不仅包括输入层的输出还包括上一时刻隐含层的输出。理论上，RNN能够对任何长度的序列数据进行处理。对于RNN的训练和对传统的CNN或DNN的训练一样。
既然已经有了卷积神经网络，为什么还要循环神经网络？原因很简单，在卷积神经网络中，有一个前提假设是：元素之间是相互独立的，输入与输出也是独立的，比如猫和狗。但现实世界中，很多元素都是相互连接的，比如股票随时间的变化，再比如一个人说了：我喜欢旅游，其中最喜欢的地方是云南，以后有机会一定要去。这里填空，人类应该都知道是填“云南”。因为人类会根据上下文的内容进行推断，但如何让机器做到这一步？RNN就应运而生了。RNN旨在让机器像人一样拥有记忆的能力。因此，RNN的输出就需要依赖当前的输入信息和历史的记忆信息。
(5)损失函数
在训练深度神经网络的过程中，因为希望深度神经网络的输出尽可能的接近真正想要预测的值，所以可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重向量(当然，在第一次更新之前通常会有初始化的过程，即为深度神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重向量让它预测低一些，不断地调整，直到深度神经网络能够预测出真正想要的目标值或与真正想要的目标值非常接近的值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么深度神经网络的训练就变成了尽可能缩小这个loss的过程。
(6)反向传播算法
神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的神经网络模型中参数的大小，使得神经网络模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的神经网络模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的神经网络模型的参数，例如权重矩阵。
(7)像素值
图像的像素值可以是一个红绿蓝(RGB)颜色值，像素值可以是表示颜色的长整数。例如，像素值为256*Red+100*Green+76Blue，其中，Blue代表蓝色分量，Green代表绿色分量，Red代表红色分量。各个颜色分量中，数值越小，亮度越低，数值越大，亮度越高。对于灰度图像来说，像素值可以是灰度值。
如图1所示，本申请实施例提供了一种系统架构100。在图1中，数据采集设备160用于采集训练数据。针对本申请实施例的图像分类方法来说，训练数据可以包括训练图像以及训练图像对应的分类结果，其中，训练图像的分类结果可以是人工预先标注的结果。
在采集到训练数据之后，数据采集设备160将这些训练数据存入数据库130，训练设备120基于数据库130中维护的训练数据训练得到目标模型/规则101。
下面对训练设备120基于训练数据得到目标模型/规则101进行描述，训练设备120对输入的原始图像进行处理，将输出的图像与原始图像进行对比，直到训练设备120输出的图像与原始图像的差值小于一定的阈值，从而完成目标模型/规则101的训练。
上述目标模型/规则101能够用于实现本申请实施例的图像分类方法，即，将待处理图像通过相关预处理后输入该目标模型/规则101，即可得到图像的分类结果。本申请实施例中的目标模型/规则101具体可以为神经网络。需要说明的是，在实际的应用中，所述数据库130中维护的训练数据不一定都来自于数据采集设备160的采集，也有可能是从其他设备接收得到的。另外需要说明的是，训练设备120也不一定完全基于数据库130维护的训练数据进行目标模型/规则101的训练，也有可能从云端或其他地方获取训练数据进行模型训练，上述描述不应该作为对本申请实施例的限定。
根据训练设备120训练得到的目标模型/规则101可以应用于不同的系统或设备中，如应用于图1所示的执行设备110，所述执行设备110可以是终端，如手机终端，平板电脑，笔记本电脑，增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)，车载终端等，还可以是服务器或者云端设备等。在图1中，执行设备110配置输入/输出(input/output，I/O)接口112，用于与外部设备进行数据交互，用户可以通过客户设备140向I/O接口112输入数据，所述输入数据在本申请实施例中可以包括：客户设备输入的待处理图像。
预处理模块113和预处理模块114用于根据I/O接口112接收到的输入数据(如待处理图像)进行预处理，在本申请实施例中，也可以没有预处理模块113和预处理模块114(也可以只有其中的一个预处理模块)，而直接采用计算模块111对输入数据进行处理。
在执行设备110对输入数据进行预处理，或者在执行设备110的计算模块111执行计算等相关的处理过程中，执行设备110可以调用数据存储系统150中的数据、代码等以用于相应的处理，也可以将相应处理得到的数据、指令等存入数据存储系统150中。
最后，I/O接口112将处理结果，如上述得到的待处理图像的分类结果返回给客户设备140，从而提供给用户。
值得说明的是，训练设备120可以针对不同的目标或称不同的任务，基于不同的训练数据生成相应的目标模型/规则101，该相应的目标模型/规则101即可以用于实现上述目标或完成上述任务，从而为用户提供所需的结果。
在图1所示情况下，用户可以手动给定输入数据，该手动给定可以通过I/O接口112提供的界面进行操作。另一种情况下，客户设备140可以自动地向I/O接口112发送输入数据，如果要求客户设备140自动发送输入数据需要获得用户的授权，则用户可以在客户设备140中设置相应权限。用户可以在客户设备140查看执行设备110输出的结果，具体的呈现形式可以是显示、声音、动作等具体方式。客户设备140也可以作为数据采集端，采集如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果作为新的样本数据，并存入数据库130。当然，也可以不经过客户设备140进行采集，而是由I/O接口112直接将如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果，作为新的样本数据存入数据库130。
值得注意的是，图1仅是本申请实施例提供的一种系统架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制，例如，在图1中，数据存储系统150相对执行设备110是外部存储器，在其它情况下，也可以将数据存储系统150置于执行设备110中。
如图1所示，根据训练设备120训练得到目标模型/规则101，该目标模型/规则101 在本申请实施例中可以是本申请中的神经网络，具体的，本申请实施例提供的神经网络可以CNN，深度卷积神经网络(deep convolutional neural networks,DCNN)，循环神经网络(recurrent neural network，RNNS)等等。
由于CNN是一种非常常见的神经网络，下面结合图2重点对CNN的结构进行详细的介绍。如上文的基础概念介绍所述，卷积神经网络是一种带有卷积结构的深度神经网络，是一种深度学习(deep learning)架构，深度学习架构是指通过机器学习的算法，在不同的抽象层级上进行多个层次的学习。作为一种深度学习架构，CNN是一种前馈(feed-forward)人工神经网络，该前馈人工神经网络中的各个神经元可以对输入其中的图像作出响应。
如图2所示，卷积神经网络(CNN)200可以包括输入层210，卷积层/池化层220(其中池化层为可选的)，以及神经网络层230。下面对这些层的相关内容做详细介绍。
卷积层/池化层220：
卷积层：
如图2所示卷积层/池化层220可以包括如示例221-226层，举例来说：在一种实现方式中，221层为卷积层，222层为池化层，223层为卷积层，224层为池化层，225为卷积层，226为池化层；在另一种实现方式中，221、222为卷积层，223为池化层，224、225为卷积层，226为池化层。即卷积层的输出可以作为随后的池化层的输入，也可以作为另一个卷积层的输入以继续进行卷积操作。
下面将以卷积层221为例，介绍一层卷积层的内部工作原理。
卷积层221可以包括很多个卷积算子，卷积算子也称为核，其在图像处理中的作用相当于一个从输入图像矩阵中提取特定信息的过滤器，卷积算子本质上可以是一个权重矩阵，这个权重矩阵通常被预先定义，在对图像进行卷积操作的过程中，权重矩阵通常在输入图像上沿着水平方向一个像素接着一个像素(或两个像素接着两个像素……这取决于步长stride的取值)的进行处理，从而完成从图像中提取特定特征的工作。该权重矩阵的大小应该与图像的大小相关，需要注意的是，权重矩阵的纵深维度(depth dimension)和输入图像的纵深维度是相同的，在进行卷积运算的过程中，权重矩阵会延伸到输入图像的整个深度。因此，和一个单一的权重矩阵进行卷积会产生一个单一纵深维度的卷积化输出，但是大多数情况下不使用单一权重矩阵，而是应用多个尺寸(行×列)相同的权重矩阵，即多个同型矩阵。每个权重矩阵的输出被堆叠起来形成卷积图像的纵深维度，这里的维度可以理解为由上面所述的“多个”来决定。不同的权重矩阵可以用来提取图像中不同的特征，例如一个权重矩阵用来提取图像边缘信息，另一个权重矩阵用来提取图像的特定颜色，又一个权重矩阵用来对图像中不需要的噪点进行模糊化等。该多个权重矩阵尺寸(行×列)相同，经过该多个尺寸相同的权重矩阵提取后的特征图的尺寸也相同，再将提取到的多个尺寸相同的特征图合并形成卷积运算的输出。
这些权重矩阵中的权重值在实际应用中需要经过大量的训练得到，通过训练得到的权重值形成的各个权重矩阵可以用来从输入图像中提取信息，从而使得卷积神经网络200进行正确的预测。
当卷积神经网络200有多个卷积层的时候，初始的卷积层(例如221)往往提取较多的一般特征，该一般特征也可以称之为低级别的特征；随着卷积神经网络200深度的加深， 越往后的卷积层(例如226)提取到的特征越来越复杂，比如高级别的语义之类的特征，语义越高的特征越适用于待解决的问题。
池化层/池化层220：
由于常常需要减少训练参数的数量，因此卷积层之后常常需要周期性的引入池化层，在如图2中220所示例的221-226各层，可以是一层卷积层后面跟一层池化层，也可以是多层卷积层后面接一层或多层池化层。在图像处理过程中，池化层的唯一目的就是减少图像的空间大小。池化层可以包括平均池化算子和/或最大池化算子，以用于对输入图像进行采样得到较小尺寸的图像。平均池化算子可以在特定范围内对图像中的像素值进行计算产生平均值作为平均池化的结果。最大池化算子可以在特定范围内取该范围内值最大的像素作为最大池化的结果。另外，就像卷积层中用权重矩阵的大小应该与图像尺寸相关一样，池化层中的运算符也应该与图像的大小相关。通过池化层处理后输出的图像尺寸可以小于输入池化层的图像的尺寸，池化层输出的图像中每个像素点表示输入池化层的图像的对应子区域的平均值或最大值。
神经网络层230：
在经过卷积层/池化层220的处理后，卷积神经网络200还不足以输出所需要的输出信息。因为如前所述，卷积层/池化层220只会提取特征，并减少输入图像带来的参数。然而为了生成最终的输出信息(所需要的类信息或其他相关信息)，卷积神经网络200需要利用神经网络层230来生成一个或者一组所需要的类的数量的输出。因此，在神经网络层230中可以包括多层隐含层(如图2所示的231、232至23n)以及输出层240，该多层隐含层中所包含的参数可以根据具体的任务类型的相关训练数据进行预先训练得到，例如该任务类型可以包括图像识别，图像分类，图像超分辨率重建等等。
在神经网络层230中的多层隐含层之后，也就是整个卷积神经网络200的最后层为输出层240，该输出层240具有类似分类交叉熵的损失函数，具体用于计算预测误差，一旦整个卷积神经网络200的前向传播(如图2由210至240方向的传播为前向传播)完成，反向传播(如图2由240至210方向的传播为反向传播)就会开始更新前面提到的各层的权重值以及偏差，以减少卷积神经网络200的损失，及卷积神经网络200通过输出层输出的结果和理想结果之间的误差。
需要说明的是，如图2所示的卷积神经网络200仅作为一种卷积神经网络的示例，在具体的应用中，卷积神经网络还可以以其他网络模型的形式存在。
本申请中，可以采用图2所示的卷积神经网络200对待处理图像进行处理，得到待处理图像的分类结果。如图2所示，待处理图像经过输入层210、卷积层/池化层220以及神经网络层230的处理后输出待处理图像的分类结果。
图3为本申请实施例提供的一种芯片硬件结构，该芯片包括神经网络处理器50。该芯片可以被设置在如图1所示的执行设备110中，用以完成计算模块111的计算工作。该芯片也可以被设置在如图1所示的训练设备120中，用以完成训练设备120的训练工作并输出目标模型/规则101。如图2所示的卷积神经网络中各层的算法均可在如图3所示的芯片中得以实现。
神经网络处理器NPU 50作为协处理器挂载到主CPU(host CPU)上，由主CPU分配任务。NPU的核心部分为运算电路503，控制器504控制运算电路503提取存储器(权 重存储器或输入存储器)中的数据并进行运算。
在一些实现方式中，运算电路503内部包括多个处理单元(process engine,PE)。在一些实现方式中，运算电路503是二维脉动阵列。运算电路503还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现方式中，运算电路503是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路503从权重存储器502中取矩阵B相应的数据，并缓存在运算电路503中每一个PE上。运算电路503从输入存储器501中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器(accumulator)508中。
向量计算单元507可以对运算电路503的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。例如，向量计算单元507可以用于神经网络中非卷积/非FC层的网络计算，如池化(pooling)，批归一化(batch normalization)，局部响应归一化(local response normalization)等。
在一些实现方式中，向量计算单元能507将经处理的输出的向量存储到统一缓存器506。例如，向量计算单元507可以将非线性函数应用到运算电路503的输出，例如累加值的向量，用以生成激活值。在一些实现方式中，向量计算单元507生成归一化的值、合并值，或二者均有。在一些实现方式中，处理过的输出的向量能够用作到运算电路503的激活输入，例如用于在神经网络中的后续层中的使用。
统一存储器506用于存放输入数据以及输出数据。
权重数据直接通过存储单元访问控制器505(direct memory access controller，DMAC)将外部存储器中的输入数据搬运到输入存储器501和/或统一存储器506、将外部存储器中的权重数据存入权重存储器502，以及将统一存储器506中的数据存入外部存储器。
总线接口单元(bus interface unit，BIU)510，用于通过总线实现主CPU、DMAC和取指存储器509之间进行交互。
与控制器504连接的取指存储器(instruction fetch buffer)509，用于存储控制器504使用的指令；
控制器504，用于调用指存储器509中缓存的指令，实现控制该运算加速器的工作过程。
一般地，统一存储器506，输入存储器501，权重存储器502以及取指存储器509均为片上(On-Chip)存储器，外部存储器为该NPU外部的存储器，该外部存储器可以为双倍数据率同步动态随机存储器(double data rate synchronous dynamic random access memory，简称DDR SDRAM)、高带宽存储器(high bandwidth memory，HBM)或其他可读可写的存储器。
其中，图2所示的卷积神经网络中各层的运算可以由运算电路503或向量计算单元307执行。
上文中介绍的图1中的执行设备110能够执行本申请实施例的图像分类方法的各个步骤，图2所示的CNN模型和图3所示的芯片也可以用于执行本申请实施例的图像分类方法的各个步骤。下面结合附图对本申请实施例的图像分类方法进行详细的介绍。
本申请实施例提供的图像分类方法可以在服务器上被执行，也可以在云端被执行，还 可以在终端设备上被执行。以终端设备为例，如图4所示，本发明实施例的技术方案可以应用于终端设备，本申请实施例中的图像分类方法可以对输入图像进行图像分类，得到该输入图像的分类结果。该终端设备可以为移动的或固定的，例如该终端设备可以是具有图像处理功能的移动电话、平板个人电脑(tablet personal computer，TPC)、媒体播放器、智能电视、笔记本电脑(laptop computer，LC)、个人数字助理(personal digital assistant，PDA)、个人计算机(personal computer，PC)、照相机、摄像机、智能手表、可穿戴式设备(wearable device，WD)或者自动驾驶的车辆等，本发明实施例对此不作限定。
图像(或图片)的分类是各类图像处理应用的基础，计算机视觉常常会涉及到如何对获取到的图像进行分类的问题。但是，高精度的卷积神经网络的参数量和计算量都很大，而终端设备的内存和计算资源都非常有限，并不具备有较强的运算能力和缓存能力，导致具有高精度的卷积神经网络在终端设备上难以进行部署。
本申请实施例提出了一种图像分类方法，通过少于现有卷积神经网络中的标准卷积核个数的少量标准卷积核，就可以得到所需的输出特征图，该方法有助于降低图像分类处理的计算量和参数量。
图5示出了本申请实施例提供的图像分类方法500的示意性流程图，该方法可以由能够进行图像分类装置执行，例如，该方法可以由图4中的终端设备执行。
S510，获取待处理图像的输入特征图。
当图5所示的方法由图4中的终端设备执行时，该待处理图像可以是终端设备通过摄像头拍摄到的图像，或者，该待处理图像还可以是从终端设备内部获得的图像(例如，终端设备的相册中存储的图像，或者终端设备从云端获取的图像)。
或者，该待处理图像的输入特征图可以是卷积神经网络中的其他层处理后得到的特征图。应理解，这里所说的卷积神经网络中的其他层是指卷积神经网络中的一个层，例如，该其他层可以是卷积神经网络中的输入层、卷积层、池化层或全连接层中的一个。
S520，根据神经网络的M个卷积核对所述输入特征图进行卷积处理，得到M个通道的候选输出特征图，M为正整数。
其中，所述M个卷积核可以为现有卷积神经网络中的标准卷积核。
需要说明的是，在本申请实施例中，标准卷积核的通道数与输入特征图的通道数相同，例如，若待处理图像的输入特征图的通道数为C，则现有卷积神经网络中的标准卷积核的通道数也为C，即，标准卷积核的通道数与输入特征图的通道数相同，其中，C为正整数。
S530，根据N个矩阵对所述候选输出特征图的M个通道进行矩阵变换，得到N个通道的输出特征图。
其中，所述N个矩阵中的每个矩阵的通道数小于M，N大于M，N为正整数。
上述S520和S530是本申请实施例中的特征扩增单元(feature in feature，FiF)，该特征扩增单元可以用于替换现有卷积神经网络模型中的卷积层。
该特征扩增单元使用少量标准卷积核进行卷积处理，得到少量的候选特征图，并对这些少量的候选特征图进行矩阵变换以得到所需的输出特征图，可以减少输出特征图之间的冗余性，有助于降低图像分类处理的计算量和参数量。下面结合图6和图7对上述S520和S530中的特征扩增单元进行详细描述。
图6所示的是现有卷积神经网络中的一个卷积层的卷积处理。
从图6中可以看出，该卷积层的输入特征图包括C个通道，该卷积层的输出特征图包括N个通道。在现有卷积神经网络中，若要对C个通道的输入特征图进行卷积处理，得到N个通道的输出特征图，则该卷积层中需要N个标准卷积核，该N个标准卷积核中的每个卷积核包括C个通道。
需要说明的是，上述C个通道的输入特征图可以是指一个输入特征图，该输入特征图的通道数为C；或者，上述C个通道的输入特征图也可以是指C个输入特征图，其中，每个输入特征图都是二维的(即通道数为1)。为便于理解，本申请中统一描述为C个通道的输入特征图。本申请实施例中的其他特征图的描述都可以类似地理解，这里不再赘述。
例如，在上述S520中，得到M个通道的候选输出特征图，该M个通道的候选输出特征图既可以认为是一个包括M个通道的候选输出特征图，也可以是M个通道数为1的候选输出特征图。
图7所示的是本申请实施例中的特征扩增单元的特征扩增处理。该特征扩增单元可以用于替换现有卷积神经网络中的卷积层(例如图6所示的卷积层)。
在图7所示的特征扩增单元中，在输入特征图包括C个通道的情况下，若期望得到N个通道的输出特征图，可以先通过M个标准卷积核进行卷积处理，得到M个通道的候选特征图(如上述S520)，该M个标准卷积核中的每个卷积核包括C个通道；再利用N个矩阵对候选输出特征图的M个通道进行矩阵变换，得到N个通道的输出特征图(如上述S530)；其中，该N个矩阵中的每个矩阵的通道数可以小于M，N可以大于M，N为正整数。
在现有卷积神经网络中，同一个卷积层中可以有大量的标准卷积核，其中的很多卷积核的提取模式都是相似的，也就是说提取的特征都是类似的，导致得到输出特征图的冗余性很高。而在本申请中，该特征扩增单元基于少量标准卷积核进行卷积处理，并对得到的少量候选特征图的各个通道进行矩阵变量，可以有效减少各输出特征图之间的冗余性。
同时，该特征扩增单元中的标准卷积核的数量少于现有卷积神经网络，而且矩阵变换中使用的矩阵的通道数也小于标准卷积核，因此，有助于减少神经网络模型的计算量和参数量，从而降低图像分类处理的计算量和参数量。
在本申请中，上述矩阵变换中的所述N个矩阵可以是N个通道数为1的矩阵；或者上述矩阵变换中的所述N个矩阵也可以是N个通道数大于1的矩阵，例如，所述N个矩阵的通道数为2。
同时，S530中是根据N个矩阵对特征图(所述M个通道的候选输出特征图)进行矩阵变换，而通常卷积核也可以认为是矩阵，因此，所述N个矩阵也可以认为是N个卷积核。在图5中的后续描述中，所述N个矩阵可以理解为所述N个卷积核，所述N个卷积核也可以指对所述M个通道的候选输出特征图进行矩阵变换的所述N个矩阵。
需要说明的是，这里所说的N个卷积核(上述N个卷积核)与S520中的卷积核不同，S520中的卷积核为现有卷积神经网络中的标准卷积核，标准卷积核的通道数与进行卷积的输入特征图的通道数相同，而本申请中的上述N个卷积核中的每个卷积核的通道数可以小于进行卷积的输入特征图的通道数M(即所述M个通道的候选输出特征图)。例如，所述N个卷积核的通道数可以为1。
进一步地，由于矩阵变换中的所述N个矩阵(即上述N个卷积核)的通道数可以小 于所述候选输出特征图的通道数M，因此，可以减少特征扩增单元的计算量和参数量，在利用该特征扩增单元替换现有卷积神经网络中的卷积层(例如图6所示的卷积层)的情况下，有助于减少神经网络模型的计算量和参数量，从而降低图像分类处理的计算量和参数量。
在本申请中，根据矩阵变换中的所述N个矩阵的通道数不同，可以分为以下两种情况。
情况一：
可选地，上述矩阵变换中的所述N个矩阵的通道数可以为1。
可选地，所述N个矩阵可以分为M组卷积核，所述M组卷积核可以分别与所述候选输出特征图的M个通道对应。
可选地，所述M组卷积核可以与所述候选输出特征图的M个通道一一对应。
例如，所述M组卷积核中的第一组可以与所述候选输出特征图的M个通道中的第一个通道对应，所述M组卷积核中的第二组可以与所述候选输出特征图的M个通道中的第二个通道对应，……，所述M组卷积核中的第M组可以与所述候选输出特征图的M个通道中的第M个通道对应。
在本申请中，所述根据N个矩阵对所述M个候选输出特征图进行矩阵变换，得到N个通道的输出特征图，可以包括：根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行卷积，得到所述N个通道的输出特征图。
具体地，可以使用所述M组卷积核中的第一组，对所述候选输出特征图的M个通道中的第一个通道进行卷积，……，可以使用所述M组卷积核中的第M组，对所述候选输出特征图的M个通道中的第M个通道进行卷积。
例如，若所述M组卷积核中的第一组包括S个卷积核(该卷积核的通道数为1)，如图7所示的Φ 1,1至Φ 1,s，则可以使用这S个卷积核，对所述候选输出特征图的M个通道中的第一个通道进行卷积，得到S个输出特征图(或者也可以认为是一个通道数为S的输出特征图)。
所述M组卷积核中的其他组卷积核类似，也可以S个卷积核，这里不再赘述。
可选地，在上述矩阵变换中的所述N个矩阵的通道数为1的情况下，该特征扩增单元的浮点计算次数(floating point operations，FLOPs)r S可以近似用下述公式(1)表示：
Figure PCTCN2020105830-appb-000011
其中，S为所述M组卷积核中的每个组包括的卷积个数，C为所述特征扩增单元的输入特征图的通道数。
可选地，在S远小于C的情况，上述公式(1)可以近似用下述公式(2)表示：
Figure PCTCN2020105830-appb-000012
类似地，在上述矩阵变换中的所述N个矩阵的通道数为1的情况下，该特征扩增单元的参数量的压缩比r C可以近似用下述公式(3)表示：
Figure PCTCN2020105830-appb-000013
其中，S为所述M组卷积核中的每个组包括的卷积个数，C为所述特征扩增单元的输入特征图的通道数。
可选地，在S远小于C的情况，上述公式(3)可以近似用下述公式(4)表示：
Figure PCTCN2020105830-appb-000014
在本申请中，所述根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行卷积，得到所述N个通道的输出特征图，可以包括：根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行深度卷积(depthwise convolution)，得到所述N个通道的输出特征图。
具体地，所述深度卷积可以参考现有技术，这里不再赘述。
在本申请中，所述M组卷积核中的每一组的卷积核与所述M组卷积核中的其他组的卷积核相同。
也就是说，所述候选输出特征图的M个通道可以复用相同的卷积核进行卷积。
例如，所述M组卷积核中的第一组包括S个卷积核，比如图7所示的Φ 1,1至Φ 1,s，可以依次使用这S个卷积核，对所述候选输出特征图的M个通道中的第一个通道进行卷积；所述M组卷积核中的第二组包括的S个卷积核也可以是图7所示的Φ 1,1至Φ 1,s，可以依次使用这S个卷积核，对所述候选输出特征图的M个通道中的第二个通道进行卷积。
类似地，所述M组卷积核中的其他组包括的S个卷积核也可以是图7所示的Φ 1,1至Φ 1,s，这里不再赘述。
由于矩阵变换中的所述N个矩阵(即上述N个卷积核)的通道数为1(小于所述候选输出特征图的通道数M)，因此，可以减少特征扩增单元的计算量和参数量，降低图像分类处理的计算量和参数量。
情况二：
可选地，上述矩阵变换中的所述N个矩阵的通道数可以大于1。
以上述矩阵变换中的所述N个矩阵的通道数等于2为例，所述N个矩阵可以分为M/2组卷积核，所述M/2组卷积核可以分别与所述候选输出特征图的M个通道中的2个通道一一对应。
需要说明的是，本领域技术人员可以理解，在N个卷积核的通道数为2的情况下，若要根据N个卷积核对输入特征图(即所述M个通道的候选输出特征图)进行矩阵变换，则所述N个卷积核中的每个卷积核的通道数需要与输入特征图的通道数一致(或者说相同)。
此时，所述M个通道的候选输出特征图相当于划分为M/2个通道数为2的特征图，与所述N个卷积核进行卷积，相应地，所述N个卷积核也相当于划分为M/2组卷积核。应理解，这里所说的“划分”只是便于理解进行的解释性描述，在实际中可能并不存在划分的操作。
例如，所述M/2组卷积核中的第一组可以与所述候选输出特征图的M个通道中的第一个通道及第二个通道对应，所述M/2组卷积核中的第二组可以与所述候选输出特征图的M个通道中的第三个通道及第四个通道对应，……，所述M/2组卷积核中的第M/2组可以与所述候选输出特征图的M个通道中的第M-1个通道及第M个通道对应。
可选地，可以根据所述M/2组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行卷积，得到所述N个通道的输出特征图。
例如，可以使用所述M/2组卷积核中的第一组，对所述候选输出特征图的M个通道中的第一个通道及第二个通道进行卷积，……，可以使用所述M/2组卷积核中的第M/2组，对所述候选输出特征图的M个通道中的第M-1个通道及第M个通道进行卷积。
可选地，所述M/2组卷积核中的每一组的卷积核与所述M/2组卷积核中的其他组的卷积核相同。也就是说，所述候选输出特征图的M个通道可以复用相同的卷积核进行卷积。
在本申请实施例中，矩阵变换中的所述N个矩阵的通道数大于2的情况与上述矩阵变换中的所述N个矩阵的通道数等于2的实施例类似，这里不再赘述。
由于矩阵变换中的所述N个矩阵(即上述N个卷积核)的通道数可以小于所述候选输出特征图的通道数M，因此，可以减少特征扩增单元的计算量和参数量，降低图像分类处理的计算量和参数量。
可选地，矩阵变换中的所述N个矩阵(即上述N个卷积核)的通道数可以等于所述候选输出特征图的通道数M。
在本申请中，所述根据所述输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果，可以包括：对所述候选输出特征图和所述输出特征图进行特征拼接，得到特征拼接特征图，所述特征拼接特征图的通道数为M+N；根据所述特征拼接特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
上述特征拼接是指所述候选输出特征图和所述输出特征图在深度方向上组成新的特征图，即上述特征拼接特征图。
例如，所述候选输出特征图的通道数为M，所述输出特征图的通道数为N，所述候选输出特征图和所述输出特征图可以进行特征拼接，得到一个通道数为M+N的特征拼接特征图。
特征拼接可以通过恒等特征映射的方式，可以将更多的细节(或特征)引入输出特征图，同时，这种恒等映射并不会引入额外的参数或者计算量，因此可以在不增加参数量和计算量的情况下，提升图像分类的效果。
S540，根据所述输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
图8示出了本申请另一个实施例提供的图像分类方法800的示意性流程图，该方法可以由能够进行图像分类装置执行，例如，该方法可以由图4中的终端设备执行。
S810，获取待处理图像的输入特征图。
当图5所示的方法由图4中的终端设备执行时，该待处理图像可以是终端设备通过摄像头拍摄到的图像，或者，该待处理图像还可以是从终端设备内部获得的图像(例如，终端设备的相册中存储的图像，或者终端设备从云端获取的图像)。
或者，该待处理图像的输入特征图可以是卷积神经网络中的其他层处理后得到的特征图。应理解，这里所说的卷积神经网络中的其他层是指卷积神经网络中的一个层，例如，该其他层可以是卷积神经网络中的输入层、卷积层、池化层或全连接层中的一个。
S820，根据神经网络的M个第一卷积核对所述输入特征图进行卷积处理，得到M个通道的第一候选输出特征图，M为正整数。
其中，所述M个第一卷积核可以为现有卷积神经网络中的标准卷积核。
S830，根据K个第一矩阵对所述M个第一候选输出特征图进行矩阵变换，得到K个通道的第一输出特征图。
其中，所述K个第一矩阵中的每个矩阵的通道数小于M，K大于M，K为正整数。
可选地，上述S820和S830可以是图5方法500中的一个特征扩增单元(feature in feature，FiF)。
为便于理解，下述实施例中也可以将S820和S830称为第一特征扩增单元。
S840，根据神经网络的P个第二卷积核对所述第一输出特征图进行卷积处理，得到P个通道的第二候选输出特征图，P为正整数。
其中，所述P个第二卷积核可以为现有卷积神经网络中的标准卷积核。
可选地，所述根据神经网络的P个第二卷积核对所述第一输出特征图进行卷积处理，得到P个通道的第二候选输出特征图，可以包括：对所述第一输出特征图进行深度卷积，得到深度卷积特征图；根据所述P个第二卷积核对所述深度卷积特征图进行卷积处理，得到所述第二候选输出特征图。
可选地，所述对所述第一输出特征图进行深度卷积，得到深度卷积特征图，可以包括：对所述第一输出特征图进行步幅大于1的深度卷积，得到所述深度卷积特征图。
S850，根据N个第二矩阵对所述P个第二候选输出特征图进行矩阵变换，得到N个通道的第二输出特征图。
其中，所述N个第二矩阵中的每个矩阵的通道数小于P，N大于P，N为正整数。
可选地，上述S840和S850也可以是图5方法500中的一个特征扩增单元(feature in feature，FiF)。
为便于理解，下述实施例中也可以将S840和S850称为第二特征扩增单元。
S860，根据所述第二输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
可选地，所述根据所述第二输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果，可以包括：对所述输入特征图和所述第二输出特征图进行残差连接，得到残差连接特征图；根据所述残差连接特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
在本申请中，上述S820至S850可以是本申请实施例中的纺锤模块(spindle block)，该纺锤模块可以用于替换现有卷积神经网络模型中的一个模块(block)，比如，现有卷积神经网络模型中的一个模块可以包括两个卷积层。
为便于理解，下述实施例中的纺锤模块都是指上述图8方法800(S820至S850)中的所述纺锤模块。
也就是说，该纺锤模块可以包括上述第一特征扩增单元和上述第二特征扩增单元。
纺锤模块可以由至少两个特征扩增单元(比如图5中所示的特征扩增单元)构成，该特征扩增单元使用少量标准卷积核进行卷积处理，得到少量的候选特征图，并对这些少量的候选特征图进行矩阵变换以得到所需的输出特征图，可以减少输出特征图之间的冗余性，有助于降低图像分类处理的计算量和参数量。下面结合图9和图10对上述S820至S850中的纺锤模块进行详细描述。
在本申请中，根据纺锤模块的步长(stride)不同，可以分为以下两种情况。
情况一：
图9所示的是本申请实施例中的步长为1的纺锤模块。该纺锤可以由至少两个特征扩增单元(比如图5中所示的特征扩增单元)构成。
对于步长为1的纺锤模块，输入特征图的尺寸(宽和高)与输出特征图的尺寸相同。
例如，上述纺锤模块的输入特征图为所述待处理图像的输入特征图，上述纺锤模块的输出特征图为N个通道的第二输出特征图，若所述输入特征图的尺寸为A*B，则所述第二输出特征图的尺寸也为A*B。
可选地，在所述纺锤模块中，在所述第一特征扩增单元和所述第二特征扩增单元之间还可以进行深度卷积。可选地，所述深度卷积的步长可以为1。
可选地，在所述纺锤模块的步长为1的情况下，所述纺锤模块的输入特征图的尺寸(宽和高)与输出特征图的尺寸相同，因此，还可以对所述纺锤模块进行残差连接，即对所述输入特征图和所述第二输出特征图进行残差连接。
此时，可以对所述输入特征图和所述第二输出特征图进行残差连接，得到残差连接特征图；相应地，可以根据所述残差连接特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
关于残差连接的详细描述可以参考现有技术，这里不再赘述。
通过残差连接可以将更多的细节(或特征)引入输出特征图，而残差连接并不会引入额外的参数或者计算量，因此，可以在不增加参数量和计算量的情况下，提升图像分类的效果。
情况二：
图10所示的是本申请实施例中的步长大于1的纺锤模块。该纺锤可以由至少两个特征扩增单元(比如图5中所示的特征扩增单元)构成。
对于步长大于1的纺锤模块，输出特征图的尺寸(宽和高)小于输入特征图的尺寸。
以纺锤模块的步长等于2为例进行说明，此时，输出特征图的尺寸(宽和高)为输入特征图的尺寸的一半。
例如，上述纺锤模块的输入特征图为所述待处理图像的输入特征图，上述纺锤模块的输出特征图为N个通道的第二输出特征图，若所述输入特征图的尺寸为A*B，则所述第二输出特征图的尺寸也为(A/2)*(B/2)。
可选地，在所述纺锤模块中，在所述第一特征扩增单元和所述第二特征扩增单元之间还可以进行深度卷积。可选地，所述深度卷积的步长可以大于1。
在本申请实施例中，所述纺锤模块的输出特征图的通道数可以为N(即N个通道的第二输出特征图)，所述第一特征扩增单元的输出特征图的通道数可以为K(即K个通道的第一输出特征图)，可以使所述第一特征扩增单元的输出的通道数K大于N，相应地，再通过所述第二特征扩增单元将通道数K降为N。
也就是说，在所述纺锤模块中，可以通过所述第一特征扩增单元提高通道数，再通过所述第二特征扩增单元降低通道数，以满足所述纺锤模块输出的通道数。
例如，所述纺锤模块的输出特征图的通道数为100(即100个通道的第二输出特征图)，所述第一特征扩增单元的输出特征图的通道数可以为1000(即1000个通道的第一输出特征图)，此时，所述第一特征扩增单元输出的通道数1000大于所述纺锤模块输出的通道 数100，相应地，可以再通过所述第二特征扩增单元将通道数1000降为100。
在本申请实施例中，通过所述第一特征扩增单元提高通道数，可以提取更多的特征，从而能够提升图像分类的效果。
进一步地，本申请实施例中的特征扩增单元可以通过少量标准卷积核及矩阵变换实现现有卷积神经网络中的卷积处理，可以有效减少各输出特征图之间的冗余性，减少神经网络模型的计算量和参数量，从而降低图像分类处理的计算量和参数量，因此，本申请实施例中的纺锤模块可以不增加参数量和计算量(或者减少参数量和计算量)的情况下，提升图像分类的效果。
图11是本申请实施例提供的一个神经网络的示意性框图。图11所示的神经网络可以用于实现图8所示的图像分类方法。
图11中的神经网络可以包括一个或多个图8方法800中的纺锤模块，该纺锤模块可以用于替换现有卷积神经网络模型中的一个模块(block)，比如，现有卷积神经网络模型中的一个模块可以包括两个卷积层。
该纺锤模块可以包括至少两个图5所示的特征扩增单元，例如，该纺锤模块可以如图8方法800中所述，包括两个特征扩增单元：第一特征扩增单元和第二特征扩增单元。其中，一个特征扩增单元可以用于替换现有卷积神经网络模型中的一个卷积层。
在图11所示的神经网络中，还可以包括卷积层、池化层或全连接层等，本申请对此并不限定。
根据图11所示的神经网络结构，本申请实施例提出一种高效的神经网络模型HWNet。HWNet如图11中神经网络所示，包括多个纺锤模块，其中，每个纺锤模块包括特征扩增模块，HWNet的网络结构可以参考现有神经网络的设计准则。例如，在现有神经网络的设计中，随着特征图尺寸的逐渐下降，特征图的通道数逐渐增加。HWNet的具体结构可以如下述表1所示。
表1本申请中的HWNet的网络结构
Figure PCTCN2020105830-appb-000015
Figure PCTCN2020105830-appb-000016
如表1所示，HWNet的第一层是带有16个标准卷积核的卷积层，然后是12个输入特征图的通道数逐渐增加的纺锤模块，这些纺锤模块组被分为5个阶段，每一阶段内的特征图大小相同。
下述表2是对HWNet和现有的几个神经网络模型，在ImageNet数据集上进行图像分类的测试实验数据。
表2本申请中的HWNet和现有的几个神经网络模型的测试数据
Figure PCTCN2020105830-appb-000017
其中，MobileNet是谷歌公司(Google)提出的一种卷积神经网络模型，ShuffleNet是旷视科技公司提出的一种为移动终端设备而设计的卷积神经网络模型，IGCV3是交错低秩分组卷积。
从上表2可以看出，相比现有的几个神经网络模型，本申请实施例中提出的HWNet在参数更少、计算更快的情况下，模型的精度更高。
图12是本申请实施例的图像分类装置的硬件结构示意图。图12所示的图像分类装置4000包括存储器4001、处理器4002、通信接口4003以及总线4004。其中，存储器4001、 处理器4002、通信接口4003通过总线4004实现彼此之间的通信连接。
存储器4001可以是只读存储器(read only memory，ROM)，静态存储设备，动态存储设备或者随机存取存储器(random access memory，RAM)。存储器4001可以存储程序，当存储器4001中存储的程序被处理器4002执行时，处理器4002和通信接口4003用于执行本申请实施例的图像分类方法的各个步骤。
处理器4002可以采用通用的中央处理器(central processing unit，CPU)，微处理器，应用专用集成电路(application specific integrated circuit，ASIC)，图形处理器(graphics processing unit，GPU)或者一个或多个集成电路，用于执行相关程序，以实现本申请实施例的图像分类装置中的单元所需执行的功能，或者执行本申请方法实施例的图像分类方法。
处理器4002还可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，本申请实施例的图像分类方法的各个步骤可以通过处理器4002中的硬件的集成逻辑电路或者软件形式的指令完成。
上述处理器4002还可以是通用处理器、数字信号处理器(digital signal processing，DSP)、ASIC、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。上述通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器4001，处理器4002读取存储器4001中的信息，结合其硬件完成本申请实施例的图像分类装置中包括的单元所需执行的功能，或者执行本申请方法实施例的图像分类方法。
通信接口4003使用例如但不限于收发器一类的收发装置，来实现装置4000与其他设备或通信网络之间的通信。例如，可以通过通信接口4003获取待处理图像。
总线4004可包括在装置4000各个部件(例如，存储器4001、处理器4002、通信接口4003)之间传送信息的通路。
图13是本申请实施例的神经网络训练装置5000的硬件结构示意图。与上述装置4000类似，图13所示的神经网络训练装置5000包括存储器5001、处理器5002、通信接口5003以及总线5004。其中，存储器5001、处理器5002、通信接口5003通过总线5004实现彼此之间的通信连接。
存储器5001可以存储程序，当存储器5001中存储的程序被处理器5002执行时，处理器5002用于执行训练本申请实施例的图像分类装置的训练方法的各个步骤。
处理器5002可以采用通用的CPU，微处理器，ASIC，GPU或者一个或多个集成电路，用于执行相关程序，以实现训练本申请实施例的图像分类装置的训练方法。
处理器5002还可以是一种集成电路芯片，具有信号的处理能力。在实现训练过程中，本申请实施例的图像分类装置的训练方法的各个步骤可以通过处理器5002中的硬件的集成逻辑电路或者软件形式的指令完成。
应理解，通过图13所示的神经网络训练装置5000对图像分类装置进行训练，训练得到的图像分类装置就可以用于执行本申请实施例的图像分类方法了。具体地，通过装置 5000对神经网络进行训练能够得到图5或图8所示的方法中的神经网络。
具体地，图13所示的装置可以通过通信接口5003从外界获取训练数据以及待训练的图像分类装置，然后由处理器根据训练数据对待训练的图像分类装置进行训练。
应注意，尽管上述装置4000和装置5000仅仅示出了存储器、处理器、通信接口，但是在具体实现过程中，本领域的技术人员应当理解，装置4000和装置5000还可以包括实现正常运行所必须的其他器件。同时，根据具体需要，本领域的技术人员应当理解，装置4000和装置5000还可包括实现其他附加功能的硬件器件。此外，本领域的技术人员应当理解，装置4000和装置5000也可仅仅包括实现本申请实施例所必须的器件，而不必包括图12和图13中所示的全部器件。
应理解，本申请实施例中的处理器可以为中央处理单元(central processing unit，CPU)，该处理器还可以是其他通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
还应理解，本申请实施例中的存储器可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的随机存取存储器(random access memory，RAM)可用，例如静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data rate SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。
上述实施例，可以全部或部分地通过软件、硬件、固件或其他任意组合来实现。当使用软件实现时，上述实施例可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令或计算机程序。在计算机上加载或执行所述计算机指令或计算机程序时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以为通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集合的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质。半导体介质可以是固态硬盘。
应理解，本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况，其中A,B可以是单数或者复数。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系，但也可能表示的是一种“和/或”的关系，具体可参考前后文进行理解。
本申请中，“至少一个”是指一个或者多个，“多个”是指两个或两个以上。“以下至少一项(个)”或其类似表达，是指的这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a,b,或c中的至少一项(个)，可以表示：a,b,c,a-b,a-c,b-c,或a-b-c，其中a,b,c可以是单个，也可以是多个。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟 悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (21)

  1. 一种图像分类方法，其特征在于，包括：
    获取待处理图像的输入特征图；
    根据神经网络的M个卷积核对所述输入特征图进行卷积处理，得到M个通道的候选输出特征图，M为正整数；
    根据N个矩阵对所述候选输出特征图的M个通道进行矩阵变换，得到N个通道的输出特征图，其中，所述N个矩阵中的每个矩阵的通道数小于M，N大于M，N为正整数；
    根据所述输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
  2. 根据权利要求1所述的方法，其特征在于，所述N个矩阵包括M组卷积核，所述M组卷积核分别与所述候选输出特征图的M个通道对应；
    所述根据N个矩阵对所述M个候选输出特征图进行矩阵变换，得到N个通道的输出特征图，包括：
    根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行卷积，得到所述N个通道的输出特征图。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行卷积，得到所述N个通道的输出特征图，包括：
    根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行深度卷积，得到所述N个通道的输出特征图。
  4. 根据权利要求2所述的方法，其特征在于，所述M组卷积核中的每一组的卷积核与所述M组卷积核中的其他组的卷积核相同。
  5. 根据权利要求1至4中任一项所述的方法，其特征在于，所述根据所述输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果，包括：
    对所述候选输出特征图和所述输出特征图进行特征拼接，得到特征拼接特征图，所述特征拼接特征图的通道数为M+N；
    根据所述特征拼接特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
  6. 一种图像分类方法，其特征在于，包括：
    获取待处理图像的输入特征图；
    根据神经网络的M个第一卷积核对所述输入特征图进行卷积处理，得到M个通道的第一候选输出特征图，M为正整数；
    根据K个第一矩阵对所述M个第一候选输出特征图进行矩阵变换，得到K个通道的第一输出特征图，其中，所述K个第一矩阵中的每个矩阵的通道数小于M，K大于M，K为正整数；
    根据神经网络的P个第二卷积核对所述第一输出特征图进行卷积处理，得到P个通道的第二候选输出特征图，P为正整数；
    根据N个第二矩阵对所述P个第二候选输出特征图进行矩阵变换，得到N个通道的 第二输出特征图，其中，所述N个第二矩阵中的每个矩阵的通道数小于P，N大于P，N为正整数；
    根据所述第二输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
  7. 根据权利要求6所述的方法，其特征在于，所述根据神经网络的P个第二卷积核对所述第一输出特征图进行卷积处理，得到P个通道的第二候选输出特征图，包括：
    对所述第一输出特征图进行深度卷积，得到深度卷积特征图；
    根据所述P个第二卷积核对所述深度卷积特征图进行卷积处理，得到所述第二候选输出特征图。
  8. 根据权利要求7所述的方法，其特征在于，所述对所述第一输出特征图进行深度卷积，得到深度卷积特征图，包括：
    对所述第一输出特征图进行步幅大于1的深度卷积，得到所述深度卷积特征图。
  9. 根据权利要求6或7所述的方法，其特征在于，所述根据所述第二输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果，包括：
    对所述输入特征图和所述第二输出特征图进行残差连接，得到残差连接特征图；
    根据所述残差连接特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
  10. 一种图像分类装置，其特征在于，包括：
    获取单元，用于获取待处理图像的输入特征图；
    卷积单元，用于根据神经网络的M个卷积核对所述输入特征图进行卷积处理，得到M个通道的候选输出特征图，M为正整数；
    矩阵变换单元，用于根据N个矩阵对所述候选输出特征图的M个通道进行矩阵变换，得到N个通道的输出特征图，其中，所述N个矩阵中的每个矩阵的通道数小于M，N大于M，N为正整数；
    分类单元，用于根据所述输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
  11. 根据权利要求10所述的装置，其特征在于，所述N个矩阵包括M组卷积核，所述M组卷积核分别与所述候选输出特征图的M个通道对应；
    所述矩阵变换单元具体用于：根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行卷积，得到所述N个通道的输出特征图。
  12. 根据权利要求11所述的装置，其特征在于，所述矩阵变换单元具体用于：
    根据所述M组卷积核中的每一组卷积核，对所述候选输出特征图的M个通道中对应的通道进行深度卷积，得到所述N个通道的输出特征图。
  13. 根据权利要求11所述的装置，其特征在于，所述M组卷积核中的每一组的卷积核与所述M组卷积核中的其他组的卷积核相同。
  14. 根据权利要求10至13中任一项所述的装置，其特征在于，所述分类单元具体用于：
    对所述候选输出特征图和所述输出特征图进行特征拼接，得到特征拼接特征图，所述特征拼接特征图的通道数为M+N；
    根据所述特征拼接特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
  15. 一种图像分类装置，其特征在于，包括：
    获取单元，用于获取待处理图像的输入特征图；
    第一卷积单元，用于根据神经网络的M个第一卷积核对所述输入特征图进行卷积处理，得到M个通道的第一候选输出特征图，M为正整数；
    第一矩阵变换单元，用于根据K个第一矩阵对所述M个第一候选输出特征图进行矩阵变换，得到K个通道的第一输出特征图，其中，所述K个第一矩阵中的每个矩阵的通道数小于M，K大于M，K为正整数；
    第二卷积单元，用于根据神经网络的P个第二卷积核对所述第一输出特征图进行卷积处理，得到P个通道的第二候选输出特征图，P为正整数；
    第二矩阵变换单元，用于根据N个第二矩阵对所述P个第二候选输出特征图进行矩阵变换，得到N个通道的第二输出特征图，其中，所述N个第二矩阵中的每个矩阵的通道数小于P，N大于P，N为正整数；
    分类单元，用于根据所述第二输出特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
  16. 根据权利要求15所述的装置，其特征在于，所述图像分类装置还包括深度卷积单元，用于：
    对所述第一输出特征图进行深度卷积，得到深度卷积特征图；
    所述第二卷积单元具体用于：根据所述P个第二卷积核对所述深度卷积特征图进行卷积处理，得到所述第二候选输出特征图。
  17. 根据权利要求16所述的装置，其特征在于，所述深度卷积单元具体用于：
    对所述第一输出特征图进行步幅大于1的深度卷积，得到所述深度卷积特征图。
  18. 根据权利要求15或16所述的装置，其特征在于，所述图像分类装置还包括残差连接单元，用于：
    对所述输入特征图和所述第二输出特征图进行残差连接，得到残差连接特征图；
    所述分类单元具体用于：根据所述残差连接特征图对所述待处理图像进行分类，得到所述待处理图像的分类结果。
  19. 一种图像分类装置，其特征在于，包括处理器和存储器，所述存储器用于存储程序指令，所述处理器用于调用所述程序指令来执行权利要求1-5或6-9中任一项所述的方法。
  20. 一种计算机可读存储介质，其特征在于，所述计算机可读介质存储用于设备执行的程序代码，该程序代码包括用于执行如权利要求1-5或6-9中任一项所述的方法。
  21. 一种芯片，其特征在于，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，以执行如权利要求求1-5或6-9中任一项所述的方法。
PCT/CN2020/105830 2019-07-30 2020-07-30 图像分类方法及装置 WO2021018245A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20848554.0A EP4006776A4 (en) 2019-07-30 2020-07-30 METHOD AND APPARATUS FOR CLASSIFYING IMAGES
US17/587,689 US20220157041A1 (en) 2019-07-30 2022-01-28 Image classification method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910697287.0A CN111797882A (zh) 2019-07-30 2019-07-30 图像分类方法及装置
CN201910697287.0 2019-07-30

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/587,689 Continuation US20220157041A1 (en) 2019-07-30 2022-01-28 Image classification method and apparatus

Publications (1)

Publication Number Publication Date
WO2021018245A1 true WO2021018245A1 (zh) 2021-02-04

Family

ID=72805479

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/105830 WO2021018245A1 (zh) 2019-07-30 2020-07-30 图像分类方法及装置

Country Status (4)

Country Link
US (1) US20220157041A1 (zh)
EP (1) EP4006776A4 (zh)
CN (1) CN111797882A (zh)
WO (1) WO2021018245A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022179599A1 (zh) * 2021-02-27 2022-09-01 华为技术有限公司 一种感知网络及数据处理方法

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112700362B (zh) * 2020-12-28 2023-07-25 北京达佳互联信息技术有限公司 一种图像处理方法、装置、电子设备和存储介质
CN113033324B (zh) * 2021-03-03 2024-03-08 广东省地质环境监测总站 一种地质灾害前兆因子识别方法、装置，电子设备及存储介质
CN117223008A (zh) * 2021-07-30 2023-12-12 华为技术有限公司 计算机系统和数据处理方法
CN113673422A (zh) * 2021-08-19 2021-11-19 苏州中科先进技术研究院有限公司 一种宠物种类识别方法及识别系统
CN117620448B (zh) * 2024-01-25 2024-04-16 东莞市昂图智能科技有限公司 激光雕刻机的加工控制方法、装置、设备及存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180365580A1 (en) * 2017-06-15 2018-12-20 Microsoft Technology Licensing, Llc Determining a likelihood of a user interaction with a content element
CN109670506A (zh) * 2018-11-05 2019-04-23 中国科学院计算技术研究所 基于克罗内克卷积的场景分割方法和系统
CN110020682A (zh) * 2019-03-29 2019-07-16 北京工商大学 一种基于小样本学习的注意力机制关系对比网络模型方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180365580A1 (en) * 2017-06-15 2018-12-20 Microsoft Technology Licensing, Llc Determining a likelihood of a user interaction with a content element
CN109670506A (zh) * 2018-11-05 2019-04-23 中国科学院计算技术研究所 基于克罗内克卷积的场景分割方法和系统
CN110020682A (zh) * 2019-03-29 2019-07-16 北京工商大学 一种基于小样本学习的注意力机制关系对比网络模型方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4006776A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022179599A1 (zh) * 2021-02-27 2022-09-01 华为技术有限公司 一种感知网络及数据处理方法

Also Published As

Publication number Publication date
US20220157041A1 (en) 2022-05-19
EP4006776A4 (en) 2022-10-05
EP4006776A1 (en) 2022-06-01
CN111797882A (zh) 2020-10-20

Similar Documents

Publication Publication Date Title
WO2020216227A9 (zh) 图像分类方法、数据处理方法和装置
WO2020238293A1 (zh) 图像分类方法、神经网络的训练方法及装置
WO2021043273A1 (zh) 图像增强方法和装置
WO2021043168A1 (zh) 行人再识别网络的训练方法、行人再识别方法和装置
WO2021120719A1 (zh) 神经网络模型更新方法、图像处理方法及装置
WO2021018245A1 (zh) 图像分类方法及装置
WO2021013095A1 (zh) 图像分类方法、图像分类模型的训练方法及其装置
WO2020177607A1 (zh) 图像去噪方法和装置
WO2021043112A1 (zh) 图像分类方法以及装置
WO2020192736A1 (zh) 物体识别方法及装置
WO2021057056A1 (zh) 神经网络架构搜索方法、图像处理方法、装置和存储介质
WO2022001805A1 (zh) 一种神经网络蒸馏方法及装置
WO2022052601A1 (zh) 神经网络模型的训练方法、图像处理方法及装置
WO2021155792A1 (zh) 一种处理装置、方法及存储介质
WO2021018251A1 (zh) 图像分类方法及装置
WO2021022521A1 (zh) 数据处理的方法、训练神经网络模型的方法及设备
WO2021227787A1 (zh) 训练神经网络预测器的方法、图像处理方法及装置
WO2022001372A1 (zh) 训练神经网络的方法、图像处理方法及装置
CN110222718B (zh) 图像处理的方法及装置
WO2021073311A1 (zh) 图像识别方法、装置、计算机可读存储介质及芯片
WO2022179606A1 (zh) 一种图像处理方法及相关装置
WO2024067884A1 (zh) 一种数据处理方法及相关装置
WO2022156475A1 (zh) 神经网络模型的训练方法、数据处理方法及装置
WO2021136058A1 (zh) 一种处理视频的方法及装置
WO2021057690A1 (zh) 构建神经网络的方法与装置、及图像处理方法与装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20848554

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020848554

Country of ref document: EP

Effective date: 20220228