WO2020216227A9

WO2020216227A9 - Image classification method and apparatus, and data processing method and apparatus

Info

Publication number: WO2020216227A9
Application number: PCT/CN2020/086015
Authority: WO
Inventors: 韩凯; 王云鹤; 许春景
Original assignee: 华为技术有限公司
Priority date: 2019-04-24
Filing date: 2020-04-22
Publication date: 2020-11-26
Also published as: WO2020216227A1; CN110188795A; CN110188795B

Abstract

Provided in the present application are an image classification method and apparatus, which relate to the field of artificial intelligence, and specifically relate to the field of computing vision. The image classification method comprises: acquiring a convolution kernel parameter of a reference convolution kernel of a neural network and a mask tensor of the neural network, and performing a Hadamard product operation on the reference convolution kernel of the neural network and the mask tensor corresponding to the reference convolution kernel to obtain a plurality of sub-convolution kernels; and according to the plurality of sub-convolution kernels, performing convolution processing on an image to be processed, and according to a convolution feature map finally obtained from convolution, classifying the image to be processed to obtain a classification result of the image to be processed. Due to the fact that the mask tensor occupies a smaller storage space relative to the convolution kernel, some devices with limited storage resources can also deploy a neural network including the reference convolution kernel and the mask tensor, such that image classification is realized.

Description

Image classification method, data processing method and device

This application claims the priority of the Chinese patent application filed with the Chinese Patent Office on April 24, 2019, the application number is 201910335678.8, and the application name is "Image Classification Method, Data Processing Method and Device", the entire content of which is incorporated herein by reference Applying.

Technical field

This application relates to the field of artificial intelligence, and more specifically, to an image classification method, data processing method and device.

Background technique

With the rapid development of artificial intelligence technology, the processing power of neural networks has become stronger and stronger, and the parameters contained in neural networks have become more and more. This makes these neural networks often take up a lot of time when they are deployed or applied. Storage space to store the parameters of the neural network. This affects the deployment and application of neural networks on some devices with limited storage resources.

Take the neural network for image classification as an example. Many neural networks used for image classification (especially some neural networks with more complex network structure and more powerful functions) contain many parameters, so it is difficult to deploy to some storage Devices with limited space (for example, mobile phones, cameras, smart homes) affect the application of neural networks. Therefore, how to reduce the storage overhead of neural networks is a problem that needs to be solved.

Summary of the invention

This application provides an image classification method, data processing method and device, so that a neural network can be deployed on some devices with limited storage resources and perform image classification processing.

In the first aspect, an image classification method is provided. The method includes: obtaining convolution kernel parameters of M reference convolution kernels of a neural network; obtaining N groups of mask tensors of the neural network; Each reference convolution kernel in and each reference convolution kernel performs the Hadamard product operation on the corresponding set of mask tensors in the N groups of mask tensors to obtain multiple subconvolution kernels; according to the multiple subconvolution kernels respectively Perform convolution processing on the image to be processed to obtain multiple convolution feature maps; classify the image to be processed according to the multiple convolution feature maps to obtain the classification result of the image to be processed.

Among them, the above M and N are both positive integers, each of the above N groups of mask tensors is composed of multiple mask tensors, and the number of bits occupied by the elements in the N groups of mask tensors is less than M benchmarks. The number of bits occupied by elements in the convolution kernel parameters in the convolution kernel. Each reference convolution kernel in the M reference convolution kernels corresponds to a group of mask tensors in the N groups of mask tensors.

In addition, the above-mentioned reference convolution kernel is a relatively basic convolution kernel used to obtain or obtain other sub-convolution kernels of the neural network, and the reference convolution kernel may also be called a basic convolution kernel.

The above-mentioned image classification method can be executed by an image classification device, which can be an electronic device with image processing function, the electronic device can be a mobile terminal (for example, a smart phone), a computer, a personal digital assistant, a wearable device, a vehicle Equipment, Internet of Things equipment or other equipment capable of image processing.

Optionally, the above method further includes: acquiring the image to be processed.

The foregoing image to be processed may be an image or picture to be classified.

The image to be processed can be obtained from a camera or an album.

Specifically, when the above method is executed by an image classification device, the picture can be obtained through the camera of the image classification device (for example, real-time shooting pictures), or the to-be-processed album can be obtained from the album stored in the internal storage space of the image classification device. image.

Optionally, the convolution kernel parameters of the aforementioned M reference convolution kernels are stored in a register.

Optionally, the foregoing acquiring the convolution kernel parameters of the M reference convolution kernels of the neural network includes: acquiring (reading) the convolution kernel parameters of the M reference convolution kernels of the neural network from a register.

When the convolution kernel parameters of the M reference convolution kernels are stored in the registers, the convolution kernel parameters of the M reference convolution kernels can be obtained from the register relatively quickly (as opposed to obtaining from external storage, from the register The acquisition speed will be faster), which can improve the execution speed of the above method to a certain extent.

Optionally, the above N groups of mask tensors are stored in a register.

Optionally, the foregoing obtaining N groups of mask tensors of the neural network includes: obtaining (reading) the N groups of mask tensors of the neural network from a register.

When the above N groups of mask tensors are stored in registers, N groups of mask tensors can be obtained from the register relatively quickly (compared to obtaining from external storage, the speed of obtaining parameters from the register will be faster), Can improve the execution speed of the above method to a certain extent.

The aforementioned register may specifically be a weight memory.

In this application, when the image to be processed is classified, it is only necessary to obtain the convolution kernel parameters of the reference convolution kernel and the corresponding mask tensor from the storage space, and then the reference convolution kernel and the corresponding mask tensor can be used. Realize the convolution processing of the image to be processed, and then realize the classification of the image to be processed, without having to obtain the parameters of each convolution kernel in the neural network, which can reduce the storage overhead generated during the deployment of the neural network, so that the neural network can be deployed in some Perform image classification processing on devices with limited storage resources.

Specifically, compared to the elements in the parameters of the reference convolution kernel, the elements in the mask tensor occupy less storage space. Therefore, the method of combining the reference convolution kernel and the mask tensor to obtain the subconvolution kernel , The number of convolution kernel parameters is reduced, and the compression of convolution kernel parameters is realized, so that the neural network can be deployed on some devices with limited storage resources to perform image classification tasks.

Optionally, each group of mask tensors in the above N groups of mask tensors includes T mask tensors, and each reference convolution kernel in the M reference convolution kernels, and each reference convolution kernel is masked in the N groups The corresponding set of mask tensors in the code tensor are subjected to the Hadamard product operation to obtain multiple subconvolution kernels, including: each reference convolution kernel in the M reference convolution kernels, and each reference convolution kernel is in N The corresponding group of mask tensors in the group mask tensor are subjected to the Hadamard product operation to obtain M×T subconvolution kernels.

Specifically, for a reference convolution kernel, the reference convolution kernel and T mask tensors in the corresponding set of mask tensors can get T subconvolution kernels by performing the Hadamard product operation, then, for M For the reference convolution kernel, by performing the Hadamard product operation with the corresponding mask tensor, a total of M*T subconvolution kernels can be obtained.

Optionally, the foregoing classifying the image to be processed according to multiple convolution feature maps to obtain the classification result of the image to be processed includes: splicing multiple convolution feature maps to obtain the target convolution feature map; according to the target convolution feature Figure classifies the image lines to be processed, and obtains the classification result of the image to be processed.

The width and height of the above multiple convolution features should be the same. The above splicing of multiple convolution feature maps is essentially to superimpose the number of channels of the multiple convolution feature maps to obtain a channel number of multiple convolutions. The target convolution feature map of the sum of the number of channels in the feature map.

For example, there are 3 convolution feature maps in total, and the sizes of these 3 convolution feature maps are c ₁ ×d ₁ ×d ₂ , c ₂ ×d ₁ ×d ₂ , and c ₃ ×d ₁ ×d ₂ , then , The size of the target feature map obtained by splicing the three convolutional feature maps is c×d ₁ ×d ₂ , where c=c ₁ +c ₂ +c ₃ .

It should be understood that each of the above M reference convolution kernels corresponds to a set of mask tensors in the N sets of mask tensors, and one set of mask tensors in the N sets of mask tensors may correspond to the foregoing M sets of mask tensors. One or more convolution kernels in the reference convolution kernel.

With reference to the first aspect, in some implementations of the first aspect, N is less than M, and at least two of the M reference convolution kernels correspond to a group of mask tensors in the N group of mask tensors.

When N is less than M, there will be a situation in which multiple reference convolution kernels jointly correspond to the same set of mask tensors (sharing a set of mask tensors). This situation can be referred to as the case of mask tensor sharing. In the case of mask tensor sharing, when the Hadamard product operation is performed, part of the reference convolution kernel will be operated with the same mask tensor to obtain the subconvolution kernel, which can further reduce the number of mask tensors. Can further reduce storage overhead.

Further, the above N=1, only a set of mask tensors need to be saved at this time, and the effect of saving storage overhead is more obvious.

In combination with the first aspect, in some implementations of the first aspect, N=M, and M reference convolution kernels correspond to N sets of mask tensors one-to-one.

Each reference convolution kernel corresponds to a group of mask tensors, and each group of mask tensors also corresponds to a reference convolution kernel. There is a one-to-one correspondence between the reference convolution kernel and the mask tensor group. This situation can be This is called the case of mask tensor independence. In this case, the mask tensor group is not shared between the reference convolution kernels. Compared with the case of mask tensor sharing, although the amount of parameters included in the case of independent mask tensors is slightly larger, because each reference convolution kernel is operated with different sets of mask tensors Obtaining sub-convolution kernels, which makes the image features extracted based on these sub-convolution kernels more distinguishable and discriminative, which can improve the effect of image classification to a certain extent.

With reference to the first aspect, in some implementation manners of the first aspect, at least part of the mask tensors in at least one group of the above-mentioned N groups of mask tensors satisfy pairwise orthogonality.

When the two mask tensors are orthogonal, it means that the parameters in the two mask tensors are quite different. According to the two mask tensors and the same reference convolution kernel or different reference convolution kernels, the Hadamard product is done Operation, the difference between the sub-convolution kernels obtained will also be relatively large, so that when the corresponding sub-convolution kernel is used for convolution processing, the extracted image features are more discriminative and discriminative, which can to a certain extent Improve the effect of image classification.

Optionally, all mask tensors in at least one group of mask tensors in the foregoing N groups of mask tensors satisfy pairwise orthogonality.

When any two mask tensors in at least one group of mask tensors in the N groups of mask tensors meet pairwise orthogonality, the features of the image extracted by the convolution processing according to the reference convolution kernel and the mask tensor are more abundant, and you can Improve the final processing effect of the image.

Optionally, all mask tensors in each group of mask tensors in the foregoing N groups of mask tensors satisfy pairwise orthogonality.

When all mask tensors in each group of mask tensors in N groups of mask tensors meet pairwise orthogonality, the features of the image extracted by convolution processing based on the reference convolution kernel and mask tensor are more abundant, which can improve The final image processing effect.

Optionally, the reference convolution kernel of the aforementioned neural network is composed of the aforementioned M reference convolution kernels, and the mask tensor of the aforementioned neural network is composed of the aforementioned N groups of mask tensors.

Among them, the size of M and N can be determined according to the construction of the neural network. For example, the above M and N can be determined according to the complexity of the neural network structure and the application requirements of the neural network. When the network structure of the neural network has high complexity or high application requirements (for example, the processing capacity requirements are relatively high). High), M and/or N can be set to larger values, and when the network structure of the above neural network is relatively simple or the application requirements are low (for example, the requirements for processing capabilities are low), M and/or Or N is set to a smaller value.

Optionally, the sizes of the aforementioned M reference convolution kernels are completely the same or completely different or partially the same.

When there are reference convolution kernels of different sizes in the M reference convolution kernels, it is possible to finally extract richer image features from the image to be processed. Specifically, the subconvolution kernels obtained when the different reference convolution kernels and the corresponding mask tensors perform the Hadamard product operation are generally different. According to these different subconvolution kernels, it is possible to extract from the image to be processed More comprehensive and distinctive features.

Further, when the sizes of the M reference convolution kernels are all different, it is possible to further extract richer image features from the image to be processed, so as to facilitate the subsequent better classification of the image to be processed.

Optionally, the above N groups of mask tensors are completely the same or completely different or partially the same.

It should be understood that the mask tensors contained in each group of mask tensors in the foregoing N groups of mask tensors are the same.

Each reference convolution kernel in the aforementioned M groups of reference convolution kernels will correspond to a group of mask tensors in the N groups of mask tensors. In this application, since the reference convolution kernel can correspond to the corresponding group of mask tensors The mask tensor performs the Hadamard product operation, so the size of the reference convolution kernel is the same as the size of the corresponding mask tensor. Only in this way can the reference convolution kernel perform the Hadamard product operation with the corresponding mask tensor to obtain the subconvolution kernel.

Optionally, any group of mask tensors in the aforementioned N groups of mask tensors has the same size as the corresponding reference convolution kernel.

That is to say, in a set of mask tensors corresponding to a certain reference convolution kernel, the size of each mask tensor is the same as the size of the corresponding reference convolution kernel.

If the first group of mask tensors in the above N groups of masks corresponds to the first reference convolution kernel in the M reference convolution kernels, then the value of each mask tensor in the first group of mask tensors The size is the same as the size of the first reference convolution kernel.

Specifically, if the size of the first reference convolution kernel is c×d ₁ ×d ₂ , where c represents the number of channels, and d ₁ and d ₂ represent height and width respectively. Then, the size of any first mask tensor in the first group of mask tensors is also c×d ₁ ×d ₂ (where c is the number of channels, and d ₁ and d ₂ are height and width respectively).

With reference to the first aspect, in some implementations of the first aspect, the convolution kernel parameters of the M reference convolution kernels and the N sets of mask tensors are obtained by training the neural network according to the training image.

Wherein, the image category of the aforementioned training image is the same as the image category of the image to be processed. For example, when the image to be processed is an image of human motion, the training image may be an image containing various types of human motion.

Specifically, when building a neural network, it can be determined according to factors such as the performance requirements of the network to be built, the complexity of the network structure, and the storage space required to store the corresponding convolution kernel parameters and mask tensor parameters. The values of M and N and the number of mask tensors contained in each group of mask tensors, and then initialize the convolution kernel parameters of M reference convolution kernels and N groups of mask tensors (that is, for these reference volume Set an initial value for the product kernel and mask tensor), and construct a loss function. Next, you can use the training image to train the neural network. During the training process, you can update the parameter values in the reference convolution kernel and the mask tensor according to the size of the loss function. When the loss function converges or the function value of the loss function When the requirements are met, or the number of training times reaches the preset number of times, training can be stopped, and the parameter values in the reference convolution kernel and mask tensor at this time are determined as the final parameter values of the reference convolution kernel and mask tensor. Next, The neural network containing the corresponding parameter values (that is, the final parameter values of the reference convolution kernel and mask tensor obtained by training) can be deployed to the required equipment as needed, and then the equipment that deploys the neural network can be used for Image classification.

In the second aspect, an image classification method is provided. The method includes: obtaining convolution kernel parameters of M reference convolution kernels of a neural network; obtaining N groups of mask tensors of the neural network; and according to M reference convolution kernels Perform convolution processing on the image to be processed to obtain M reference convolution feature maps of the image to be processed; perform Hadamard product operation on the M reference convolution feature maps and N groups of mask tensors to obtain multiple convolutions of the image to be processed Product feature map: classify the image to be processed according to multiple convolution feature maps of the image to be processed to obtain the classification result of the image to be processed.

The image to be processed can be obtained from a camera or an album.

When the convolution kernel parameters of the M reference convolution kernels are stored in the registers, the convolution kernel parameters of the M reference convolution kernels can be obtained from the register relatively quickly (as opposed to obtaining from external storage, from the register The speed of obtaining parameters will be faster), which can improve the execution speed of the above method to a certain extent.

Optionally, the above N groups of mask tensors are stored in a register.

The aforementioned register may specifically be a weight memory.

With reference to the second aspect, in some implementations of the second aspect, N is less than M, and at least two reference convolution kernels in the M reference convolution kernels correspond to one group of mask tensors in the N group of mask tensors.

When N is less than M, there will be a situation where multiple reference convolution kernels jointly correspond to the same (shared) set of mask tensors. This situation can be referred to as a shared mask tensor. In the case of mask tensor sharing, when the Hadamard product operation is performed, part of the reference convolution kernel will be operated with the same mask tensor to obtain the subconvolution kernel, which can further reduce the number of mask tensors. Can further reduce storage overhead.

With reference to the second aspect, in some implementations of the second aspect, at least part of the mask tensors in at least one set of mask tensors in the N sets of mask tensors satisfy pairwise orthogonality.

With reference to the second aspect, in some implementation manners of the second aspect, at least part of the mask tensors in the at least one set of mask tensors in the foregoing N sets of mask tensors satisfy pairwise orthogonality. When the two mask tensors are orthogonal, it means that the parameters in the two mask tensors are quite different. According to the two mask tensors and the same reference convolution kernel or different reference convolution kernels, the Hadamard product is done Operation, the difference between the sub-convolution kernels obtained will also be relatively large, so that when the corresponding sub-convolution kernel is used for convolution processing, the extracted image features are more discriminative and discriminative, which can to a certain extent Improve the effect of image classification.

With reference to the second aspect, in some implementations of the second aspect, the convolution kernel parameters of the M reference convolution kernels and the N sets of mask tensors are obtained by training the neural network according to the training image.

It should be understood that the expansion, limitation, explanation and description of the related content in the above-mentioned first aspect are also applicable to the same content in the second aspect, which will not be described in detail in the second aspect here.

In a third aspect, a data processing method is provided. The method includes: obtaining convolution kernel parameters of M reference convolution kernels of a neural network; obtaining N groups of mask tensors of the neural network; Each reference convolution kernel in and each reference convolution kernel performs the Hadamard product operation on the corresponding set of mask tensors in the N groups of mask tensors to obtain multiple subconvolution kernels; according to the multiple subconvolution kernels respectively Convolution processing is performed on the multimedia data to obtain multiple convolution feature maps of the multimedia data; the multimedia data is processed according to the multiple convolution feature maps of the multimedia data.

Optionally, the above-mentioned multimedia data is text, sound, picture (image), video, animation, etc.

Optionally, when the foregoing multimedia data is an image, processing the multimedia data according to multiple convolution feature maps of the multimedia data includes: classifying or identifying the multimedia data according to the multiple convolution feature maps of the multimedia data.

Optionally, when the foregoing multimedia data is an image, processing the multimedia data according to multiple convolution feature maps of the multimedia data includes: performing image processing on the multimedia data according to the multiple convolution feature maps of the multimedia data.

For example, perform convolution processing on the acquired face image to obtain a convolution feature map of the face image, and then process the convolution feature map of the face image to generate an animated expression corresponding to the facial expression. Alternatively, other expressions can also be transferred to the input face image and then output.

In this application, when using a neural network to process multimedia data, only the convolution kernel parameters of the reference convolution kernel of the neural network and the corresponding mask tensor need to be obtained, and then the reference convolution kernel and the corresponding mask can be used The tensor realizes the convolution processing of the data to be processed, which can reduce the storage overhead when the neural network is used for the convolution processing, thereby enabling the neural network to be deployed on more devices with limited storage resources and process multimedia data.

In a fourth aspect, a data processing method is provided. The method includes: obtaining convolution kernel parameters of M reference convolution kernels of a neural network; obtaining N groups of mask tensors of the neural network; according to the M reference convolution kernels Perform convolution processing on the image to be processed to obtain M reference convolution feature maps of the image to be processed; perform Hadamard product operation on M reference convolution feature maps and N groups of mask tensors to obtain multiple convolutions of multimedia data Feature map: The multimedia data is processed according to multiple convolution feature maps of the multimedia data.

For example, convolution processing is performed on the acquired face image to obtain a convolution feature map of the face image, and then the convolution feature map of the face image is processed to generate an animated expression corresponding to the facial expression.

In a fifth aspect, an image processing method is provided. The method includes: obtaining convolution kernel parameters of M reference convolution kernels of a neural network; obtaining N groups of mask tensors of the neural network; according to the M reference convolution kernels Convolution kernel parameters and N groups of mask tensors are used to convolve the road image to obtain multiple convolution feature maps of the road image; perform deconvolution processing on the multiple convolution feature maps of the road image to obtain the road image The result of semantic segmentation.

In this application, when the neural network is used for image processing on road pictures, only the convolution kernel parameters of the reference convolution kernel of the neural network and the corresponding mask tensor need to be obtained, and then the reference convolution kernel and the corresponding mask tensor can be used. The code tensor realizes the convolution processing of the road image, which can reduce the storage overhead when the neural network is used for the convolution processing, so that the neural network can be deployed to more devices with limited storage resources and image processing of the road image .

Optionally, the above method further includes: acquiring a road picture.

The execution subject of the foregoing method may be an image processing device in an autonomous driving vehicle, and the foregoing road image may be acquired by a roadside monitoring device, or may be an image acquired by an autonomous driving vehicle in real time according to a camera.

Optionally, performing deconvolution processing on multiple convolution feature maps of the road image to obtain the semantic segmentation result of the road image includes: splicing multiple convolution feature maps of the road image to obtain the target of the road image Convolution feature map: Deconvolution is performed on the target convolution feature map of the road image to obtain the semantic segmentation result of the road image.

The width and height of the multiple convolution features of the above road image should be the same. The above splicing of multiple convolution feature maps is essentially to superimpose the number of channels of the multiple convolution feature maps to obtain one channel number. The target convolution feature map of the sum of the number of channels of the convolution feature maps.

Optionally, the above-mentioned convolution processing is performed on the road image according to the convolution kernel parameters of the M reference convolution kernels and N sets of mask tensors to obtain multiple convolution feature maps of the road image, including: Each reference convolution kernel in the convolution kernel and each reference convolution kernel performs the Hadamard product operation on the corresponding set of mask tensors in the N groups of mask tensors to obtain multiple subconvolution kernels; according to multiple subconvolutions The kernel respectively performs convolution processing on the road image to obtain multiple convolution feature maps of the road image.

Optionally, the above-mentioned convolution processing is performed on the road image according to the convolution kernel parameters of the M reference convolution kernels and the N sets of mask tensors to obtain multiple convolution feature maps of the road image, including: according to the M reference volumes The product kernel performs convolution processing on the road image to obtain M reference convolution feature maps of the road image; performs Hadamard product operation on the M reference convolution feature maps and N groups of mask tensors to obtain multiple convolutions of the road image Feature map.

By first using the reference convolution kernel to convolve the road image, after obtaining a reference convolution feature map, combine the mask tensor to obtain multiple convolution feature maps of the road image, which can reduce the number of convolution calculations , Can reduce the amount of calculation to a certain extent.

In a sixth aspect, an image processing method is provided. The method includes: obtaining convolution kernel parameters of M reference convolution kernels of a neural network; obtaining N groups of mask tensors of the neural network; Each reference convolution kernel in and each reference convolution kernel performs the Hadamard product operation on the corresponding set of mask tensors in the N groups of mask tensors to obtain multiple subconvolution kernels; according to the multiple subconvolution kernels respectively Perform convolution processing on the road image to obtain multiple convolution feature maps of the road image; compare the multiple convolution feature maps of the face image with the convolution feature map of the image corresponding to the identity document of the face image to obtain the face Image verification result.

In this application, when using a neural network to perform image processing on a face image, only the convolution kernel parameters of the reference convolution kernel of the neural network and the corresponding mask tensor need to be obtained, and then the reference convolution kernel and the corresponding The mask tensor realizes the convolution processing of the face image, which can reduce the storage overhead when the neural network is used for the convolution processing, so that the neural network can be deployed on more storage resource-constrained devices and the face image Perform image processing.

Optionally, the above method further includes: acquiring a face image.

Optionally, the above-mentioned convolution processing is performed on the face image according to the convolution kernel parameters of the M reference convolution kernels and the N sets of mask tensors to obtain multiple convolution feature maps of the face image, including: Each reference convolution kernel in the reference convolution kernel, and each reference convolution kernel performs the Hadamard product operation on the corresponding set of mask tensors in the N groups of mask tensors to obtain multiple subconvolution kernels; according to the multiple subconvolution kernels The convolution kernel performs convolution processing on the face image to obtain multiple convolution feature maps of the face image.

Optionally, the aforementioned convolution processing is performed on the face image according to the convolution kernel parameters of the M reference convolution kernels and the N sets of mask tensors to obtain multiple convolution feature maps of the face image, including: according to M The reference convolution kernel performs convolution processing on the face image to obtain M reference convolution feature maps of the face image; performs the Hadamard product operation on the M reference convolution feature maps and N groups of mask tensors to obtain the face image Of multiple convolutional feature maps.

By first using the reference convolution kernel to perform convolution processing on the face image, after obtaining a reference convolution feature map, combine the mask tensor to obtain multiple convolution feature maps of the road image, which can reduce the convolution calculation The number of times can reduce the amount of calculation to a certain extent.

In a seventh aspect, an image classification device is provided. The image classification device includes: a memory for storing convolution kernel parameters and N groups of mask tensors of M reference convolution kernels of a neural network; a processor for Obtain the convolution kernel parameters and N sets of mask tensors of the M reference convolution kernels of the neural network, and perform the following operations: for each reference convolution kernel in the M reference convolution kernels, and each reference convolution The kernel performs the Hadamard product operation on the corresponding set of mask tensors in the N sets of mask tensors to obtain multiple subconvolution kernels; according to the multiple subconvolution kernels, the image to be processed is convolved to obtain multiple convolution feature maps. ; Classify the image to be processed according to multiple convolution feature maps to obtain the classification result of the image to be processed.

Among them, the above M and N are both positive integers, each group of mask tensors in the N groups of mask tensors is composed of multiple mask tensors, and the number of bits occupied by the elements in the N groups of mask tensors is less than the number of M reference volumes. The number of bits occupied by the elements in the convolution kernel parameters in the convolution kernel. Each reference convolution kernel in the M reference convolution kernels corresponds to a group of mask tensors in the N groups of mask tensors.

With reference to the seventh aspect, in some implementations of the seventh aspect, N is less than M, and at least two reference convolution kernels in the M reference convolution kernels correspond to a set of mask tensors in the N groups of mask tensors.

With reference to the seventh aspect, in some implementation manners of the seventh aspect, at least part of the mask tensors in at least one set of mask tensors in the N sets of mask tensors satisfy pairwise orthogonality.

With reference to the seventh aspect, in some implementations of the seventh aspect, the convolution kernel parameters of the M reference convolution kernels and the N sets of mask tensors are obtained by training the neural network according to the training image.

It should be understood that the image classification device in the seventh aspect described above corresponds to the image classification method in the first aspect, and the image classification device in the seventh aspect can execute the image classification method in the first aspect. The expansion, limitation, explanation and description of related content are also applicable to the same content in the seventh aspect, and the related content of the seventh aspect will not be described in detail here.

In an eighth aspect, an image classification device is provided. The image classification device includes: a memory for storing convolution kernel parameters and N groups of mask tensors of M reference convolution kernels of a neural network; a processor for Obtain the convolution kernel parameters and N sets of mask tensors of the M reference convolution kernels of the neural network, and perform the following operations: perform convolution processing on the image to be processed according to the M reference convolution kernels to obtain M of the image to be processed Reference convolution feature map; Hadamard product operation is performed on M reference convolution feature maps and N groups of mask tensors to obtain multiple convolution feature maps of the image to be processed; multiple convolution feature maps of the image to be processed The image to be processed is classified, and the classification result of the image to be processed is obtained.

Among them, the above M and N are both positive integers, each group of mask tensors in the N groups of mask tensors is composed of multiple mask tensors, and the number of bits occupied by the elements in the N groups of mask tensors is less than the number of M reference volumes. The number of bits occupied by the elements in the convolution kernel parameters in the convolution kernel. Each reference convolution kernel in the M reference convolution kernels corresponds to a set of mask tensors in the N groups of mask tensors

With reference to the eighth aspect, in some implementations of the eighth aspect, N is less than M, and at least two reference convolution kernels in the M reference convolution kernels correspond to a set of mask tensors in the N groups of mask tensors.

With reference to the eighth aspect, in some implementation manners of the eighth aspect, at least part of the mask tensors in at least one set of mask tensors in the N sets of mask tensors satisfy pairwise orthogonality.

With reference to the eighth aspect, in some implementations of the eighth aspect, the convolution kernel parameters of the M reference convolution kernels and the N sets of mask tensors are obtained by training the neural network according to the training image.

It should be understood that the image classification device in the eighth aspect described above corresponds to the image classification method in the second aspect. The image classification device in the eighth aspect can execute the image classification method in the second aspect. The expansion, limitation, explanation and description of related content are also applicable to the same content in the eighth aspect, and the related content of the eighth aspect will not be described in detail here.

In a ninth aspect, a data processing device is provided. The data processing device includes: a memory for storing convolution kernel parameters and N groups of mask tensors of M reference convolution kernels of a neural network; a processor for Obtain the convolution kernel parameters and N sets of mask tensors of the M reference convolution kernels of the neural network, and perform the following operations: for each reference convolution kernel in the M reference convolution kernels, and each reference convolution The kernel performs the Hadamard product operation on the corresponding set of mask tensors in the N sets of mask tensors to obtain multiple subconvolution kernels; according to the multiple subconvolution kernels, the multimedia data are respectively convolved to obtain multiple volumes of multimedia data Product feature map: The multimedia data is processed according to multiple convolution feature maps of the multimedia data.

It should be understood that the data processing device in the ninth aspect described above corresponds to the data processing method in the third aspect, and the data processing device in the ninth aspect can execute the data processing method in the third aspect. The expansion, limitation, explanation and description of related content are also applicable to the same content in the ninth aspect, and the related content of the ninth aspect will not be described in detail here.

In a tenth aspect, a data processing device is provided. The data processing device includes: a memory for storing convolution kernel parameters and N groups of mask tensors of M reference convolution kernels of a neural network; a processor for Obtain the convolution kernel parameters and N groups of mask tensors of the M reference convolution kernels of the neural network, and perform the following operations: perform convolution processing on the multimedia data according to the M reference convolution kernels to obtain M reference volumes of the multimedia data Product feature maps; Hadamard product operations are performed on M reference convolution feature maps and N groups of mask tensors to obtain multiple convolution feature maps of multimedia data; multimedia data are processed according to multiple convolution feature maps of multimedia data deal with.

It should be understood that the data processing device in the tenth aspect described above corresponds to the data processing method in the fourth aspect, and the data processing device in the tenth aspect can execute the data processing method in the fourth aspect. The expansion, limitation, explanation and description of related content are also applicable to the same content in the tenth aspect, and the related content of the tenth aspect will not be described in detail here.

In an eleventh aspect, a computer-readable medium is provided, the computer-readable medium stores program code for execution by a device, and the program code includes a method for executing any one of the first to sixth aspects.

A twelfth aspect provides a computer program product containing instructions, when the computer program product runs on a computer, the computer executes the method in any one of the first to sixth aspects.

In a thirteenth aspect, a chip is provided. The chip includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface, and executes any one of the first to sixth aspects above. The method in the aspect.

Optionally, as an implementation manner, the chip may further include a memory in which instructions are stored, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the The processor is used to execute the method in any one of the first aspect to the sixth aspect.

Description of the drawings

FIG. 1 is a schematic structural diagram of a system architecture provided by an embodiment of the present application;

2 is a schematic diagram of image classification according to a convolutional neural network model provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application;

Figure 4 is a schematic diagram of a mobile phone taking a selfie scene;

Figure 5 is a schematic diagram of a face verification scene;

Figure 6 is a schematic diagram of a speech recognition and machine translation scenario;

FIG. 7 is a schematic flowchart of an image classification method according to an embodiment of the present application;

FIG. 8 is a schematic diagram of obtaining a sub-convolution kernel according to a reference convolution kernel and a mask tensor;

FIG. 9 is a schematic diagram of obtaining a sub-convolution kernel according to a reference convolution kernel and a mask tensor;

Figure 10 is a schematic diagram of the process of image classification using neural networks;

11 is a schematic diagram of the process of obtaining convolution kernel parameters and mask tensor of the reference convolution kernel;

FIG. 12 is a schematic flowchart of a data processing method according to an embodiment of the present application;

FIG. 13 is a schematic diagram of the hardware structure of the neural network training device according to an embodiment of the present application;

FIG. 14 is a schematic diagram of the hardware structure of an image classification device according to an embodiment of the present application;

FIG. 15 is a schematic diagram of the hardware structure of a data processing device according to an embodiment of the present application.

Detailed ways

The technical solution in this application will be described below in conjunction with the drawings.

The embodiment of the application provides an image classification method and a data processing method.

Among them, the data processing method of the embodiment of this application can be applied to various scenarios in the fields of computer vision, for example, the processing method of the embodiment of this application can be applied to scenes such as face recognition, image classification, target detection, semantic segmentation, etc. .

In order to more vividly understand the application scenario of the data processing method in the embodiment of the present application, the following takes a specific scenario as an example for description.

Terminal equipment object detection:

This is a problem of target detection. When a user uses a terminal device (for example, mobile phone, tablet, etc.) to take a picture, the terminal device can automatically capture human faces, animals and other targets (in this process, the terminal device realizes the detection of human faces or other Object recognition and grabbing, etc.), which can help terminal equipment to automatically focus and beautify. Therefore, terminal devices need a small, fast-running target detection convolutional neural network model, so as to bring users a better user experience and improve the product quality of the terminal device.

For example, as shown in Figure 4, when a user uses a mobile phone to take a selfie, the mobile phone can automatically recognize the face according to the neural network model, and automatically capture the face to generate a prediction frame. The neural network model in Figure 4 can be a target detection convolutional neural network model located in a mobile phone. The target detection convolutional neural network model has the characteristics of fewer parameters (the convolution kernel has fewer parameters) and can be deployed in storage On mobile phones with limited resources. In addition, it should be understood that the prediction box shown in FIG. 4 is only for illustration. For ease of understanding, the prediction box is directly displayed in the picture. In fact, the prediction box is displayed on the shooting interface of the selfie mobile phone.

Semantic segmentation in autonomous driving scenarios:

The camera of an autonomous vehicle will capture the road image in real time. In order to enable the autonomous vehicle to recognize different objects on the road, the smart device in the autonomous vehicle needs to segment the captured road image to separate the road surface, roadbed, and vehicle. , Pedestrians, and other objects, and feed this information back to the control system of the autonomous vehicle, so that the autonomous vehicle can drive on the correct road area. Since autonomous driving has extremely high requirements for safety, smart devices in autonomous vehicles need to be able to quickly process and analyze captured real-time road images to obtain semantic segmentation results.

Face verification at entrance gate:

This is an image similarity comparison problem. At the gates at the entrances of high-speed railways and airports, when passengers perform face authentication, the camera will take facial images. For the captured facial images, convolutional neural networks can be used to extract image features, and then the extracted image features will be stored The similarity calculation is performed on the image features of the ID documents in the system, and the verification is successful if the similarity is high.

For example, as shown in Figure 5, the neural network model processes the captured face image to obtain feature A, and the neural network model processes the image of the ID document to obtain feature B. Next, by comparing feature A and feature B Similarity can determine whether the person being photographed and the person on the ID card belong to the same person, if the similarity of feature A and feature B meets the requirements (for example, the similarity of feature A and feature B is greater than or equal to the preset similarity threshold) , Then, it can be determined that the person being photographed and the person on the ID card belong to the same person.

Simultaneous interpretation by translator:

This is a speech recognition and machine translation problem. In speech recognition and machine translation, convolutional neural networks are also commonly used recognition models. In the scenario of simultaneous interpretation, an efficient neural network must be used to achieve real-time speech recognition and translation, so as to bring a better user experience.

For example, as shown in Figure 6, the input voice is English "Hello world!", the received voice is recognized through the neural network model, and machine translation is performed according to the recognition result, and the corresponding translation is output as Chinese "World, Hello !", the translation here can include both the voice and the text of the translation.

In the above several application scenarios (terminal device object detection, semantic segmentation in autonomous driving scenarios, face verification at entrance gates, and simultaneous interpretation by translators), relatively high-performance neural network models need to be adopted for corresponding data However, in many cases, the storage space of the devices that need to be deployed is limited. Therefore, how to deploy neural networks with relatively high performance but relatively few parameters on these devices with limited storage resources is very important for data processing. Therefore, this application provides a data processing method that deploys some neural network models with fewer parameters, so that some devices with limited storage resources can also achieve efficient data processing. The specific process will be described below A detailed description.

Since the embodiments of the present application involve a large number of applications of neural networks, in order to facilitate understanding, the following first introduces related terms and concepts of neural networks that may be involved in the embodiments of the present application.

(1) Neural network

A neural network can be composed of neural units. A neural unit can refer to an arithmetic unit that takes x _s and intercept 1 as inputs. The output of the arithmetic unit can be:

Among them, s=1, 2,...n, n is a natural number greater than 1, W _s is the weight of x _s , and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be a region composed of several neural units.

(2) Deep neural network

Deep neural network (DNN), also called multi-layer neural network, can be understood as a neural network with multiple hidden layers. DNN is divided according to the positions of different layers. The neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the number of layers in the middle are all hidden layers. The layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer.

Although DNN looks complicated, it is not complicated in terms of the work of each layer. In simple terms, it is the following linear relationship expression:

among them,

Is the input vector,

Is the output vector,

Is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just the input vector

After such a simple operation, the output vector is obtained

Due to the large number of DNN layers, the coefficient W and the offset vector

The number is also relatively large. The definition of these parameters in the DNN is as follows: Take the coefficient W as an example: Suppose that in a three-layer DNN, the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as

The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.

In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

It should be noted that the input layer has no W parameter. In deep neural networks, more hidden layers make the network more capable of portraying complex situations in the real world. Theoretically speaking, a model with more parameters is more complex and has a greater "capacity", which means it can complete more complex learning tasks. Training a deep neural network is also a process of learning a weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by vectors W of many layers).

(3) Convolutional neural network

Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with convolutional structure. The convolutional neural network contains a feature extractor composed of a convolution layer and a sub-sampling layer. The feature extractor can be regarded as a filter. The convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network. In the convolutional layer of a convolutional neural network, a neuron can be connected to only part of the neighboring neurons. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Sharing weight can be understood as the way to extract image information has nothing to do with location. The convolution kernel can be initialized in the form of a matrix of random size. During the training of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

(4) Recurrent Neural Networks (RNN) are used to process sequence data. In the traditional neural network model, from the input layer to the hidden layer and then to the output layer, the layers are fully connected, and each node in each layer is disconnected. Although this ordinary neural network has solved many problems, it is still incapable of many problems. For example, if you want to predict what the next word of a sentence will be, you generally need to use the previous word, because the preceding and following words in a sentence are not independent. The reason why RNN is called recurrent neural network is that the current output of a sequence is also related to the previous output. The specific form is that the network will memorize the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layer are no longer unconnected but connected, and the input of the hidden layer includes not only The output of the input layer also includes the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN.

Now that there is already a convolutional neural network, why bother to recycle neural networks? The reason is simple. In convolutional neural networks, there is a premise that the elements are independent of each other, and the input and output are also independent, such as cats and dogs. But in the real world, many elements are connected to each other, such as the change of stocks over time, and another person said: I like traveling, and my favorite place is Yunnan, and I must go if I have a chance in the future. Filling in the blanks here, humans should all know that it is filling in "Yunnan". Because humans will make inferences based on the content of the context, but how to make the machine do this step? RNN came into being. RNN aims to make machines have memory capabilities like humans. Therefore, the output of RNN needs to rely on current input information and historical memory information.

(5) Loss function

In the process of training a deep neural network, because it is hoped that the output of the deep neural network is as close as possible to the value that you really want to predict, you can compare the predicted value of the current network with the target value you really want, and then based on the difference between the two To update the weight vector of each layer of neural network (of course, there is usually an initialization process before the first update, that is, pre-configured parameters for each layer in the deep neural network), for example, if the predicted value of the network If it is high, adjust the weight vector to make its prediction lower, and keep adjusting until the deep neural network can predict the really wanted target value or a value very close to the really wanted target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value". This is the loss function or objective function, which is used to measure the difference between the predicted value and the target value. Important equation. Among them, take the loss function as an example. The higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of reducing this loss as much as possible.

(6) Backpropagation algorithm

The neural network can use an error back propagation (BP) algorithm to modify the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forwarding the input signal to the output will cause error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss is converged. The backpropagation algorithm is a backpropagation motion dominated by error loss, and aims to obtain the optimal neural network model parameters, such as the weight matrix.

(7) Pixel value

The pixel value of the image can be a red-green-blue (RGB) color value, and the pixel value can be a long integer representing the color. For example, the pixel value is 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. In each color component, the smaller the value, the lower the brightness, and the larger the value, the higher the brightness. For grayscale images, the pixel values can be grayscale values.

As shown in FIG. 1, an embodiment of the present application provides a system architecture 100. In FIG. 1, a data collection device 160 is used to collect training data. For the image classification method of the embodiment of the present application, the training data may include training images and classification results corresponding to the training images, where the results of the training images may be manually pre-labeled results. For the data processing method of the embodiment of this application, the specific type of training data is the same as the data type of the data to be processed and the specific process of data processing is related. For example, when the data to be processed is an image to be processed, this application implements If the data processing method of the example is to perform noise reduction processing on the image to be processed, then the training data corresponding to the data processing method of the embodiment of the present application may include the original image and the noise image after adding noise to the original image.

After the training data is collected, the data collection device 160 stores the training data in the database 130, and the training device 120 trains to obtain the target model/rule 101 based on the training data maintained in the database 130.

The following describes the target model/rule 101 obtained by the training device 120 based on the training data. The training device 120 processes the input original image and compares the output image with the original image until the output image of the training device 120 differs from the original image. The difference is less than a certain threshold, thereby completing the training of the target model/rule 101.

The above-mentioned target model/rule 101 can be used to implement the image classification method or data processing method of the embodiment of the present application, that is, the image to be processed is input into the target model/rule 101 after relevant preprocessing to obtain the denoising processed image image. The target model/rule 101 in the embodiment of the present application may specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the collection of the data collection device 160, and may also be received from other devices. In addition, it should be noted that the training device 120 does not necessarily perform the training of the target model/rule 101 completely based on the training data maintained by the database 130. It may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to this application. Limitations of Examples.

The target model/rule 101 trained according to the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG. 1. The execution device 110 may be a terminal, such as a mobile phone terminal, a tablet computer, Notebook computers, augmented reality (AR) AR/virtual reality (VR), vehicle-mounted terminals, etc., can also be servers or clouds. In FIG. 1, the execution device 110 is configured with an input/output (input/output, I/O) interface 112 for data interaction with external devices. The user can input data to the I/O interface 112 through the client device 140. The input data in this embodiment of the application may include: the image to be processed input by the client device.

The preprocessing module 113 and the preprocessing module 114 are used for preprocessing according to the input data (such as the image to be processed) received by the I/O interface 112. In the embodiment of the present application, the preprocessing module 113 and the preprocessing module may not be provided 114 (there may only be one preprocessing module), and the calculation module 111 is directly used to process the input data.

When the execution device 110 preprocesses input data, or when the calculation module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 may call data, codes, etc. in the data storage system 150 for corresponding processing , The data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 150.

Finally, the I/O interface 112 returns the processing result, such as the denoising processed image obtained as described above, to the client device 140 to provide it to the user.

It is worth noting that the training device 120 can generate corresponding target models/rules 101 based on different training data for different goals or tasks, and the corresponding target models/rules 101 can be used to achieve the above goals or complete The above tasks provide the user with the desired result.

In the case shown in FIG. 1, the user can manually set input data, and the manual setting can be operated through the interface provided by the I/O interface 112. In another case, the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send the input data and the user's authorization is required, the user can set the corresponding authority in the client device 140. The user can view the result output by the execution device 110 on the client device 140, and the specific presentation form may be a specific manner such as display, sound, and action. The client device 140 can also be used as a data collection terminal to collect the input data of the input I/O interface 112 and the output result of the output I/O interface 112 as new sample data, and store it in the database 130 as shown in the figure. Of course, it is also possible not to collect through the client device 140, but the I/O interface 112 directly uses the input data input to the I/O interface 112 and the output result of the output I/O interface 112 as a new sample as shown in the figure. The data is stored in the database 130.

It is worth noting that Fig. 1 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in Fig. 1 The data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

As shown in FIG. 1, the target model/rule 101 is obtained by training according to the training device 120. The target model/rule 101 may be the neural network in the present application in the embodiment of the application. Specifically, the neural network provided in the embodiment of the present application Can be CNN, deep convolutional neural networks (deep convolutional neural networks, DCNN), recurrent neural networks (recurrent neural network, RNNS) and so on.

Since CNN is a very common neural network, the structure of CNN will be introduced in detail below in conjunction with Figure 2. As mentioned in the introduction to the basic concepts above, a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture. A deep learning architecture refers to a machine learning algorithm. Multi-level learning is carried out on the abstract level of As a deep learning architecture, CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the input image.

As shown in FIG. 2, a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer/pooling layer 220 (the pooling layer is optional), and a neural network layer 230. The following is a detailed introduction to the relevant content of these layers.

Convolutional layer/pooling layer 220:

Convolutional layer:

The convolutional layer/pooling layer 220 as shown in FIG. 2 may include layers 221-226, for example: in an implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, and layer 223 is a convolutional layer. Layers, 224 is the pooling layer, 225 is the convolutional layer, and 226 is the pooling layer; in another implementation, 221 and 222 are the convolutional layers, 223 is the pooling layer, and 224 and 225 are the convolutional layers. Layer, 226 is the pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

The following will take the convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

The convolution layer 221 can include many convolution operators. The convolution operator is also called a kernel. Its function in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. In the process of convolution on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. ...It depends on the value of stride) to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix and the depth dimension of the input image are the same. During the convolution operation, the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension convolution output, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row × column) are applied. That is, multiple homogeneous matrices. The output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to eliminate unwanted noise in the image. Perform fuzzification, etc. The multiple weight matrices have the same size (row×column), the size of the convolution feature maps extracted by the multiple weight matrices of the same size are also the same, and then the multiple extracted convolution feature maps of the same size are combined to form The output of the convolution operation.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications. Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions. .

When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (such as 221) often extracts more general features, which can also be called low-level features; with the convolutional neural network As the network 200 deepens, the features extracted by the subsequent convolutional layers (for example, 226) become more and more complex, such as features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.

Pooling layer:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolutional layer. In the 221-226 layers as illustrated by 220 in Figure 2, it can be a convolutional layer followed by a layer The pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the size of the image space. The pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of the maximum pooling. In addition, just as the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image of the input pooling layer, and each pixel in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

Neural network layer 230:

After processing by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 200 needs to use the neural network layer 230 to generate one or a group of required classes of output. Therefore, the neural network layer 230 may include multiple hidden layers (231, 232 to 23n as shown in FIG. 2) and an output layer 240. The parameters contained in the multiple hidden layers can be based on specific task types. The relevant training data of the, for example, the task type can include image recognition, image classification, image super-resolution reconstruction and so on.

After the multiple hidden layers in the neural network layer 230, that is, the final layer of the entire convolutional neural network 200 is the output layer 240. The output layer 240 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (as shown in Figure 2 from 210 to 240 is the forward propagation) is completed, the back propagation (as shown in Figure 2 is the propagation from 240 to 210 is the back propagation) Start to update the weight values and deviations of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the output result of the convolutional neural network 200 through the output layer and the ideal result.

It should be noted that the convolutional neural network 200 shown in FIG. 2 is only used as an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

FIG. 3 is a chip hardware structure provided by an embodiment of the application, and the chip includes a neural network processor 50. The chip may be set in the execution device 110 as shown in FIG. 1 to complete the calculation work of the calculation module 111. The chip can also be set in the training device 120 as shown in FIG. 1 to complete the training work of the training device 120 and output the target model/rule 101. The algorithms of each layer in the convolutional neural network as shown in Figure 2 can be implemented in the chip as shown in Figure 3.

The neural network processor NPU 50 NPU is mounted as a coprocessor to a main central processing unit (central processing unit, CPU) (host CPU), and the main CPU distributes tasks. The core part of the NPU is the arithmetic circuit 50. The controller 504 controls the arithmetic circuit 503 to extract data from the memory (weight memory or input memory) and perform calculations.

In some implementations, the arithmetic circuit 503 includes multiple processing units (process engines, PE). In some implementations, the arithmetic circuit 503 is a two-dimensional systolic array. The arithmetic circuit 503 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the corresponding data of matrix B from the weight memory 502 and buffers it on each PE in the arithmetic circuit. The arithmetic circuit fetches matrix A data and matrix B from the input memory 501 to perform matrix operations, and the partial or final result of the obtained matrix is stored in an accumulator 508.

The vector calculation unit 507 can perform further processing on the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on. For example, the vector calculation unit 507 can be used for network calculations in the non-convolutional/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .

In some implementations, the vector calculation unit 507 can store the processed output vector in the unified buffer 506. For example, the vector calculation unit 507 may apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate the activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a combined value, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 503, for example for use in subsequent layers in a neural network.

The unified memory 506 is used to store input data and output data.

The weight data directly transfers the input data in the external memory to the input memory 501 and/or the unified memory 506 through the storage unit access controller 505 (direct memory access controller, DMAC), and stores the weight data in the external memory into the weight memory 502, And the data in the unified memory 506 is stored in the external memory.

The bus interface unit (BIU) 510 is used to implement interaction between the main CPU, the DMAC, and the fetch memory 509 through the bus.

An instruction fetch buffer 509 connected to the controller 504 is used to store instructions used by the controller 504;

The controller 504 is configured to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

Entrance: It can be explained according to the actual invention that the data here is explanatory data, such as the detected vehicle speed? Obstacle distance etc.

Generally, the unified memory 506, the input memory 501, the weight memory 502, and the instruction fetch memory 509 are all on-chip (On-Chip) memories, and the external memory is a memory external to the NPU. The external memory can be a double data rate synchronous dynamic random access memory. Memory (double data rate synchronous dynamic random access memory, referred to as DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.

Among them, the operations of each layer in the convolutional neural network shown in FIG. 2 can be executed by the arithmetic circuit 303 or the vector calculation unit 307.

The execution device 110 in FIG. 1 introduced above can execute each step of the image classification method or data processing method of the embodiment of this application. The CNN model shown in FIG. 2 and the chip shown in FIG. 3 can also be used to execute this application. Each step of the image classification method or data processing method of the embodiment. The image classification method of the embodiment of the present application and the data processing method of the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

When the image classification method and data processing method of the embodiments of the present application are introduced in the following, it will involve the convolution processing of the image to be processed or the data to be processed, where the convolution processing can be called convolution feature map or It can be called a feature map directly.

Fig. 7 is a schematic flowchart of an image classification method according to an embodiment of the present application. The method shown in FIG. 7 may be executed by an image classification device, which may be an electronic device with image processing functions. The electronic device may specifically be a mobile terminal (for example, a smart phone), a computer, a personal digital assistant, a wearable device, a vehicle-mounted device, an Internet of Things device or other devices capable of image processing.

The method shown in FIG. 7 includes steps 1001 to 1004, which are described in detail below.

1001. Obtain convolution kernel parameters of M reference convolution kernels of the neural network.

Wherein, the above M is a positive integer.

1002. Obtain N sets of mask tensors of the neural network.

Among them, each group of mask tensors in the above N groups of mask tensors is composed of multiple mask tensors, and the number of bits occupied by elements in the above N groups of mask tensors is less than the convolution kernel parameters in the M reference convolution kernels. The number of bits occupied by the elements in the storage (in general, the storage space occupied by the elements in the mask tensor will be much less than the storage space occupied by the elements in the convolution kernel parameters), each of the M reference convolution kernels The convolution kernel corresponds to a group of mask tensors in N groups of mask tensors.

The convolution kernel parameters of the M reference convolution kernels and the N sets of mask tensors can be stored in registers. At this time, the convolution kernel parameters of the M reference convolution kernels and the N sets of mask tensors can be read from the register. The register may specifically be a weight register, that is, a register used to store convolution kernel parameters in a neural network.

It should be understood that the reference convolution kernel of the foregoing neural network is composed of the foregoing M reference convolution kernels, and the mask tensor of the foregoing neural network is composed of the foregoing N groups of mask tensors. The neural network only needs to save the convolution kernel parameters of M reference convolution kernels and N groups of mask tensors during deployment, instead of storing the parameters of each convolution kernel one by one. The storage space required for the deployment of the neural network can be saved, so that the neural network can also be deployed on some devices with limited storage resources.

In addition, the size of the above M and N can be determined according to the construction of the neural network. For example, the above M and N can be determined according to the complexity of the neural network structure and the application requirements of the neural network. When the network structure of the neural network has high complexity or high application requirements (for example, the processing capacity requirements are relatively high). High), M and/or N can be set to larger values, and when the network structure of the above neural network is relatively simple or the application requirements are low (for example, the requirements for processing capabilities are low), M and/or Or N is set to a smaller value.

It should be understood that the sizes of the aforementioned M reference convolution kernels may be completely the same, completely different, or partially the same.

When there are reference convolution kernels of different sizes among the M reference convolution kernels, more image features can be extracted from the image to be processed.

Further, when the sizes of the M reference convolution kernels are all different, more image features can be further extracted from the image to be processed, so that the subsequent image to be processed can be better classified.

Similar to the foregoing M reference convolution kernels, the foregoing N sets of mask tensors are completely the same, completely different, or partially the same.

Optionally, the mask tensors contained in each of the above N groups of mask tensors have the same size.

Optionally, each of the foregoing M reference convolution kernels corresponds to a set of mask tensors in N sets of mask tensors, and one set of mask tensors in the foregoing N sets of mask tensors may correspond to the foregoing M One or more convolution kernels among the reference convolution kernels.

In this application, the size of the mask tensor is the same as the size of the corresponding reference convolution kernel. Make the size of the convolution kernel obtained by calculation between the reference convolution kernel and the mask tensor the same as the size of the reference convolution kernel, so that the volume of the same size can be obtained according to the reference convolution kernel and the mask tensor The integration kernel facilitates subsequent unified processing of the image to be processed based on the obtained convolution kernel.

When the size of the reference convolution kernel is the same as the size of the mask tensor, it can be ensured that the Hadamard product operation can be performed normally between the reference convolution kernel and the mask tensor, so that according to the reference convolution kernel and the mask The tensor gets the sub-convolution kernel.

Among them, the Hadamard product operation can also be called the element multiplication operation, which is an operation in the matrix. If A=(a _ij ) and B=(b _ij ) are two matrices of the same order, and c _ij =a _ij ×b _ij , then the matrix C=(c _ij ) is called the Hadamard product of A and B, or It is called the basic product.

Optionally, the above-mentioned mask tensor is an L-value mask tensor. In other words, for a certain mask tensor, there may be L kinds of values of the elements in the mask tensor. Among them, L is a positive integer greater than or equal to 2.

Generally speaking, the smaller the value of L, the smaller the storage space occupied by the mask tensor.

Optionally, the above-mentioned mask tensor is a binary mask tensor. At this time, each element in the mask tensor has only two possible values, and the occupied bits are greatly reduced.

When the mask tensor in the information database is a binary mask tensor, the storage space occupied is small, and the effect of saving storage space is obvious.

The candidate values of the elements in the binary mask tensor can be [0, 1] or [0, -1] or [1, -1].

For the aforementioned M reference convolution kernels and N sets of mask tensors, the value of M is generally greater than or equal to N. That is to say, each of the above M reference convolution kernels can correspond to a set of mask tensors in the N sets of mask tensors, and a set of mask tensors in the above N sets of mask tensors can correspond to the above One or more convolution kernels among the M reference convolution kernels. When M is greater than N or M=N, the above-mentioned M reference convolution kernels have different correspondences with N groups of mask tensors. The two cases of M>N and M=N are respectively introduced below.

The first case: M>N

In the first case, at least two reference convolution kernels in the M reference convolution kernels jointly correspond to one group of mask tensors in the N group of mask tensors.

For example, M=3, N=2, M reference convolution kernels include the first reference convolution kernel, the second reference convolution kernel and the third reference convolution kernel, and the N sets of mask tensors include the first set of masks Tensor and the second set of masks, then, the correspondence between M reference convolution kernels and N sets of mask tensors can be shown in Table 1.

Table 1

As shown in Table 1, both the first reference convolution kernel and the second reference convolution kernel correspond to the first group of mask tensors, and the third reference convolution kernel corresponds to the second group of mask tensors. When the convolution kernel and mask tensor are used for convolution processing on the image to be processed, the first reference convolution kernel and the first group of mask tensors, the second reference convolution kernel and the first group of mask tensors, and the third The reference convolution kernel and the second set of mask tensor respectively perform convolution processing on the image to be processed, and finally obtain the convolution feature map of the image to be processed.

In the first case, N can also be equal to 1. At this time, the M groups of reference convolution kernels correspond to a set of mask tensors, and the mask tensor is shared by multiple reference convolution kernels (this case can be It is called mask tensor sharing), and the use of mask tensor sharing can further reduce the storage overhead caused by the mask tensor.

In the first case above, there will be multiple reference convolution kernels corresponding to the same set of mask tensors, that is, in the first case, different reference convolution kernels can share the same mask tensor Therefore, the first case mentioned above can also be called the case of mask tensor sharing.

The following describes the sharing of the mask tensor with reference to FIG. 8.

As shown in FIG. 8, the reference convolution kernel 1 and the reference convolution kernel 2 share a set of mask tensors, and the shared set of mask tensors includes mask tensor 1 and mask tensor 2. Through the operation of the reference convolution kernel 1 and the mask tensor 1 and the mask tensor 2, the subconvolution kernel 1 and the subconvolution kernel 2 can be obtained. The reference convolution kernel 2 is respectively compared with the mask tensor 1 and the mask tensor 2. Operation of code tensor 2 can obtain sub-convolution kernel 3 and sub-convolution kernel 4.

When performing operations based on the reference convolution kernel 1 and the mask tensor 1, specifically it can be the Hadamard product operation (that is, the element multiplication operation) on the mask tensor corresponding to the reference convolution kernel 1 and the mask tensor 1. ), the parameters of sub-convolution kernel 1 are obtained, and the calculation process of other sub-convolution kernels is similar.

The following describes in detail the related operations when the mask tensor is shared with the formula.

Suppose the input data (equivalent to the image to be processed above) is

Among them, c is the number of channels, h and w respectively represent the length and width of the input data (when the input data is an image, h and w represent the length and width of the image respectively). A convolution kernel in the neural network can be written as

Among them, c still represents the number of channels, and d ₁ ×d ₂ represents the size of the convolution kernel. In a neural network, a convolutional layer often contains many convolution kernels, and the convolution operation of the convolutional layer in the neural network can be expressed by formula (1).

[Y ₁ ,...,Y _n ]=[F ₁ *X,...,F _n *X] (1)

In the above formula (1), X represents the input data, F ₁ , F ₂ ,...F _n respectively represent n convolution kernels in the convolution layer, and * represents the convolution operation,

It is the convolution feature map output after convolution processing on the input data. H'and W'respectively represent the length and width of the output convolution feature map.

It can be seen from formula (1) that the convolution operation of a convolution layer often requires calculation of many convolution parameters. In order to reduce the parameters of the convolution kernel, a reference convolution kernel and a set of mask tensors can be used. Generate a large number of sub-convolution kernels to reduce the parameters of the convolution kernel.

The following describes the case where multiple sub-convolution kernels are obtained according to the reference convolution kernel and the binarized mask tensor (here, the binarized mask tensor is used as an example), and then the convolution operation is performed.

Suppose, the reference convolution kernel is

The binary mask tensor is

In other words, then, by performing the Hadamard product operation on the reference convolution kernel and the binarized mask tensor, multiple subconvolution kernels can be obtained, and the specific calculation process can be as shown in formula (2).

In the above formula (2), B _i represents the i-th reference convolution kernel, the value range of i is [1, k], M _j represents the j-th binarized mask tensor, and the value range of j Is [1, s], ο represents the Hadamard product operation (also called element-wise multiplication operation), and s subconvolutions can be obtained by operating a reference convolution kernel with s binarization mask tensors In this way, through k reference convolution kernels and s binary masks (the k reference convolution kernels share the s binary masks), the original convolution operation (such as formula (1 As shown in ), there are n convolution kernels for convolution calculation) The same number (k×s=n) of subconvolution kernels, use these subconvolution kernels to perform convolution calculations, and obtain the output feature maps of n channels The calculation process of is shown in formula (3).

That is to say, n sub-convolution kernels are obtained through k reference convolution kernels and s binary mask tensors, and n sub-convolution kernels are used for convolution operation, which can achieve the direct use of n convolutions in the traditional scheme The effect of convolution calculation by the kernel, and the number of parameters can be greatly reduced by using k reference convolution kernels and s binary masks. Specifically, since k is less than n, the parameter amount of the convolution kernel is reduced. In addition, the binarization mask has very low storage requirements. Compared with the convolution kernel, there are few parameters that need to be saved. Therefore, k The combination of a reference convolution kernel and s binary masks can reduce the number of parameters.

When k reference convolution kernels and s binarization mask tensors are used to obtain n sub-convolution kernels, the parameters of the convolution kernels can be compressed, and the specific parameter compression ratio can be as shown in formula (4).

In the above formula (4), r ₁ is the parameter compression ratio, k is the number of reference convolution kernels, n is the number of sub-convolution kernels, c is the number of channels of the convolution kernel, and d ₁ and d ₂ are convolutions. The size of the kernel, s is the number of binarized mask tensors.

It can be seen from formula (4) that, compared to the method of directly using n convolution kernels, the method of using k reference convolution kernels and s binarization mask tensors can achieve effective compression of convolution kernel parameters.

The second case: M=N

In the second case, the M reference convolution kernels have a one-to-one correspondence with N sets of mask tensors (this correspondence method can be called mask tensor independence).

For example, M=3, N=3, M reference convolution kernels include the first reference convolution kernel, the second reference convolution kernel, and the third reference convolution kernel. The N sets of mask tensors include the first set of masks. The tensor, the second group of mask tensors, and the third group of mask tensors, then, the correspondence between the M reference convolution kernels and the N groups of mask tensors can be shown in Table 2.

Table 2

基准卷积核Reference convolution kernel	掩码张量组Mask tensor group
第一基准卷积核The first benchmark convolution kernel	第一组掩码张量The first set of mask tensors
第二基准卷积核The second benchmark convolution kernel	第二组掩码张量The second set of mask tensors
第三基准卷积核The third benchmark convolution kernel	第三组掩码张量The third set of mask tensors

As shown in Table 2, the first reference convolution kernel corresponds to the first group of mask tensors, the second reference convolution kernel corresponds to the second group of mask tensors, and the third reference convolution kernel corresponds to the third group of masks. Code tensor. When performing convolution processing on the image to be processed according to the reference convolution kernel and mask tensor, the first reference convolution kernel and the first set of mask tensors, the second reference convolution kernel and the second set of masks can be used respectively. The tensor, the third reference convolution kernel and the third set of mask tensors are subjected to convolution processing on the image to be processed, and finally the convolution feature map of the image to be processed is obtained.

The following describes the related content of mask tensor independence in detail with the formula.

Through k reference convolution kernels and ks binarization masks, the same number of subconvolution kernels as the original convolution operation (k×s=n) are generated, and the convolution calculation process is performed according to these subconvolution kernels It can be as shown in formula (5).

Compared with the mask tensor sharing method, although the mask tensor independent method corresponds to a slightly larger amount of parameters, since each reference convolution kernel corresponds to a different set of mask tensors, the final convolution The generated features are more discriminative and discriminative.

When k reference convolution kernels and ks binarization mask tensors are used to obtain n sub-convolution kernels, the parameters of the convolution kernels can be compressed, and the specific parameter compression ratio can be as shown in formula (6).

In the above formula (6), r ₂ is the parameter compression rate, k is the number of reference convolution kernels, n is the number of sub-convolution kernels, c is the number of channels of the convolution kernel, and d ₁ and d ₂ are convolutions. The size of the kernel, ks is the number of binary mask tensors.

It can be seen from formula (6) that using k reference convolution kernels and ks binarization mask tensors to obtain n subconvolution kernels can also achieve effective compression of convolution kernel parameters.

In order to understand the situation of mask tensor independence more vividly, the following description is made with reference to FIG. 9.

As shown in Figure 9, the reference convolution kernel 1 and the reference convolution kernel 2 respectively correspond to different sets of mask tensors. The reference convolution kernel 1 corresponds to the first group of mask tensors, and the reference convolution kernel 2 corresponds to Is the second set of mask tensors. Among them, the first group of mask tensors includes mask tensor 1 and mask tensor 2, and the second group of mask tensors includes mask tensor 3 and mask tensor 4. When the subconvolution kernel is obtained, the reference convolution kernel 1 is operated with the mask tensor 1 and the mask tensor 2, and the subconvolution kernel 1 and the subconvolution kernel 2 can be obtained. The reference convolution kernel 2 and the mask tensor 2 can be obtained. Code tensor 3 and mask tensor 4 are operated to obtain sub-convolution kernel 3 and sub-convolution kernel 4.

Optionally, at least part of the mask tensors in the at least one group of mask tensors in the above N groups of mask tensors satisfy pairwise orthogonality.

When using the convolution kernel in the neural network to perform convolution processing on the input image, generally the larger the difference between different convolution kernels, the richer the features extracted by the convolution kernel, which can be relatively better processed As a result, therefore, when at least part of the mask tensors in at least one group of mask tensors in the above N groups of mask tensors satisfy pairwise orthogonality, the possibility of obtaining richer features during subsequent convolution processing can be increased. May improve the final processing effect.

Assuming that there are s binary mask tensors in a group of mask tensors, then these s binary mask tensors can be vectorized and assembled into a matrix M, in order to make any two of the s binary mask tensors A binarization mask tensor meets the requirement of pairwise orthogonality. The matrix M should be approximately an orthogonal matrix so that the convolution kernel generated from the s binarization mask tensor and the reference convolution kernel has obvious The difference. Therefore, the regular term shown in formula (7) can be added to the above s binary mask tensors:

In the above formula (7), I is an identity matrix, ||·|| _F is the Frobenius norm, d ₁ and d ₂ represent the height and width of the convolution kernel respectively, and c is the number of input channels of the convolution kernel, L _orth represents the regular term. Through the constraint of the regular term, the correlation between the above s binary mask tensors can be made small, and the convolution kernel generated according to the same reference convolution kernel is also more diverse and distinguishable.

1003. Perform convolution processing on the image to be processed according to the convolution kernel parameters of the M reference convolution kernels and the N sets of mask tensors to obtain multiple convolution feature maps of the image to be processed.

It should be understood that before step 1003, the image to be processed may be acquired first.

The foregoing image to be processed may be an image or picture to be classified. When the method shown in FIG. 7 is executed by an electronic device, the image to be processed may be an image captured by the electronic device through a camera, or the image to be processed may also be an image stored inside the electronic device (for example, the image of the electronic device). Picture in the album).

In step 1003, the image to be processed is processed to obtain multiple convolution feature maps of the image to be processed. There are many specific implementation manners. Two common methods are introduced below.

The first method: first obtain multiple convolution kernels, and then use the multiple convolution kernels to perform convolution processing on the image to be processed to obtain multiple convolution feature maps of the image to be processed.

Specifically, in the first manner, the specific process of obtaining multiple convolution feature maps of the image to be processed includes:

(1) Perform the Hadamard product operation on each reference convolution kernel in the M reference convolution kernels and a set of mask tensors corresponding to each reference convolution kernel in the N groups of mask tensors to obtain multiple subvolumes product;

(2) Perform convolution processing on the image to be processed according to multiple sub-convolution kernels to obtain multiple convolution feature maps.

The second way: first perform convolution processing on the image to be processed according to M reference convolution kernels to obtain M reference convolution feature maps, and then obtain the to be processed according to M reference convolution feature maps and N sets of mask tensors Multiple convolution feature maps of the image.

Specifically, in the second way, the specific process of obtaining multiple convolution feature maps of the image to be processed includes:

(3) Perform convolution processing on the image to be processed according to M reference convolution kernels to obtain M reference convolution feature maps of the image to be processed;

(4) Perform Hadamard product operation on M reference convolution feature maps and N groups of mask tensors to obtain multiple convolution feature maps of the image to be processed.

Using the second method can reduce the number of convolution calculations. When there are M reference convolution kernels, only M convolution calculations are required, instead of generating M*N convolution kernels and then performing M*N convolutions. Product operations are likely to reduce the complexity of operations as a whole and improve data processing efficiency.

It should be understood that the aforementioned reference convolution feature map refers to a convolution feature map obtained by performing convolution processing on an image to be processed using a reference convolution kernel.

The above-mentioned second calculation method may also be referred to as an efficient forward calculation method. In this method, by advancing the convolution calculation and using the reference convolution kernel to perform the convolution calculation, the calculation amount of the convolution calculation can be reduced. The following describes the reduction of the calculation amount of the convolution calculation in the second method in combination with a specific formula.

For an image block

In other words, when the traditional convolution calculation method is used for calculation, the image block needs to be multiplied by the elements of each convolution kernel, and then summed, as shown in formula (8).

Among them, in the above formula (8), F ₁ to F _n represent n convolution kernels, X represents the image block to be processed, ο represents the element multiplication operation, Y represents the convolution feature map obtained by convolution, assuming F ₁ to F _n are the corresponding convolution kernel parameters are c×d ₁ ×d ₂ , then, the traditional convolution process shown in formula (8) includes ncd ₁ d ₂ multiplications and ncd ₁ d ₂ additions .

Using the reference convolution kernel and the mask tensor to obtain multiple subconvolution kernels, and then using the multiple subconvolution kernels to perform convolution processing on the image block, the calculation process can be as shown in formula (9). The image block to be processed requires Multiply the elements of each subconvolution kernel, and then add them.

Among them, in the above formula (9), F ₁₁ to F _ks are multiple sub-convolution kernels, X represents the image block to be processed, ο represents the element multiplication operation, Y represents the convolution feature map obtained by convolution, B _i Represents the i-th reference convolution kernel, and M _j represents the j-th mask tensor.

It can be seen from the above formula (9) that the multiplication of the image block and the element of the reference convolution kernel XοB _{i has} been repeatedly calculated s times, but in fact it only needs to be calculated once and the calculation result is cached. The intermediate result of the cache is

In this way, formula (9) can be simplified to formula (10).

When M _j is a binarized mask tensor, C _i οM _j here can be implemented by a masking operation that takes very little time. Efficient forward calculation above comprising kcd ₁ d ₂ multiplications, ncd ₁ d ₂ is calculated and additions can be ignored ncd ₁ d ₂ times masking. Compared with the traditional convolution operation, the reduction ratio of the reference convolution kernel to the multiplication operation is r ₂ =s, which greatly reduces the number of multiplication operations and reduces the computational complexity.

1004. Classify the image to be processed according to multiple convolution feature maps of the image to be processed to obtain a classification result of the image to be processed.

Optionally, the foregoing classifying the image to be processed according to the multiple convolution feature maps to obtain the classification result of the image to be processed includes: splicing the multiple convolution feature maps to obtain the target volume Product feature map; classify the image lines to be processed according to the target convolution feature map to obtain the classification result of the image to be processed.

Specifically, compared with the elements in the parameters of the reference convolution kernel, the storage space occupied by the elements in the mask tensor is smaller. Therefore, the subconvolution kernel is obtained by combining the reference convolution kernel and the mask tensor. , The number of convolution kernel parameters is reduced, and the compression of convolution kernel parameters is realized, so that the neural network can be deployed on some devices with limited storage resources to perform image classification tasks.

The following analyzes specific reasons why the neural network model corresponding to the image classification method of the embodiment of the present application can reduce storage overhead. For a convolutional layer in a neural network, the parameter of its convolution kernel is nⅹcⅹd ₁ ⅹd ₂ , where n is the number of convolution kernels contained in the convolution layer, and c is the number of channels of the convolution kernel , D ₁ and d ₂ are the height and width of the convolution kernel, respectively. The calculation amount for the convolutional layer to perform convolution calculation on an input image is hⅹwⅹnⅹcⅹd ₁ ⅹd ₂ multiplications and additions, where h and w represent the height and width of the convolution feature map output by the convolution layer.

Due to the parameter redundancy between n convolution kernels in a convolutional layer, while keeping the input feature and output feature dimension of the convolutional layer constant, you can consider using a small number of k reference convolution kernels (k less than n) and a binarization mask with extremely low storage requirements. Through the pairwise combination of the reference convolution kernel and the mask tensor, n sub-convolution kernels (k<n) can be derived. Among them, the sub-convolution kernel The parameters all come from the reference convolution kernel and the binarization mask, so that the parameter amount of the convolution kernel can be reduced, and the storage overhead caused by saving the convolution kernel parameters during neural network deployment can be reduced.

In order to illustrate the image classification method of the embodiment of the present application more vividly, the entire process of the embodiment of the present application will be introduced below with reference to FIG. 10. As shown in Figure 10, the reference convolution kernel and the mask tensor can be operated to obtain the sub-convolution kernels of the neural network. These sub-convolution kernels can perform on the input picture (the input picture is a picture of a cat) After processing, the convolution feature map of the input picture is obtained. Next, the neural network classifier can be used to process the convolution feature map of the input picture to obtain the probability that the input picture belongs to each type of picture (the input picture belongs to the cat The highest probability), then the category with the probability value greater than a certain value can be determined as the category of the input picture (because the input picture has the highest probability of being a cat, the category of the input picture can be determined as cat), and the input picture The category information is output.

It can be seen from Figure 10 that for the neural network, it is only necessary to save the convolution kernel parameters and mask tensor of the reference convolution kernel, and then many subconvolution kernels can be obtained through subsequent derivation, instead of saving each subconvolution kernel. Parameters can save the storage space occupied by neural network deployment or application, and it is convenient to deploy the neural network on some devices with limited storage resources, and then realize the classification or recognition of images on these devices.

Still taking the processing process shown in Figure 10 as an example, the neural network based on the reference convolution kernel includes N layers of convolutional layers (Figure 10 shows one of them), assuming that the original convolutional layer in the neural network contains a total of 16 ordinary convolution kernels with a size of 3*7*7. Then, when the mask tensor is used for processing independently, the convolutional layer may need 4 3*7*7 full-stack convolution kernels and 16 3*7*7 binarized mask tensors . In this way, each reference convolution kernel can be element-level multiplied by the corresponding 4 mask tensors to obtain 4 subconvolution kernels. In this way, a total of 16 sub-convolution kernels can be generated based on the 4 sub-convolution kernels, which are used to replace the 16 ordinary convolution kernels of the original network. In this case, the parameter quantity of the full-stack convolution kernel of this layer is 4*3*7*7=588, the parameter quantity of the binary mask tensor is 16*3*7*7/32=73.5, and the total parameter quantity It is 588+73.5=661.5. The parameter amount of the convolution layer using the common convolution kernel is 16*3*7*7=2352, and the parameter amount is compressed by 2352/661.5=3.56 times, thereby realizing effective compression of the parameters.

It should be understood that the image classification method shown in FIG. 7 can be applied to the scene shown in FIG. 4. Specifically, after the image to be shot is obtained through the self-portrait of the mobile phone, the image to be shot can be classified according to the method shown in FIG. 7. After the classification result is obtained, the prediction frame is generated on the shooting interface according to the image classification result, which is convenient for the user to perform Better shooting.

The image classification method shown in Fig. 7 can be applied in an autonomous driving scene. The image classification method shown in Fig. 7 is used to classify the road pictures captured during the driving process of the vehicle to identify objects of different categories, and then obtain the road image Semantic segmentation results.

Optionally, the convolution kernel parameters of the reference convolution kernel and the mask tensor in the reference convolution kernel parameter library are obtained by training the neural network according to the training image.

In order to better understand the process of obtaining the convolution kernel parameters of the reference convolution kernel and the mask tensor, the process of obtaining the convolution kernel parameters of a reference convolution kernel and a set of mask tensors will be described below in conjunction with Figure 11. .

FIG. 11 is a schematic diagram of the process of obtaining convolution kernel parameters and mask tensor of the reference convolution kernel.

The process shown in FIG. 11 includes steps S1 to S6, and the convolution kernel parameters of the reference convolution kernel and the mask tensor parameters can be obtained through the steps S1 to S6.

These steps are described in detail below.

S1. Initialize the reference convolution kernel and mask tensor.

It should be understood that the convolution kernel parameters of a reference convolution kernel and the value of each element in the corresponding set of mask tensors can be initialized in S1. Through the initialization operation, the first reference convolution kernel and the first reference convolution kernel shown in FIG. 11 can be obtained. A set of mask tensors. Among them, the first group of mask tensors includes mask tensor 1, mask tensor 2, and mask tensor 3 (not shown in FIG. 11).

S2. Generate a subconvolution kernel according to the first reference convolution kernel and the first set of mask tensors.

Among them, in S2, the subconvolution kernel generated according to the first reference convolution kernel and the first group of mask tensors specifically includes the subconvolution kernel A, the subconvolution kernel B, and the subconvolution kernel C.

Specifically, in S2, the subconvolution kernel A can be generated according to the first reference convolution kernel and the mask tensor 1, and the subconvolution kernel B can be generated according to the first reference convolution kernel and the mask tensor 2. A reference convolution kernel and mask tensor 3 generate subconvolution kernel C.

Among them, the aforementioned sub-convolution kernel A, sub-convolution kernel B, and sub-convolution kernel C are essentially convolution kernels in a neural network, and are used to perform convolution processing on input data.

S3. Use the sub-convolution kernel to process the input data to obtain a convolution feature map of the input data.

Specifically, in S3, the subconvolution kernel A performs convolution processing on the input data to obtain a feature map A, and the subconvolution kernel B performs convolution processing on the input data to obtain a feature map B, and the subconvolution kernel C performs a convolution process on the input data. The data is subjected to convolution processing to obtain feature maps C respectively.

The aforementioned input data may specifically be an image to be processed.

In addition, when obtaining the convolution feature map of the input data, the first convolution kernel can also be used to process the input data to obtain the initial convolution feature map, and then the feature can be generated based on the initial feature map and the mask tensor 1. In Figure A, a feature map B is generated based on the initial feature map and the mask tensor 2, and a feature map C is generated based on the initial feature map and the mask tensor 3. Using this method can reduce the number of convolution operations and reduce the amount of operations.

S4. Splicing the feature map A, the feature map B, and the feature map C to obtain a spliced feature map.

S5. Determine whether the preset loss function converges according to the splicing feature map.

When it is determined in S5 that the loss function has not converged, it means that the training of the neural model has met the requirements, and then S6 can be executed.

S6. Update the convolution kernel parameters of the first reference convolution kernel and/or the parameters in the first group of mask tensors according to a certain gradient.

In S6, the parameters of the convolution kernel of the first reference convolution kernel and the gradient of the parameters of the first group of mask tensors can be determined according to parameters such as the learning rate. After S6 is executed, S2 to S5 can be executed repeatedly until the preset loss function converges.

When it is determined in S5 that the loss function has converged, it means that the training of the neural model has met the requirements, and S7 can be executed next.

S7. Obtain the convolution kernel parameters of the first reference convolution kernel and the parameters of the mask tensor in the first group of mask tensors.

It should be understood that, for ease of understanding and explanation, the above-mentioned Figure 11 only takes one reference convolution kernel and a set of mask tensors as an example for illustration. When there are multiple reference convolution kernels and multiple sets of mask tensors, it is also The process shown in Figure 11 can be used to determine the convolution kernel parameters of the reference convolution kernel and the parameters of the mask tensor, but it is necessary to initialize the convolution kernel parameters of multiple reference convolution kernels and multiple sets of mask tensors during initialization. When updating the parameters, it is also necessary to update the convolution kernel parameters of multiple reference convolution kernels and/or the parameters of multiple mask tensors.

In the process of neural network training, it is necessary to perform convolution calculation and calculate the loss function corresponding to the neural network model. When the loss function converges, the convolution kernel parameters and mask tensor of the corresponding reference convolution kernel are finally obtained Baseline convolution kernel parameter kernel mask tensor. These processes are described in detail below in conjunction with formulas.

The convolution operation can be realized by matrix multiplication. Specifically, before the convolution calculation, the input feature map can be divided into l=H×W blocks (each block size is d ₁ × d ₂ × c) , And vectorize these blocks, the corresponding vector of these small blocks is shown in formula (11).

Similarly, the output feature map can be vectorized, and the result obtained is as shown in formula (12), and all subconvolution kernels can also be vectorized, and the result obtained is as shown in formula (13).

Here, taking the case of mask tensor sharing as an example, there are two variables that need to be optimized, and these two variables are shown in formula (14) and formula (15) respectively.

Among them, B is the reference convolution kernel and M is the mask tensor. Specifically, the reference convolution kernel includes B ₁ , ..., B _k , and the mask tensor includes M ₁ , ..., M _k .

The convolution operation of the reference convolution kernel can be expressed by formula (16).

The objective function of the neural network based on the above-mentioned reference convolution kernel is shown in formula (17).

minL=L ₀ (B,M)+λL _ortho (M) (17)

Among them, L ₀ is the loss function related to the task, such as the cross entropy loss of the classification task, η is the learning rate, and L _ortho (M) is the orthogonal loss function. As shown in formulas (18) and (19), we can calculate the gradient of the two variables through the standard backpropagation algorithm.

Next, B can be updated according to formula (20).

When updating M, since it is binarized, gradient descent cannot be directly applied. Therefore, a proxy variable M can be defined first, as shown in formula (21).

M=sin(H) (21)

Next, calculate the gradient according to formula (22), update the variable H according to formula (23), and update M indirectly by updating the variable H.

After each update of B and M, it can be determined whether formula (17) converges. If formula (17) does not converge, continue to update B and M, and then calculate formula (17). If formula (17) converges, the corresponding B and M are the final parameters to be determined.

The image classification method of the embodiment of the present application is described in detail above with reference to FIGS. 7 to 11, and the data processing method of the embodiment of the present application is described below with reference to FIG. 12.

FIG. 12 is a schematic flowchart of a data processing method according to an embodiment of the present application. The method shown in FIG. 12 may be executed by a data processing device, which may be an electronic device with data processing (especially multimedia data processing) functions. The electronic device may specifically be a mobile terminal (for example, a smart phone), a computer, a personal digital assistant, a wearable device, a vehicle-mounted device, an Internet of Things device or other devices capable of image processing.

The method shown in FIG. 12 includes steps 2001 to 2004, which are respectively introduced below.

2001. Obtain the convolution kernel parameters of the M reference convolution kernels of the neural network.

Wherein, the above M is a positive integer.

2002. Obtain N groups of mask tensors of the neural network.

Among them, the above N is a positive integer, each mask tensor in the N groups of mask tensors is composed of multiple mask tensors, and the number of bits occupied by the elements in the N groups of mask tensors is less than that of the M reference convolution kernels. The number of bits occupied by the elements in the convolution kernel parameters when storing, each of the M reference convolution kernels corresponds to a group of mask tensors in the N groups of mask tensors.

It should be understood that the execution process of the

above steps

2001 and 2002 is the same as the execution process of step 1001 and step 1002 in the method shown in FIG. To avoid unnecessary repetition, the introduction will not be repeated here.

2003. Perform convolution processing on multimedia data according to M reference convolution kernels and N groups of mask tensors to obtain multiple convolution feature maps of the multimedia data.

The process of obtaining multiple convolutional feature maps of multimedia data in step 2003 is similar to step 1003 in the method shown in FIG. 7. The main difference is that step 1003 is to perform convolution processing on the image to be processed, and step 2003 is to Multimedia data is processed. Therefore, the specific processing procedure of step 2003 can refer to the processing procedure of step 1003 above.

2004. Process multimedia data according to multiple convolution feature maps of multimedia data.

The above-mentioned multimedia data may be text, sound, picture (image), video, animation, etc.

Specifically, when the foregoing multimedia data is an image to be processed, the multimedia data may be identified or classified according to multiple convolution feature maps.

Or, when the multimedia data is an image to be processed, the multimedia data can be image processed according to multiple convolution feature maps. For example, convolution processing is performed on the acquired face image to obtain a convolution feature map of the face image, and then the convolution feature map of the face image is processed to generate an animated expression corresponding to the facial expression. Alternatively, other expressions can also be transferred to the input face image and then output.

The data processing method shown in Figure 12 can be applied to the scene shown in Figure 5. At this time, the multimedia data is a face image. By convolution processing on the face image, the convolution feature map of the face image can be obtained. Next, by comparing the convolution feature map of the face image with the convolution feature map corresponding to the corresponding ID document, the identity of the person being photographed can be determined.

In order to verify the effect of using the reference convolution kernel and mask tensor to reduce storage overhead in the embodiments of the present application. The following uses the ImageNet data set to test the effect of the benchmark convolution kernel of the embodiment of the application. Here, the CNN using the full-stack convolution kernel is called minimum viable networks (MVnet). Table 1 shows the test results of the image classification methods of the embodiments of the present application using the standard models VGG-16, ResNet-50 and ImageNet datasets.

When testing the effects of using the reference convolution kernel and mask tensor in the embodiments of this application, the structure of the existing neural network model (the number of layers, the size of the convolution kernel of each layer, parameters, etc.) is not changed, but only According to the calculation method of the reference convolution kernel proposed in this application, the number of convolution kernels in each layer is reduced.

Table 3 shows the results of the application using the benchmark convolution kernel on the ImageNet 2012 data set, where MVNet-A represents the CNN that uses the reference convolution kernel shared by the mask tensor, and MVNet-B represents the use of the mask. Code tensor independent reference convolution kernel CNN, s in parentheses represents the number of mask tensors.

table 3

As shown in Table 3, under the VGG-16 model, the top 1 prediction error rate and the top 5 prediction error rate corresponding to the reference convolution kernel shared by the mask tensor or the reference convolution kernel independent of the mask tensor It is basically the same as the previous method, but the corresponding parameter amount and corresponding memory overhead are significantly reduced. In particular, the memory overhead reduced by the reference convolution kernel sharing the mask tensor is more obvious.

Under the ResNet-50 model, whether it is a reference convolution kernel shared by the mask tensor or a reference convolution kernel independent of the mask tensor, the corresponding parameter amount and memory overhead are also significantly reduced. At the same time, the first 1 The prediction error rate and the top 5 prediction error rates are basically consistent with the previous method.

In the last two rows of Table 3, when the mask tensors are independent, when a smaller reference convolution kernel and more mask tensors are used, the corresponding parameter amount and memory overhead are further reduced.

Table 3 mainly shows the effect of reducing the storage overhead after replacing the traditional convolution kernel in the existing deep convolutional neural network model with the reference convolution kernel proposed in this application.

In addition, in Table 3, MV Net-A (s=4), MV Net-B (s=4), MV Net-A (s=4), MV Net-B (s=4) and MV Net- B(s=32) all adopt the method of forward calculation (first use the reference convolution kernel to convolve the image to be processed, and then combine the mask tensor to obtain the convolution feature map of the image to be processed) to obtain the convolution Feature map. It can be seen from Table 3 that in these cases, the amount of multiplication is greatly reduced, which has the effect of reducing the amount of calculation.

In addition, in Table 3 above, the first column respectively represents different methods or architectures. Among them, the links to papers corresponding to related methods or architectures are as follows:

BN low-rank: https://arxiv.org/pdf/1511.06067.pdf

ThiNet-Conv, ThiNet-30: http://openaccess.thecvf.com/content_ICCV_2017/papers/Luo_ThiNet_A_Filter_ICCV_2017_paper.pdf

ShiftResNet: http://openaccess.thecvf.com/content_cvpr_2018/papers/Wu_Shift_A_Zero_CVPR_2018_paper.pdf

Versatile-v2: https://papers.nips.cc/paper/7433-learning-versatile-filters-for-efficient-convolutional-neural-networks

In fact, it is also possible to embed the reference convolution kernel proposed in this application into some lightweight deep convolutional neural network models to verify the effect of reducing the amount of parameters and memory overhead. As shown in Table 4, by embedding the reference convolution kernel and mask tensor provided in this application into MobileNet, replacing the traditional convolution kernel, and training on the ImageNet data set. Although most of the convolution kernels in MobileNet are 1x1 in size, applying the benchmark convolution kernel proposed in this application can still reduce its memory and computational overhead by nearly half.

Table 4

方法method	内存 RAM	乘法量Multiplier		前1预测错误率(％)Top 1 prediction error rate (%)
MobileNet-v1MobileNet-v1	16.116.1	569569	29.429.4
MV Net-B(s＝2,MobileNet-v1)MV Net-B(s=2,MobileNet-v1)	10.510.5	299299	29.929.9
MobileNet-v2MobileNet-v2	13.213.2	300300	28.228.2
MV Net-B(s＝4,MobileNet-v2)MV Net-B(s=4,MobileNet-v2)	7.57.5	9393	29.929.9

As shown in Table 4, MV Net-B (s=2, MobileNet-v1) embeds the reference convolution kernel on the original structure MobileNet-v1. Among them, s=2 means that a set of mask tensors contains The number of mask tensors, MV Net-B (s=2, MobileNet-v1) Compared with MobileNet-v1, the memory and multiplication amount are significantly reduced. MV Net-B (s=2, MobileNet-v2) embeds the reference convolution kernel on the original structure MobileNet-v2, where s=2 represents the number of mask tensors contained in a set of mask tensors Compared with MobileNet-v2, MV Net-B (s=2, MobileNet-v2) also has a significant reduction in memory and multiplication (memory is reduced by almost half).

In addition, in Table 4, MV Net-B (s=2, MobileNet-v1) and MV Net-B (s=2, MobileNet-v2) both use the forward calculation method (first use the reference convolution kernel to treat The processed image is subjected to convolution processing, and then combined with the mask tensor to obtain the convolution feature map of the image to be processed) to obtain the convolution feature map.

Among them, MV Net-B (s=2, MobileNet-v1) compared with the traditional calculation method of MobileNet-v1 (using each sub-convolution kernel to process the image to be processed to obtain the convolution feature map), the amount of multiplication has dropped by nearly half. Compared with the traditional calculation method of MobileNet-v2, MV Net-B (s=2, MobileNet-v2) reduces the amount of multiplication by more than three times.

It can be seen that after embedding the reference convolution kernel proposed in the embodiments of this application into some lightweight deep convolutional neural network models, the effect of reducing storage overhead is very obvious. In addition, when the reference convolution kernel is combined with the forward calculation The effect of reducing the amount of calculation is also obvious when performing calculations.

It should be understood that when comparing the test results in Table 3 and Table 4 above, the number of reference convolution kernels and the number of mask tensor groups in each case are not given. This is mainly because of the benchmark in each case The number of convolution kernels and the number of mask tensors need to be determined according to the network architecture of the specific application.

FIG. 13 is a schematic diagram of the hardware structure of a neural network training device provided by an embodiment of the present application. The neural network training device 3000 shown in FIG. 13 (the device 3000 may specifically be a computer device) includes a memory 3001, a processor 3002, a communication interface 3003, and a bus 3004. Among them, the memory 3001, the processor 3002, and the communication interface 3003 implement communication connections between each other through the bus 3004.

The memory 3001 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 3001 may store a program. When the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 and the communication interface 3003 are used to execute each step of the neural network training method of the embodiment of the present application.

The processor 3002 may adopt a general-purpose CPU, a microprocessor, an application specific integrated circuit (application specific integrated circuit, ASIC), a graphics processing unit (GPU), or one or more integrated circuits for executing related programs. Implement the functions required by the units in the neural network training device of the embodiment of the application, or execute the neural network training method of the method embodiment of the application.

The processor 3002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the neural network training method of the present application can be completed by hardware integrated logic circuits in the processor 3002 or instructions in the form of software. The aforementioned processor 3002 may also be a general-purpose processor, a digital signal processing (digital signal processing, DSP), an application specific integrated circuit (ASIC), a ready-made programmable gate array (field programmable gate array, FPGA) or other programmable logic devices , Discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the memory 3001, and the processor 3002 reads the information in the memory 3001, and combines its hardware to complete the functions required by the units included in the neural network training device of the embodiment of the application, or perform the functions of the method embodiment of the application. Training method of neural network.

The communication interface 3003 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 3000 and other devices or communication networks. For example, training data (such as the original image in the embodiment of the present application and the noise image obtained after adding noise to the original image) can be obtained through the communication interface 3003.

The bus 3004 may include a path for transferring information between various components of the device 3000 (for example, the memory 3001, the processor 3002, and the communication interface 3003).

FIG. 14 is a schematic diagram of the hardware structure of an image classification device according to an embodiment of the present application. The image classification device 4000 shown in FIG. 14 includes a memory 4001, a processor 4002, a communication interface 4003, and a bus 4004. Among them, the memory 4001, the processor 4002, and the communication interface 4003 implement communication connections between each other through the bus 4004.

The memory 4001 may be ROM, static storage device and RAM. The memory 4001 may store a program. When the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 and the communication interface 4003 are used to execute each step of the image classification method of the embodiment of the present application.

The processor 4002 may adopt a general-purpose CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits to execute related programs to realize the functions required by the units in the image classification device of the embodiment of the present application. Or execute the image classification method in the method embodiment of this application.

The processor 4002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image classification method in the embodiment of the present application can be completed by an integrated logic circuit of hardware in the processor 4002 or instructions in the form of software. The aforementioned processor 4002 may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the memory 4001, and the processor 4002 reads the information in the memory 4001, and combines its hardware to complete the functions required by the units included in the image classification apparatus of the embodiment of the present application, or perform the image classification of the method embodiment of the present application method.

The communication interface 4003 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 4000 and other devices or a communication network. For example, the training data can be obtained through the communication interface 4003.

The bus 4004 may include a path for transferring information between various components of the device 4000 (for example, the memory 4001, the processor 4002, and the communication interface 4003).

It should be noted that although the devices 3000 and 4000 only show a memory, a processor, and a communication interface, in a specific implementation process, those skilled in the art should understand that the devices 3000 and 4000 also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the devices 3000 and 4000 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the devices 3000 and 4000 may also only include the necessary components for implementing the embodiments of the present application, and not necessarily all the components shown in FIG. 13 or FIG. 14.

FIG. 15 is a schematic diagram of the hardware structure of a data processing device according to an embodiment of the present application. The data processing device 5000 shown in FIG. 15 is similar to the image classification device 4000 in FIG. 14. The data processing device 5000 includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. Among them, the memory 5001, the processor 5002, and the communication interface 5003 implement communication connections between each other through the bus 5004.

The memory 5001 may be ROM, static storage device and RAM. The memory 5001 may store a program. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 and the communication interface 5003 are used to execute each step of the image classification method of the embodiment of the present application.

The processor 5002 may adopt a general-purpose CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits to execute related programs to realize the functions required by the units in the image classification device of the embodiment of the present application. Or execute the data processing method in the method embodiment of this application.

The foregoing descriptions of the modules and units inside the image classification device 4000 shown in FIG. 14 are also applicable to the modules and units inside the data processing device 5000 in FIG. 15. In order to avoid unnecessary repetition, relevant descriptions are omitted here. .

It can be understood that the foregoing device 3000 is equivalent to the training device 120 in 1, and the foregoing device 4000 and the device 5000 are equivalent to the execution device 110 in FIG. 1.

In addition, the above-mentioned device 4000 may specifically be an electronic device with an image classification function, and the above-mentioned device 5000 may specifically be an electronic device with a data processing (especially multimedia data processing) function. The electronic device here may specifically move a terminal (for example, a smart phone). ), computers, personal digital assistants, wearable devices, in-vehicle devices, IoT devices, etc.

A person of ordinary skill in the art may be aware that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the above-described system, device, and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

An image classification method, characterized in that it includes:

Obtain the convolution kernel parameters of the M reference convolution kernels of the neural network, where M is a positive integer;

Obtain N groups of mask tensors of the neural network, where N is a positive integer, each group of mask tensors in the N groups of mask tensors is composed of multiple mask tensors, and the elements in the N groups of mask tensors The number of bits occupied during storage is less than the number of bits occupied during storage of elements in the convolution kernel parameters in the M reference convolution kernels. Each reference convolution kernel of the M reference convolution kernels corresponds to the N sets of masks. A set of mask tensors in the code tensor;

Perform a Hadamard product operation on each reference convolution kernel in the M reference convolution kernels and a set of mask tensors corresponding to each reference convolution kernel in the N groups of mask tensors to obtain Multiple sub-convolution kernels;

Performing convolution processing on the image to be processed respectively according to the multiple sub-convolution kernels to obtain multiple convolution feature maps;

The image to be processed is classified according to the multiple convolution feature maps to obtain a classification result of the image to be processed.
The method according to claim 1, wherein N is less than M, and at least two reference convolution kernels in the M reference convolution kernels correspond to a group of mask tensors in the N groups of mask tensors.
The method according to claim 1 or 2, wherein at least part of the mask tensors in the at least one group of mask tensors in the N groups of mask tensors satisfy pairwise orthogonality.
An image classification method, characterized in that it includes:

Obtain the convolution kernel parameters of the M reference convolution kernels of the neural network, where M is a positive integer;

Obtain N groups of mask tensors of the neural network, where N is a positive integer, each group of mask tensors in the N groups of mask tensors is composed of multiple mask tensors, and the elements in the N groups of mask tensors The number of bits occupied during storage is less than the number of bits occupied during storage of elements in the convolution kernel parameters in the M reference convolution kernels. Each reference convolution kernel of the M reference convolution kernels corresponds to the N sets of masks. A set of mask tensors in the code tensor;

Performing convolution processing on the image to be processed according to the M reference convolution kernels to obtain M reference convolution feature maps of the image to be processed;

Performing a Hadamard product operation on the M reference convolution feature maps and the N groups of mask tensors to obtain multiple convolution feature maps of the image to be processed;

The image to be processed is classified according to a plurality of convolution feature maps of the image to be processed to obtain a classification result of the image to be processed.
The method according to claim 4, wherein N is less than M, and at least two reference convolution kernels in the M reference convolution kernels correspond to a group of mask tensors in the N groups of mask tensors.
The method according to claim 4 or 5, wherein at least part of the mask tensors in the at least one group of mask tensors in the N groups of mask tensors satisfy pairwise orthogonality.
A data processing method, characterized by comprising:

Obtain the convolution kernel parameters of the M reference convolution kernels of the neural network, where M is a positive integer;

Obtain N groups of mask tensors of the neural network, where N is a positive integer, each group of mask tensors in the N groups of mask tensors is composed of multiple mask tensors, and the elements in the N groups of mask tensors The number of bits occupied during storage is less than the number of bits occupied during storage of elements in the convolution kernel parameters in the M reference convolution kernels. Each reference convolution kernel of the M reference convolution kernels corresponds to the N sets of masks. A set of mask tensors in the code tensor;

Perform a Hadamard product operation on each reference convolution kernel in the M reference convolution kernels and a set of mask tensors corresponding to each reference convolution kernel in the N groups of mask tensors to obtain Multiple sub-convolution kernels;

Performing convolution processing on the multimedia data respectively according to the multiple sub-convolution kernels to obtain multiple convolution feature maps of the multimedia data;

The multimedia data is processed according to multiple convolution feature maps of the multimedia data.
A data processing method, characterized by comprising:

Obtain the convolution kernel parameters of the M reference convolution kernels of the neural network, where M is a positive integer;

Obtain N groups of mask tensors of the neural network, where N is a positive integer, each group of mask tensors in the N groups of mask tensors is composed of multiple mask tensors, and the elements in the N groups of mask tensors The number of bits occupied during storage is less than the number of bits occupied during storage of elements in the convolution kernel parameters in the M reference convolution kernels. Each reference convolution kernel of the M reference convolution kernels corresponds to the N sets of masks. A set of mask tensors in the code tensor;

Performing convolution processing on multimedia data according to the M reference convolution kernels to obtain M reference convolution feature maps of the multimedia data;

Performing a Hadamard product operation on the M reference convolution feature maps and the N groups of mask tensors to obtain multiple convolution feature maps of the multimedia data;

The multimedia data is processed according to multiple convolution feature maps of the multimedia data.
An image classification device, characterized by comprising:

The memory is used to store the convolution kernel parameters of the M reference convolution kernels of the neural network and N groups of mask tensors, where M and N are both positive integers, and each group of mask tensors in the N groups of mask tensors The quantity is composed of multiple mask tensors. The number of bits occupied by the elements in the N groups of mask tensors when stored is less than the number of bits occupied by the elements in the convolution kernel parameters in the M reference convolution kernels. The M Each of the reference convolution kernels corresponds to a group of mask tensors in the N groups of mask tensors;

The processor is configured to obtain the convolution kernel parameters and N groups of mask tensors of the M reference convolution kernels of the neural network, and perform the following operations:

Perform a Hadamard product operation on each reference convolution kernel in the M reference convolution kernels and a set of mask tensors corresponding to each reference convolution kernel in the N groups of mask tensors to obtain Multiple sub-convolution kernels;

Performing convolution processing on the image to be processed respectively according to the multiple sub-convolution kernels to obtain multiple convolution feature maps;

The image to be processed is classified according to the multiple convolution feature maps to obtain a classification result of the image to be processed.
The apparatus according to claim 9, wherein N is less than M, and at least two reference convolution kernels in the M reference convolution kernels correspond to a group of mask tensors in the N groups of mask tensors.
The device according to claim 9 or 10, wherein at least part of the mask tensors in the at least one group of mask tensors in the N groups of mask tensors satisfy pairwise orthogonality.
An image classification device, characterized by comprising:

The memory is used to store the convolution kernel parameters of the M reference convolution kernels of the neural network and N groups of mask tensors, where M and N are both positive integers, and each group of mask tensors in the N groups of mask tensors The quantity is composed of multiple mask tensors. The number of bits occupied by the elements in the N groups of mask tensors when stored is less than the number of bits occupied by the elements in the convolution kernel parameters in the M reference convolution kernels. The M Each of the reference convolution kernels corresponds to a group of mask tensors in the N groups of mask tensors;

The processor is configured to obtain the convolution kernel parameters and N groups of mask tensors of the M reference convolution kernels of the neural network, and perform the following operations:

Performing convolution processing on the image to be processed according to the M reference convolution kernels to obtain M reference convolution feature maps of the image to be processed;

Performing a Hadamard product operation on the M reference convolution feature maps and the N groups of mask tensors to obtain multiple convolution feature maps of the image to be processed;

The image to be processed is classified according to a plurality of convolution feature maps of the image to be processed to obtain a classification result of the image to be processed.
The device according to claim 12, wherein N is less than M, and at least two reference convolution kernels in the M reference convolution kernels correspond to a group of mask tensors in the N groups of mask tensors.
The device according to claim 12 or 13, wherein at least part of the mask tensors in the at least one group of mask tensors in the N groups of mask tensors satisfy pairwise orthogonality.
A data processing device, characterized by comprising:

The memory is used to store the convolution kernel parameters of the M reference convolution kernels of the neural network and N groups of mask tensors, where M and N are both positive integers, and each group of mask tensors in the N groups of mask tensors The quantity is composed of multiple mask tensors. The number of bits occupied by the elements in the N groups of mask tensors when stored is less than the number of bits occupied by the elements in the convolution kernel parameters in the M reference convolution kernels. The M Each of the reference convolution kernels corresponds to a group of mask tensors in the N groups of mask tensors;

The processor is configured to obtain the convolution kernel parameters and N groups of mask tensors of the M reference convolution kernels of the neural network, and perform the following operations:

Perform a Hadamard product operation on each reference convolution kernel in the M reference convolution kernels and a set of mask tensors corresponding to each reference convolution kernel in the N groups of mask tensors to obtain Multiple sub-convolution kernels;

Performing convolution processing on the multimedia data respectively according to the multiple sub-convolution kernels to obtain multiple convolution feature maps of the multimedia data;

The multimedia data is processed according to multiple convolution feature maps of the multimedia data.
A data processing device, characterized by comprising:

The memory is used to store the convolution kernel parameters of the M reference convolution kernels of the neural network and N groups of mask tensors, where M and N are both positive integers, and each group of mask tensors in the N groups of mask tensors The quantity is composed of multiple mask tensors. The number of bits occupied by the elements in the N groups of mask tensors when stored is less than the number of bits occupied by the elements in the convolution kernel parameters in the M reference convolution kernels. The M Each of the reference convolution kernels corresponds to a group of mask tensors in the N groups of mask tensors;

The processor is configured to obtain the convolution kernel parameters and N groups of mask tensors of the M reference convolution kernels of the neural network, and perform the following operations:

Performing convolution processing on multimedia data according to the M reference convolution kernels to obtain M reference convolution feature maps of the multimedia data;

Performing a Hadamard product operation on the M reference convolution feature maps and the N groups of mask tensors to obtain multiple convolution feature maps of the multimedia data;

The multimedia data is processed according to multiple convolution feature maps of the multimedia data.
A computer-readable storage medium, wherein the computer-readable medium stores a program code for device execution, and the program code includes a method for executing the method according to any one of claims 1-8.
A chip, characterized in that the chip comprises a processor and a data interface, and the processor reads the instructions stored on the memory through the data interface to execute the method according to any one of claims 1-8 method.