CN111247527B - Method and device for determining characteristic images in convolutional neural network model - Google Patents
Method and device for determining characteristic images in convolutional neural network model Download PDFInfo
- Publication number
- CN111247527B CN111247527B CN201780096076.0A CN201780096076A CN111247527B CN 111247527 B CN111247527 B CN 111247527B CN 201780096076 A CN201780096076 A CN 201780096076A CN 111247527 B CN111247527 B CN 111247527B
- Authority
- CN
- China
- Prior art keywords
- convolution kernels
- convolution
- group
- kernels
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
Abstract
A method and a device for determining characteristic images in a convolutional neural network model belong to the technical field of model training. The method comprises the following steps: acquiring a plurality of input images of a target processing layer in a convolutional neural network model (S210); acquiring at least one group of convolution kernels of the target processing layer (S220), wherein different convolution kernels in the same group contain the same elements and have different arrangement sequences of the elements; based on each convolution kernel in at least one set of convolution kernels, convolution computation is performed on different input images to obtain a plurality of intermediate matrices, and the characteristic images are obtained by summing the plurality of intermediate matrices (S230). According to the method, through the characteristics that elements contained in different convolution kernels are the same but the arrangement sequences of the elements are different, resources occupied by the convolution kernels are reduced, the number of times of reading the convolution kernels is reduced, the calculated amount generated when a feature image is determined by a convolution layer is reduced, and the system operation resources consumed in the calculation process are reduced.
Description
Technical Field
The present disclosure relates to the field of model training technology, and more particularly, to a method and apparatus for determining feature images in a convolutional neural network model.
Background
The convolutional neural network is composed of a convolutional layer, a full-connection layer, an activation function and the like, and the output of a single convolutional layer comprises a plurality of characteristic images. In training a convolutional neural network model, a large number of samples need to be calculated. Wherein the calculated amount generated in the convolution layer accounts for 90% of the total calculated amount in the whole training process.
For any one convolution layer, the number of convolution kernels may be determined according to the number of input images and the number of output feature images, and a corresponding number of convolution kernels may be generated, each convolution kernel may be a small matrix, such as a 3×3 matrix, and each input image may be considered a large matrix. The processing of the convolution layer may be as follows: and carrying out convolution calculation on an input image and a convolution kernel, specifically extracting all matrixes with the same size as the convolution kernel from the input image, multiplying the extracted matrixes with the alignment elements of the convolution kernel, adding the multiplied matrixes to obtain a numerical value, forming an intermediate matrix by all the obtained numerical values, carrying out convolution calculation on each input image and the convolution kernel to obtain an intermediate matrix, and adding the intermediate matrices to obtain a characteristic image.
In carrying out the present disclosure, the inventors found that there are at least the following problems:
because the convolutional neural network comprises a plurality of convolutional layers, each convolutional layer needs to output more characteristic images, and the number of the convolutional kernels corresponding to each characteristic image is also more. The corresponding calculation amount of each convolution kernel is larger, and the total calculation amount in the whole training process increases exponentially. Therefore, the computation amount generated in the convolution layer is huge, and a large amount of processing resources are required.
Disclosure of Invention
In order to overcome the problems in the related art, the present disclosure provides the following technical solutions:
in a first aspect, there is provided a method of determining a feature image in a convolutional neural network model, the method comprising:
acquiring a plurality of input images of a target processing layer in a convolutional neural network model;
acquiring at least one group of convolution kernels of the target processing layer, wherein the elements contained in different convolution kernels in the same group are the same and the arrangement order of the elements is different;
based on each convolution kernel in the at least one group of convolution kernels, respectively carrying out convolution calculation on different input images to obtain a plurality of intermediate matrixes, and summing the plurality of intermediate matrixes to obtain a characteristic image, wherein each element of the intermediate matrixes is a polynomial obtained by multiplying a corresponding convolution kernel with the input image in the convolution calculation process and then adding the para elements.
The method provided by the embodiment obtains a plurality of input images; generating at least one group of convolution kernels, wherein different convolution kernels in the same group contain the same elements and have different arrangement sequences of the elements; based on each convolution kernel in at least one group of convolution kernels, convolution calculation is carried out on different input images respectively to obtain a plurality of intermediate matrixes, and the characteristic images are obtained by summing the plurality of intermediate matrixes. The method can reduce the resources occupied by the stored convolution kernels, reduce the number of times of reading the convolution kernels, reduce the calculated amount generated when the convolution layer determines the characteristic image and reduce the system operation resources consumed in the calculation process by the characteristics that the elements contained in different convolution kernels of the convolution kernels are identical and the arrangement sequence of the elements is different.
In one possible implementation manner, the summing the plurality of intermediate matrices to obtain a feature image includes:
adding polynomials of elements at the same position in the plurality of intermediate matrixes to obtain polynomials corresponding to each element of the characteristic image;
respectively combining the polynomials corresponding to each element of the characteristic image;
and respectively evaluating the polynomials processed by each merging similar item to obtain the characteristic image.
The number of times of multiplication addition which is required to be performed together is far more than that of the polynomials which are required to be performed with the same kind of term combination. It can be seen that as the number of convolution kernels included in a set of convolution kernels increases, and the overall calculation process for determining the feature image is completed, the number of places involved in which the amount of computation can be reduced increases greatly, ultimately accelerating the speed of determining the feature image.
In one possible implementation, before acquiring the at least one set of convolution kernels of the target processing layer, the method further comprises:
randomly generating N convolution kernels, wherein N is the preset group number;
and performing element displacement on each convolution kernel in the N convolution kernels in a row unit and/or performing element displacement in a column unit to obtain M-1 different convolution kernels, wherein M-1 convolution kernels and the convolution kernels before element displacement form a group of convolution kernels of the target processing layer, and M is the number of the convolution kernels in a preset group.
The method can reduce the resources occupied by the stored convolution kernels, reduce the number of times of reading the convolution kernels, reduce the calculated amount generated when the convolution layer determines the characteristic image and reduce the system operation resources consumed in the calculation process by the characteristics that the elements contained in different convolution kernels of the convolution kernels are identical and the arrangement sequence of the elements is different.
In one possible implementation, the number of convolution kernels in each group is greater than 2 and less than the product of the number of rows and columns of convolution kernels.
In one possible implementation manner, after summing the plurality of intermediate matrices to obtain the feature image, the method further includes:
when an output result of the convolutional neural network model is obtained, determining an adjustment value of each element in each convolutional kernel in the at least one group of convolutional kernels according to the output result of the convolutional neural network model and a preset output result;
determining the sum of the adjustment values of the same element contained in different convolution kernels in the same group as a corrected adjustment value corresponding to the adjustment value of the same element;
the respective convolution kernels are adjusted based on the corrected adjustment value for each element.
In the implementation, a plurality of convolution layers exist in the convolution neural network model, the first convolution layer outputs a characteristic image to the Z-1 th convolution layer, and the last convolution layer, namely the Z-1 th convolution layer, outputs a final output result. When the output result of the convolutional neural network model is obtained, the output result is generally in error with the preset output result because the convolutional neural network model is still in the training process. Based on the errors produced by the overall convolutional neural network model, an adjustment value for each element in each of the plurality of sets of convolutional kernels may be determined. Then, the sum of the adjustment values of the same element included in different convolution kernels in the same group is determined as the corrected adjustment value corresponding to the adjustment value of the same element.
In a second aspect, there is provided an apparatus for determining a feature image in a convolutional neural network model, the apparatus comprising at least one module for implementing the method for determining a feature image in a convolutional neural network model provided in the first aspect above.
In a third aspect, a terminal is provided, the terminal comprising a processor, a memory, the processor configured to execute instructions stored in the memory; the processor implements the method for determining a feature image in a convolutional neural network model provided in the first aspect described above by executing instructions.
In a fourth aspect, a computer readable storage medium is provided, comprising instructions which, when run on an origin server, cause the origin server to perform the method of determining a feature image in a convolutional neural network model provided in the first aspect above.
In a fifth aspect, a computer program product comprising instructions, which when run on an origin server, causes the origin server to perform the method of determining a feature image in a convolutional neural network model provided in the first aspect above.
The technical scheme provided by the embodiment of the disclosure can comprise the following beneficial effects:
the method provided by the embodiment obtains a plurality of input images; generating a plurality of groups of convolution kernels, wherein different convolution kernels in the same group contain the same elements and have different arrangement sequences of the elements; at least one feature image corresponding to the plurality of input images is determined. The method can reduce the resources occupied by the stored convolution kernels, reduce the number of times of reading the convolution kernels, reduce the calculated amount generated when the convolution layer determines the characteristic image and reduce the system operation resources consumed in the calculation process by the characteristics that the elements contained in different convolution kernels of the convolution kernels are identical and the arrangement sequence of the elements is different.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic diagram illustrating a structure of a terminal according to an exemplary embodiment;
FIG. 2 is a flow diagram illustrating a method of determining a feature image in a convolutional neural network model, in accordance with an exemplary embodiment;
FIG. 3 is a flow diagram illustrating a method of determining a feature image in a convolutional neural network model, in accordance with an exemplary embodiment;
FIG. 4 is a flow diagram illustrating a method of determining a feature image in a convolutional neural network model, in accordance with an exemplary embodiment;
FIG. 5 is a flow diagram illustrating a method of determining a feature image in a convolutional neural network model, in accordance with an exemplary embodiment;
FIG. 6 is a flow diagram illustrating a method of determining a feature image in a convolutional neural network model, in accordance with an exemplary embodiment;
FIG. 7 is a flow diagram illustrating a method of determining a feature image in a convolutional neural network model, in accordance with an exemplary embodiment;
FIG. 8 is a flow diagram illustrating a method of determining a feature image in a convolutional neural network model, in accordance with an exemplary embodiment;
FIG. 9 is a flow diagram illustrating a method of determining a feature image in a convolutional neural network model, in accordance with an exemplary embodiment;
fig. 10 is a schematic structural view showing an apparatus for determining a feature image in a convolutional neural network model according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
The embodiment of the application provides a method for determining a characteristic image in a convolutional neural network model, wherein an execution subject of the method is a terminal.
The terminal may include a processor 110, a memory 120, and the processor 110 may be connected to the memory 120 as shown in fig. 1. Processor 110 may include one or more processing units; the processor 110 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but may also be a Digital Signal Processor (DSP), application Specific Integrated Circuit (ASIC), field Programmable Gate Array (FPGA), or other programmable logic device, etc.
In particular, the program may include program code including computer-operating instructions. The terminal may also include a memory 120, the memory 120 being operable to store software programs and modules, and the processor 110 performing tasks by reading the software code and modules stored in the memory 120.
In addition, the terminal may further include a receiver 130 and a transmitter 140, wherein the receiver 130 and the transmitter 140 may be respectively connected to the processor 110, and the transmitter 130 and the receiver 140 may be collectively referred to as a transceiver. The transmitter 140 may be used to transmit messages or data and the transmitter 140 may include, but is not limited to, at least one amplifier, a tuner, one or more oscillators, a coupler, an LNA (Low Noise Amplifier ), a duplexer, and the like.
An exemplary embodiment of the present disclosure provides a method for determining a feature image in a convolutional neural network model, as shown in fig. 2, the process flow of which may include the steps of:
step S210, a plurality of input images of a target processing layer in a convolutional neural network model are acquired.
In the implementation, in the process of training the convolutional neural network model, firstly, the structure of the convolutional neural network model, such as the number of layers of convolutional layers contained in the convolutional neural network model, the number of input images, the number of convolutional kernels, the number of output characteristic images and the like in each layer, is designed. In the first training process of the convolutional neural network model, the values of elements used for performing convolutional calculation on the input image to obtain a convolutional kernel of the output image are random. The convolution kernel may be a matrix, and the elements of the convolution kernel are values at any position (positions determined by rows and columns) in the matrix. For a convolution kernel of size 3X3, there are 9 values in the convolution kernel for 3 rows and 3 columns, and these 9 values are the elements of the convolution kernel. Similarly, for a convolution kernel of size 5X5, there are 25 values for 5 rows and 5 columns in the convolution kernel, and these 25 values are the elements of the convolution kernel. Other sizes of convolution kernels are similar and are not illustrated herein. And continuously returning to optimize the values of the elements of the convolution kernel through the difference value between the result output by the convolution neural network model and the correct result in the sample in the process from the second training to the Nth training, so that the difference value between the result output by the convolution neural network model and the correct result in the sample is the lowest value as far as possible after the Nth training. For a target processing layer (a certain layer of convolution layer in the convolution neural network model), a plurality of characteristic images output by a convolution layer on the target processing layer are processed by other layers such as a Pooling layer and a RELU layer (activating function layer), namely a plurality of input images of the convolution layer on the target processing layer are obtained.
Each feature image in each convolutional layer in the convolutional neural network model may be determined using the methods provided by the present embodiments.
Step S220, at least one set of convolution kernels of the target processing layer is acquired.
The multi-dimensional tensor (including the tensor of three or more dimensions of three or three orders) formed by all the convolution kernels in each group of convolution kernels can be tensors with special structures arranged according to a certain rule, and the purpose of the tensor is to enable the elements in each group of convolution kernels to be repeatable, so that when the elements of the convolution kernels are used for calculation, the calculation amount can be reduced by combining the similar terms. The size of the convolution kernel is typically 3X3 or 5X5, and the height and width of the convolution kernel are typically the same value. The elements contained in different convolution kernels in the same group are identical and the order of the elements is different.
In implementations, multiple sets of convolution kernels may be generated in units of groups during initialization of the convolutional neural network model. For example, as shown in fig. 3, in a certain convolution layer, there are 6 input images in total, and there are 6 convolution kernels corresponding to each input image. The 6 convolution kernels may be grouped, for example, 1-3 into 1 groups and 4-6 into 1 groups.
Optionally, before acquiring at least one set of convolution kernels of the target processing layer, the method provided by the embodiment may further include: randomly generating N convolution kernels, wherein N is the preset group number; and performing element displacement on each of the N convolution kernels in a row unit and/or performing element displacement in a column unit to obtain M-1 different convolution kernels, wherein the M-1 convolution kernels and the convolution kernels before the element displacement form a group of convolution kernels of the target processing layer, and M is the number of the convolution kernels in a preset group.
In practice, in initializing the convolutional neural network model, for example, convolutional kernels 1-3 are grouped into 1 groups and convolutional kernels 4-6 are grouped into 1 groups. First, 2 convolution kernels, convolution kernel 1 and convolution kernel 4, are randomly generated. Next, the convolution kernel 1 is subjected to element displacement in units of rows, and if the size of the convolution kernel 1 is 3X3, as shown in fig. 4, the convolution kernel 1 is subjected to element displacement in units of columns to obtain a convolution kernel 2 and a convolution kernel 3. Where W1-W8 are elements in the convolution kernel. The convolution kernel 4-6 is generated in the same way.
Optionally, the number M of convolution kernels in each group is greater than 2 and less than the product of the number of rows and columns of convolution kernels. Such as up to 9 for a convolution kernel of size 3X 3. Because once M exceeds 9, the convolution kernels of size 3X3 have been shifted by elements in units of rows and by columns in all possible ways, the 10 th convolution kernel must be one of the first 9 convolution kernels repeated. That is, to ensure that the elements contained in different convolution kernels in the same group are the same and the arrangement order of the elements is different, the Mmax is controlled to be not more than the product of the number of rows and the number of columns of the convolution kernels.
In practice, for example, there are 5 convolution kernels 1-5 in the 1 set of convolution kernels. As shown in fig. 5, first, the convolution kernels 2 and 3 may be obtained by performing element displacement in units of columns from the convolution kernel 1, and then the convolution kernels 4 and 5 may be obtained by performing element displacement in units of rows from the convolution kernel 1. Or the convolution kernel 2 may perform element displacement in units of rows to obtain the convolution kernels 4 and 5.
Step S230, based on each convolution kernel in at least one group of convolution kernels, convolution calculation is performed on different input images to obtain a plurality of intermediate matrixes, and the characteristic images are obtained by summing the plurality of intermediate matrixes.
Each element of the intermediate matrix is a polynomial obtained by multiplying a corresponding convolution kernel with an input image in a convolution calculation process and then adding the para elements.
In practice, as shown in fig. 6, there are 4 input images in the same convolution layer, and the convolution layer is required to output 2 feature images, namely feature image 1 and feature image 2. The 4 input images are respectively subjected to convolution calculation through 4 convolution kernels 1-4 to obtain an intermediate matrix 1-4, and a characteristic image 1 can be obtained based on the intermediate matrix 1-4. And carrying out convolution calculation on the 4 input images again through the 4 convolution kernels 5-8 to obtain an intermediate matrix 5-8, and obtaining the characteristic image 2 based on the intermediate matrix 5-8.
For fig. 6, convolution kernels 1-4 may be divided into multiple sets of convolution kernels, and convolution kernels 5-8 may be divided into multiple sets of convolution kernels. Here, only an example is shown, in practice, the number of convolution kernels corresponding to 1 feature image is large, and the convolution kernels corresponding to 1 feature image may be divided into multiple sets of convolution kernels. For each set of convolution kernels, the different convolution kernels contain the same elements and the order of the elements is different.
As shown in fig. 7, in the determination of the intermediate matrix 1, each time the input image 1 is extracted, for example, 3X3 adjacent elements, the 3X3 adjacent elements in the input image 1 are multiplied by the elements of the convolution kernel at the corresponding position and added to obtain a polynomial. Shifting the convolution kernel in the input image 1 according to a preset number of rows or a preset number of columns each time, repeating the operation of multiplying and adding 3X3 adjacent elements in the obtained input image 1 with elements of the convolution kernel at the corresponding position to obtain a polynomial until the convolution kernel traverses all 3X3 adjacent elements in the input image, and obtaining the intermediate matrix 1.
Optionally, the step of summing the plurality of intermediate matrices to obtain the feature image may include: adding polynomials of elements at the same position in a plurality of intermediate matrixes to obtain polynomials corresponding to each element of the characteristic image; respectively combining the polynomials corresponding to each element of the characteristic image; and respectively evaluating the polynomials processed by each merging similar item to obtain a characteristic image.
In implementation, for the method provided in this embodiment, the polynomials of a plurality of intermediate matrices may be determined through a plurality of channels at the same time, and then the polynomials of the same position are added.
As shown in fig. 8, convolution kernels 1, 2 and 3 are used to perform convolution computation on 3X3 adjacent elements at the upper left corner positions in the input image 1, 2 and 3, respectively, to obtain one element of the first row and first column of the intermediate matrix 1, 2 and 3, respectively, as an example. The polynomial corresponding to one element of the first row and the first column of the intermediate matrix 1 is: w (W) 0 ×a 0 +W 1 ×a 1 +W 2 ×a 2 +W 3 ×a 3 +W 4 ×a 4 +W 5 ×a 5 +W 6 ×a 6 +W 7 ×a 7 +W 8 ×a 8 . The polynomial corresponding to one element of the first row and the first column of the intermediate matrix 2 is: w (W) 2 ×b 0 +W 0 ×b 1 +W 1 ×b 2 +W 5 ×b 3 +W 3 ×b 4 +W 4 ×b 5 +W 8 ×b 6 +W 6 ×b 7 +W 7 ×b 8 . The polynomial corresponding to one element of the first row and the first column of the intermediate matrix 3 is: w (W) 1 ×c 0 +W 2 ×c 1 +W 0 ×c 2 +W 4 ×c 3 +W 5 ×c 4 +W 3 ×c 5 +W 7 ×c 6 +W 8 ×c 7 +W 6 ×c 8 。
In determining the feature image, polynomials of elements of the same position of all intermediate matrices corresponding to the feature image are added, which includes adding polynomials of elements of the same position of intermediate matrix 1, intermediate matrix 2, and intermediate matrix 3. Of course, adding polynomials corresponding to the elements of the first row and the first column of the intermediate matrix 1, the intermediate matrix 2 and the intermediate matrix 3 is also included, to obtain: w (W) 0 ×a 0 +W 1 ×a 1 +W 2 ×a 2 +W 3 ×a 3 +W 4 ×a 4 +W 5 ×a 5 +W 6 ×a 6 +W 7 ×a 7 +W 8 ×a 8 +W 2 ×b 0 +W 0 ×b 1 +W 1 ×b 2 +W 5 ×b 3 +W 3 ×b 4 +W 4 ×b 5 +W 8 ×b 6 +W 6 ×b 7 +W 7 ×b 8 +W 1 ×c 0 +W 2 ×c 1 +W 0 ×c 2 +W 4 ×c 3 +W 5 ×c 4 +W 3 ×c 5 +W 7 ×c 6 +W 8 ×c 7 +W 6 ×c 8 。
It can be seen that the above equation can be processed to combine the same types of terms to obtain: w (W) 0 ×(a 0 +b 1 +c 2 )+W 1 ×(a 1 +b 2 +c 0 )+W 2 ×(a 2 +b 0 +c 1 )+W 3 ×(a 3 +b 4 +c 5 )+W 4 ×(a 4 +b 5 +c 3 )+W 5 ×(a 5 +b 3 +c 4 )+W 6 ×(a 6 +b 7 +c 8 )+W 7 ×(a 7 +b 8 +c 6 )+W 8 ×(a 8 +b 6 +c 7 ). It can be seen that the polynomial without homography combining takes 27 multiplications and 26 additions in total to obtain the desired result, while the polynomial with homography combining takes 9 multiplications and 26 additions in total to obtain the desired result.
It can be seen that the number of operations for 18 multiplications can be reduced for a small portion of the operations in the determined feature image. With the increase of the number of convolution kernels included in a group of convolution kernels and the completion of the whole calculation process for determining the characteristic image, the place where the operand can be reduced is greatly increased, and finally the speed for determining the characteristic image is accelerated.
Optionally, after summing the plurality of intermediate matrices to obtain the feature image, the method provided in this embodiment further includes: when an output result of the convolutional neural network model is obtained, determining an adjustment value of each element in each convolutional kernel in at least one group of convolutional kernels according to the output result of the convolutional neural network model and a preset output result; determining the sum of the adjustment values of the same element contained in different convolution kernels in the same group as a corrected adjustment value corresponding to the adjustment value of the same element; the respective convolution kernels are adjusted based on the corrected adjustment value for each element.
In the implementation, a plurality of convolution layers exist in the convolution neural network model, the first convolution layer outputs a characteristic image to the Z-1 th convolution layer, and the last convolution layer, namely the Z-1 th convolution layer, outputs a final output result. When the output result of the convolutional neural network model is obtained, the output result is generally in error with the preset output result because the convolutional neural network model is still in the training process. Based on the errors produced by the overall convolutional neural network model, an adjustment value for each element in each of the plurality of sets of convolutional kernels may be determined. Then, the sum of the adjustment values of the same element included in different convolution kernels in the same group is determined as the corrected adjustment value corresponding to the adjustment value of the same element. For example, with respect to fig. 8, if fig. 9 is present in case that convolution computation is performed on 3X3 adjacent elements in the input image 1, 3X3 adjacent elements in the input image 2, and 3X3 adjacent elements in the input image 3 through 3 channels by the convolution kernel 1, the convolution kernel 2, and the convolution kernel 3, respectively, then the following formula is present in calculating the corrected adjustment value corresponding to the adjustment value of the same element:
wherein Δw is the corrected adjustment value corresponding to the adjustment value of the same element. WH is the product of the width and height of the feature image. Delta Rk For sensitivity, delta Rk R in (2) represents the R-th feature image of the target processing layer, w_size 2 Is the product of the width of the convolution kernel and the height of the convolution kernel.
Finally, by the method provided in this example, a test was performed. Specifically, a Cifar10 data set is used for image recognition training, a convolutional neural network model is designed to be a 3-layer model, and the convolution kernel size of each layer is 5X5. The test results obtained are shown in the following table:
TABLE 1
By the method provided in this example, a test was performed. The convolutional neural network model is specifically trained for the field of image super-resolution, and a new image which is used for amplifying an original image into a size of 3 times is set. The convolutional neural network model is designed to be a 3-layer model, and the size of a convolutional kernel is 5X5. The test results obtained are shown in the following table:
TABLE 2
The PSNR is a common measurement index in the application of the super-resolution of the image, and the effect after the super-resolution of the image is better as the PSNR is larger. BaseHisrcnn is a convolutional neural network structure applied to image super-resolution.
The method provided by the embodiment obtains a plurality of input images; generating a plurality of groups of convolution kernels, wherein different convolution kernels in the same group contain the same elements and have different arrangement sequences of the elements; at least one feature image corresponding to the plurality of input images is determined. The method can reduce the resources occupied by the stored convolution kernels, reduce the number of times of reading the convolution kernels, reduce the calculated amount generated when the convolution layer determines the characteristic image and reduce the system operation resources consumed in the calculation process by the characteristics that the elements contained in different convolution kernels of the convolution kernels are identical and the arrangement sequence of the elements is different.
Yet another exemplary embodiment of the present disclosure provides an apparatus for determining a feature image in a convolutional neural network model, as shown in fig. 10, the apparatus comprising:
an acquisition module 1010, configured to acquire a plurality of input images of a target processing layer in a convolutional neural network model; at least one set of convolution kernels of the target processing layer is acquired. Wherein, the elements contained in different convolution kernels in the same group are the same and the arrangement order of the elements is different. The acquisition functions in step S210 and step S220 described above, as well as other implicit steps, may be implemented in particular.
The determining module 1020 is configured to perform convolution calculation on different input images based on each convolution kernel in the at least one set of convolution kernels to obtain a plurality of intermediate matrices, and sum the plurality of intermediate matrices to obtain a feature image, where each element of the intermediate matrix is a polynomial obtained by multiplying a corresponding convolution kernel with an input image in a convolution calculation process and adding the corresponding element. The acquisition function in step S230 described above, as well as other implicit steps, may be implemented in particular.
Optionally, the determining module 1020 is configured to add polynomials of elements at the same position in the plurality of intermediate matrices to obtain a polynomial corresponding to each element of the feature image; respectively combining the polynomials corresponding to each element of the characteristic image; and respectively evaluating the polynomials processed by each merging similar item to obtain the characteristic image.
Optionally, the apparatus further comprises:
the generating module is used for randomly generating N convolution kernels, wherein N is the preset group number;
and the displacement module is used for carrying out element displacement on each convolution kernel in the N convolution kernels in a row unit and/or carrying out element displacement on the N convolution kernels in a column unit to obtain M-1 different convolution kernels, wherein the M-1 convolution kernels and the convolution kernels before the element displacement form a group of convolution kernels of the target processing layer, and M is the number of the convolution kernels in a preset group.
Optionally, the number of convolution kernels in each group is greater than 2 and less than the product of the number of rows and columns of convolution kernels.
Optionally, the determining module 1020 is further configured to determine, when an output result of the convolutional neural network model is obtained, an adjustment value of each element in each convolution kernel in the at least one set of convolution kernels according to the output result of the convolutional neural network model and a preset output result; determining the sum of the adjustment values of the same element contained in different convolution kernels in the same group as a corrected adjustment value corresponding to the adjustment value of the same element;
the apparatus further comprises an adjustment module:
the adjustment module is used for adjusting each convolution kernel based on the corrected adjustment value of each element.
It should be noted that, the acquiring module 1010 and the determining module 1020 may be implemented by a processor, or the processor may be implemented in combination with a memory, or the processor may be implemented by executing program instructions in the memory.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
The method can reduce the resources occupied by the stored convolution kernels, reduce the number of times of reading the convolution kernels, reduce the calculated amount generated when the convolution layer determines the characteristic image and reduce the system operation resources consumed in the calculation process by the characteristics that the elements contained in different convolution kernels of the convolution kernels are identical and the arrangement sequence of the elements is different.
It should be noted that: the device for determining the feature image in the convolutional neural network model provided in the above embodiment only uses the division of the above functional modules to illustrate when determining the feature image, in practical application, the above functional allocation may be performed by different functional modules according to needs, i.e. the internal structure of the terminal is divided into different functional modules to complete all or part of the functions described above. In addition, the device for determining the feature image in the convolutional neural network model provided in the above embodiment belongs to the same concept as the method embodiment for determining the feature image in the convolutional neural network model, and the specific implementation process is detailed in the method embodiment, which is not repeated here.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (16)
1. A method of determining a feature image in a convolutional neural network model, the method comprising:
acquiring a plurality of input images of a target processing layer in a convolutional neural network model;
acquiring at least one group of convolution kernels of the target processing layer, wherein the elements contained in different convolution kernels in the same group are the same and the arrangement order of the elements is different;
based on each convolution kernel in the at least one group of convolution kernels, respectively carrying out convolution calculation on different input images to obtain a plurality of intermediate matrixes, and summing the plurality of intermediate matrixes to obtain a characteristic image, wherein each element of the intermediate matrixes is a polynomial obtained by multiplying a corresponding convolution kernel with the input image in the convolution calculation process and then adding the para elements.
2. The method of claim 1, wherein summing the plurality of intermediate matrices results in a feature image, comprising:
adding polynomials of elements at the same position in the plurality of intermediate matrixes to obtain polynomials corresponding to each element of the characteristic image;
respectively combining the polynomials corresponding to each element of the characteristic image;
and respectively evaluating the polynomials processed by each merging similar item to obtain the characteristic image.
3. The method of claim 1, wherein prior to obtaining at least one set of convolution kernels for the target processing layer, the method further comprises:
randomly generating N convolution kernels, wherein N is the preset group number;
and performing element displacement on each convolution kernel in the N convolution kernels in a row unit and/or performing element displacement in a column unit to obtain M-1 different convolution kernels, wherein M-1 convolution kernels and the convolution kernels before element displacement form a group of convolution kernels of the target processing layer, and M is the number of the convolution kernels in a preset group.
4. The method of claim 1, wherein the number of convolution kernels in each group is greater than 2 and less than the product of the number of rows and columns of convolution kernels.
5. The method of claim 1, wherein after summing the plurality of intermediate matrices to obtain a feature image, the method further comprises:
when an output result of the convolutional neural network model is obtained, determining an adjustment value of each element in each convolutional kernel in the at least one group of convolutional kernels according to the output result of the convolutional neural network model and a preset output result;
determining the sum of the adjustment values of the same element contained in different convolution kernels in the same group as a corrected adjustment value corresponding to the adjustment value of the same element;
the respective convolution kernels are adjusted based on the corrected adjustment value for each element.
6. An apparatus for determining a feature image in a convolutional neural network model, the apparatus comprising:
the acquisition module is used for acquiring a plurality of input images of a target processing layer in the convolutional neural network model; acquiring at least one group of convolution kernels of the target processing layer, wherein the elements contained in different convolution kernels in the same group are the same and the arrangement order of the elements is different;
the determining module is used for respectively carrying out convolution calculation on different input images based on each convolution kernel in the at least one group of convolution kernels to obtain a plurality of intermediate matrixes, and summing the plurality of intermediate matrixes to obtain a characteristic image, wherein each element of the intermediate matrixes is a polynomial obtained by multiplying a corresponding convolution kernel with the input image in the convolution calculation process and then adding the corresponding element.
7. The apparatus according to claim 6, wherein the determining module is configured to add polynomials of elements at the same position in the plurality of intermediate matrices to obtain a polynomial corresponding to each element of the feature image; respectively combining the polynomials corresponding to each element of the characteristic image; and respectively evaluating the polynomials processed by each merging similar item to obtain the characteristic image.
8. The apparatus of claim 6, wherein the apparatus further comprises:
the generating module is used for randomly generating N convolution kernels, wherein N is the preset group number;
and the displacement module is used for carrying out element displacement on each convolution kernel in the N convolution kernels in a row unit and/or carrying out element displacement on the N convolution kernels in a column unit to obtain M-1 different convolution kernels, wherein the M-1 convolution kernels and the convolution kernels before the element displacement form a group of convolution kernels of the target processing layer, and M is the number of the convolution kernels in a preset group.
9. The apparatus of claim 6, wherein the number of convolution kernels in each group is greater than 2 and less than a product of the number of rows and columns of convolution kernels.
10. The apparatus of claim 6, wherein the determining module is further configured to determine, when the output result of the convolutional neural network model is obtained, an adjustment value for each element in each of the at least one set of convolutional kernels according to the output result of the convolutional neural network model and a preset output result; determining the sum of the adjustment values of the same element contained in different convolution kernels in the same group as a corrected adjustment value corresponding to the adjustment value of the same element;
the apparatus further comprises an adjustment module:
the adjustment module is used for adjusting each convolution kernel based on the corrected adjustment value of each element.
11. A terminal comprising a processor and a memory, wherein:
the processor is used for acquiring a plurality of input images of a target processing layer in the convolutional neural network model stored in the memory; acquiring at least one group of convolution kernels of the target processing layer stored in the memory, wherein elements contained in different convolution kernels in the same group are the same and the arrangement order of the elements is different; based on each convolution kernel in the at least one group of convolution kernels, respectively carrying out convolution calculation on different input images to obtain a plurality of intermediate matrixes, and summing the plurality of intermediate matrixes to obtain a characteristic image, wherein each element of the intermediate matrixes is a polynomial obtained by multiplying a corresponding convolution kernel with the input image in the convolution calculation process and then adding the para elements.
12. The terminal of claim 11, wherein the processor is configured to add polynomials of elements at the same position in the plurality of intermediate matrices to obtain a polynomial corresponding to each element of the feature image; respectively combining the polynomials corresponding to each element of the characteristic image; and respectively evaluating the polynomials processed by each merging similar item to obtain the characteristic image.
13. The terminal of claim 11, wherein the processor is further configured to randomly generate N convolution kernels, wherein N is a preset number of groups; and performing element displacement on each convolution kernel in the N convolution kernels in a row unit and/or performing element displacement in a column unit to obtain M-1 different convolution kernels, wherein M-1 convolution kernels and the convolution kernels before element displacement form a group of convolution kernels of the target processing layer, and M is the number of the convolution kernels in a preset group.
14. The terminal of claim 11, wherein the number of convolution kernels in each group is greater than 2 and less than the product of the number of rows and columns of convolution kernels.
15. The terminal of claim 11, wherein the processor is further configured to determine, when an output result of the convolutional neural network model is obtained, an adjustment value for each element in each of the at least one set of convolutional kernels according to the output result of the convolutional neural network model and a preset output result; determining the sum of the adjustment values of the same element contained in different convolution kernels in the same group as a corrected adjustment value corresponding to the adjustment value of the same element; the respective convolution kernels are adjusted based on the corrected adjustment value for each element.
16. A computer readable storage medium comprising instructions which, when run on a terminal, cause the terminal to perform the method of any of claims 1-5.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2017/117503 WO2019119301A1 (en) | 2017-12-20 | 2017-12-20 | Method and device for determining feature image in convolutional neural network model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111247527A CN111247527A (en) | 2020-06-05 |
| CN111247527B true CN111247527B (en) | 2023-08-22 |
Family
ID=66992897
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201780096076.0A Active CN111247527B (en) | 2017-12-20 | 2017-12-20 | Method and device for determining characteristic images in convolutional neural network model |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111247527B (en) |
| WO (1) | WO2019119301A1 (en) |
Families Citing this family (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112541565B (en) * | 2019-09-20 | 2023-08-29 | 腾讯科技（深圳）有限公司 | Convolution calculation data stream mapping method and device |
| CN110807170B (en) * | 2019-10-21 | 2023-06-27 | 中国人民解放军国防科技大学 | Method for realizing Same convolution vectorization of multi-sample multi-channel convolution neural network |
| CN112733585B (en) * | 2019-10-29 | 2023-09-05 | 杭州海康威视数字技术股份有限公司 | image recognition method |
| CN112766474B (en) * | 2019-11-04 | 2024-03-22 | 北京地平线机器人技术研发有限公司 | Method, device, medium and electronic equipment for realizing convolution operation |
| CN110929623A (en) * | 2019-11-15 | 2020-03-27 | 北京达佳互联信息技术有限公司 | Multimedia file identification method, device, server and storage medium |
| KR20210082970A (en) | 2019-12-26 | 2021-07-06 | 삼성전자주식회사 | A method and an apparatus for performing convolution operations |
| CN113052756A (en) * | 2019-12-27 | 2021-06-29 | 武汉Tcl集团工业研究院有限公司 | Image processing method, intelligent terminal and storage medium |
| CN111241993B (en) * | 2020-01-08 | 2023-10-20 | 咪咕文化科技有限公司 | Seat number determining method and device, electronic equipment and storage medium |
| CN111414995B (en) * | 2020-03-16 | 2023-05-19 | 北京君立康生物科技有限公司 | Detection processing method and device for micro-target colony, electronic equipment and medium |
| CN111767928B (en) * | 2020-06-28 | 2023-08-08 | 中国矿业大学 | Method and device for extracting image characteristic information based on convolutional neural network |
| CN113919405B (en) * | 2020-07-07 | 2024-01-19 | 华为技术有限公司 | Data processing method and device and related equipment |
| CN114090470B (en) * | 2020-07-29 | 2023-02-17 | 深圳市中科元物芯科技有限公司 | Data preloading device and preloading method thereof, storage medium and computer equipment |
| CN112016740A (en) * | 2020-08-18 | 2020-12-01 | 北京海益同展信息科技有限公司 | Data processing method and device |
| CN112149694B (en) * | 2020-08-28 | 2024-04-05 | 特斯联科技集团有限公司 | Image processing method, system, storage medium and terminal based on convolutional neural network pooling module |
| CN112132279B (en) * | 2020-09-23 | 2023-09-15 | 平安科技（深圳）有限公司 | Convolutional neural network model compression method, device, equipment and storage medium |
| CN115861043B (en) * | 2023-02-16 | 2023-05-16 | 深圳市旗云智能科技有限公司 | Image data processing method and system based on artificial intelligence |
| CN116295188B (en) * | 2023-05-15 | 2023-08-11 | 山东慧点智能技术有限公司 | Measuring device and measuring method based on displacement sensor |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104077233A (en) * | 2014-06-18 | 2014-10-01 | 百度在线网络技术（北京）有限公司 | Single-channel convolution layer and multi-channel convolution layer handling method and device |
| CN104915322A (en) * | 2015-06-09 | 2015-09-16 | 中国人民解放军国防科学技术大学 | Method for accelerating convolution neutral network hardware and AXI bus IP core thereof |
| CN106326985A (en) * | 2016-08-18 | 2017-01-11 | 北京旷视科技有限公司 | Neural network training method, neural network training device, data processing method and data processing device |
| CN106682736A (en) * | 2017-01-18 | 2017-05-17 | 北京小米移动软件有限公司 | Image identification method and apparatus |
| CN107491787A (en) * | 2017-08-21 | 2017-12-19 | 珠海习悦信息技术有限公司 | Local binarization CNN processing method, device, storage medium and processor |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160110499A1 (en) * | 2014-10-21 | 2016-04-21 | Life Technologies Corporation | Methods, systems, and computer-readable media for blind deconvolution dephasing of nucleic acid sequencing data |
| CN106156781B (en) * | 2016-07-12 | 2019-09-10 | 北京航空航天大学 | Sort convolutional neural networks construction method and its image processing method and device |
| CN106447030B (en) * | 2016-08-30 | 2021-09-21 | 深圳市诺比邻科技有限公司 | Method and system for optimizing computing resources of convolutional neural network |
| CN106778584B (en) * | 2016-12-08 | 2019-07-16 | 南京邮电大学 | A kind of face age estimation method based on further feature Yu shallow-layer Fusion Features |
| CN106971174B (en) * | 2017-04-24 | 2020-05-22 | 华南理工大学 | CNN model, CNN training method and CNN-based vein identification method |
-
2017
- 2017-12-20 CN CN201780096076.0A patent/CN111247527B/en active Active
- 2017-12-20 WO PCT/CN2017/117503 patent/WO2019119301A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104077233A (en) * | 2014-06-18 | 2014-10-01 | 百度在线网络技术（北京）有限公司 | Single-channel convolution layer and multi-channel convolution layer handling method and device |
| CN104915322A (en) * | 2015-06-09 | 2015-09-16 | 中国人民解放军国防科学技术大学 | Method for accelerating convolution neutral network hardware and AXI bus IP core thereof |
| CN106326985A (en) * | 2016-08-18 | 2017-01-11 | 北京旷视科技有限公司 | Neural network training method, neural network training device, data processing method and data processing device |
| CN106682736A (en) * | 2017-01-18 | 2017-05-17 | 北京小米移动软件有限公司 | Image identification method and apparatus |
| CN107491787A (en) * | 2017-08-21 | 2017-12-19 | 珠海习悦信息技术有限公司 | Local binarization CNN processing method, device, storage medium and processor |
Non-Patent Citations (1)
| Title |
|---|
| 一种新的基于无损失函数的深度卷积神经网络的图像特征提取方法;李子强;《中国优秀博硕士论文全文数据库（硕士）信息科技辑》(第9期);第7-40页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019119301A1 (en) | 2019-06-27 |
| CN111247527A (en) | 2020-06-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111247527B (en) | Method and device for determining characteristic images in convolutional neural network model | |
| US11816532B2 (en) | Performing kernel striding in hardware | |
| EP3373210B1 (en) | Transposing neural network matrices in hardware | |
| EP3555814B1 (en) | Performing average pooling in hardware | |
| CN112214726B (en) | Operation accelerator | |
| KR102415576B1 (en) | Method and system for reducing computational complexity of convolutional neural networks | |
| US10083395B2 (en) | Batch processing in a neural network processor | |
| US20190095776A1 (en) | Efficient data distribution for parallel processing | |
| CN109313663B (en) | Artificial intelligence calculation auxiliary processing device, method, storage medium and terminal | |
| CN111465924A (en) | System and method for converting matrix input to vectorized input for a matrix processor | |
| CN108845828B (en) | Coprocessor, matrix operation acceleration method and system | |
| CN110109646B (en) | Data processing method, data processing device, multiplier-adder and storage medium | |
| US20210271973A1 (en) | Operation method and apparatus for network layer in deep neural network | |
| US20200301995A1 (en) | Information processing apparatus, information processing method, and program | |
| CN109255438A (en) | The method and apparatus for adjusting tensor data | |
| CN113344172A (en) | Mapping convolutions to channel convolution engines | |
| CN113538281B (en) | Image denoising method, image denoising device, computer equipment and storage medium | |
| US20230267740A1 (en) | Video data processing method and system, and relevant assemblies | |
| CN111723906A (en) | Accelerated calculation method and system of recurrent neural network and related device | |
| CN112970036A (en) | Convolution block array for implementing neural network applications, method of using the same, and convolution block circuit | |
| KR20220158768A (en) | Power reduction for accelerating machine learning | |
| CN114549945A (en) | Remote sensing image change detection method and related device | |
| CN117456562B (en) | Attitude estimation method and device | |
| EP4361892A1 (en) | Methods and systems for performing a per channel affine transformation using a neural network accelerator | |
| CN117495571B (en) | Data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |