JP2024077434A - Image processing device, image processing method, program, and storage medium - Google Patents

Abstract

[Problem] To provide an image processing device capable of suppressing a decrease in calculation accuracy due to quantization in deep learning. [Solution] The image processing device includes an acquisition means for acquiring an evaluation value representing the characteristics of an output image obtained by inputting a training image used for learning the neural network into the neural network, a generation means for generating an input image that simply represents the characteristics of the output image based on the evaluation value, and a reduction means for inputting the generated input image into the neural network and reducing the number of bits of the parameters that constitute the layers of the neural network. [Selected Figure] Figure 2

Description

The present invention relates to a technology for reducing the number of bits in the parameters that make up the layers of a neural network.

In recent years, many image processing technologies using deep learning have been proposed. Generally, the calculations of neural networks used for deep learning inference take a long time. Therefore, there is a demand for lightweight neural networks to reduce memory usage in neural network calculations and increase inference speed.

One method that has been proposed for making neural networks lighter is to reduce the number of bits in the parameters that make up the layers of the neural network. Here, this method is called quantization. In particular, in embedded devices (cameras, automobiles, etc.) in which neural networks are incorporated, the computing resources available for the neural network are limited, so quantization is often used as an approach to making neural networks lighter.

Patent document 1 also discloses a method for deriving the degree of impact of a decrease in the computational accuracy of a neural network due to quantization.

JP 2018-142049 A

However, with the above conventional technology, although it is possible to confirm the variation in the decrease in calculation accuracy due to quantization, it is not possible to suppress the variation in the decrease in calculation accuracy. Therefore, in deep learning aimed at image restoration such as noise removal and super-resolution, there is a problem in that the variation in the decrease in calculation accuracy due to quantization becomes large.

The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device that can suppress the decrease in calculation accuracy due to quantization in deep learning.

The image processing device according to the present invention is characterized by comprising: an acquisition means for acquiring an evaluation value representing the characteristics of an output image obtained by inputting a training image used for learning the neural network into the neural network; a generation means for generating an input image that simply represents the characteristics of the output image based on the evaluation value; and a reduction means for inputting the generated input image into the neural network and reducing the number of bits of the parameters that constitute the layers of the neural network.

The present invention makes it possible to suppress the decrease in computational accuracy caused by quantization in deep learning.

FIG. 1 is a diagram showing the configuration of an image processing apparatus according to a first embodiment. 5 is a flowchart showing a quantization process in the first embodiment. 5 is a flowchart showing an initialization process of a neural network according to the first embodiment. FIG. 2 is a conceptual diagram showing the structure of a neural network according to the first embodiment. 5 is a flowchart showing processing of a neural network in the first embodiment. 1A to 1C are diagrams for explaining the creation of images separated into color channels from a RAW image. FIG. 1 is a conceptual diagram for explaining a convolution operation. 1A and 1B are diagrams for explaining the creation of a RAW image from an image separated into color channels. FIG. 1 is a conceptual diagram for explaining an evaluation value relating to a pixel. 5 is a flowchart showing a quantization process according to the first embodiment. 10 is a flowchart showing a quantization process according to the second embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

First Embodiment
FIG. 1 is a diagram showing the configuration of an image processing apparatus 100 for performing neural network learning and quantization according to a first embodiment of the present invention.

In FIG. 1, the image processing device 100 includes a storage drive unit 103, a ROM 104, an image processing unit 105, a RAM 106, a CPU 107, and a GPU 108, which are connected via an internal bus 109. Each component of the image processing device 100 can exchange data with each other via the internal bus 109.

The storage device 101 is used to store huge amounts of image data as learning images and to store network parameters created during learning. It may also be used to store images to be used for quantization, to store network parameters created during quantization, and so on. The image processing device 100 exchanges data with the storage device 101 via the storage connection unit 102 and storage drive unit 103.

The image processing unit 105 performs various image processing on the image read from the storage device 101. The processing of the image processing unit 105 is also executed under the control of the CPU 107. For example, the image processing unit 105 performs processing to generate training images for learning a neural network and input images for quantization. Specifically, if the neural network is intended for noise removal, it performs noise addition processing, and if the neural network is intended for super-resolution, it performs degradation processing such as size reduction. It can also correct the brightness and color of the image and calculate the histogram of the image.

A CPU (Central Processing Unit) 107 is arranged as a unit for controlling each function of the image processing device 100, and is mainly used for image processing and quantization. A ROM (Read Only Memory) 104 and a RAM (Random Access Memory) 106 are connected to drive the CPU 107.

RAM 106 is a volatile element that temporarily stores image data and calculation results and can be read out when necessary. In recent years, DDR4-SDRAM (Dual Data Rate 4 - Synchronous Dynamic RAM) and the like are often used.

ROM 104 is a non-volatile element that stores programs for operating CPU 107, various adjustment parameters, etc. Programs read from ROM 104 are deployed to volatile RAM 106 and executed.

The GPU 108 is used to process neural networks using deep learning. Since learning requires processing a huge amount of calculations in parallel, it is preferable to use a GPU, which has higher parallel processing capabilities than a CPU. It may also be used to perform quantization.

Next, the quantization process performed by the image processing device 100 will be described with reference to Figures 2 to 10.

Figure 2 is a flowchart showing the quantization process in this embodiment. Figure 3 is a flowchart showing the process of initializing the neural network, explaining step S201 in Figure 2 in more detail. Figure 4 is a conceptual diagram showing the structure of the neural network. Figure 5 is a flowchart showing the process of the neural network, explaining step S204 in Figure 2 in more detail.

In FIG. 2, first, in step S201, the CPU 107 initializes data and parameters related to the structure of the neural network, and loads them in the GPU 108. The data and parameters related to the structure will be described later. The data and parameters related to the structure may be stored in the storage device 101, or loaded in the RAM 106. The neural network processing performed in step S201 will be described using the flowchart in FIG. 3.

First, in step S301, the CPU 107 creates data regarding the structure of a neural network. In this embodiment, an example in which a CNN (Convolutional Neural Network) is used will be described with reference to FIG. 4.

In FIG. 4, neural network 601 has a hierarchical structure of input layer 605 containing multiple neurons 603, hidden layer 606, and output layer 607, and is configured by connecting neuron nodes in each layer. When data is input from input node 602 in neural network 601, calculations are performed in multiple neurons 603, and the calculation results are output to output node 604. Examples of calculation formulas calculated by each neuron are shown in formula (1) and formula (2).

Y = f(Z) ... (1)
Z = W × X + b ... (2)
Here, Y in formula (1) represents the output data of the neuron, f(Z) represents the activation function, Z in formula (2) represents the input data of the activation function in formula (1), W represents the weight, X represents the input data of the neuron, and b represents the bias.

Here, data related to the structure of the neural network 601 includes operations at each layer, such as convolution operations, pooling operations, and upsampling operations, performed by the CNN, operations of activation functions at each layer, the number of layers in the layer, the number of neurons 603, node connectivity information, and the like. In this embodiment, the number of neurons 603 is defined as a channel.

Note that this embodiment is merely an example, and the neural network is not limited to CNN. For example, a Generative Adversarial Network (GAN) may be used, a skip connection may be included, or a recurrent type such as a Recurrent Neural Network (RNN) may be used. Also, since FIG. 4 is a conceptual diagram, the number of hidden layers and channels is small, but this is not limiting. For example, the input layer 605, hidden layer 606, and output layer 607 are not limited to these, and the channels of each layer may be changed, or the number of hidden layers 606 may be increased.

Next, in step S302, the CPU 107 initializes the parameters of the neural network deployed in the GPU 108 in step S301. Here, the parameters of the neural network are the weights, biases, and activation function parameters used in equations (1) and (2). For example, in initializing the parameters of the neural network, the He initialization method is used to set the initial values of the parameters with FLOAT32 bit accuracy.

Note that this embodiment is merely an example, and the method for initializing the neural network parameters is not limited to He initialization, and other methods may be used. Furthermore, the bit precision of the neural network parameters is not limited to FLOAT32, and may be a data format such as FLOAT64 or FLOAT16.

In step S303, the CPU 107 loads data regarding the structure of the neural network created in step S301 and the parameters of the neural network set in step S302 onto the GPU 108.

Here, the data and parameters related to the structure of the neural network may be temporarily stored not only in the GPU 108 but also in the storage device 101, the ROM 104, or the RAM 106. The data and parameters related to the structure of the neural network may be stored in the same file using a file in the Protocol Buffers data format, or may be stored separately. The data format of the saved file is not limited to Protocol Buffers, and may be another data format such as HDF (Hierarchical Data Format).

Returning to the explanation of FIG. 2, in step S202, the CPU 107 acquires a learning image from the storage device 101 and expands it in the RAM 106. Here, the learning image refers to an image that is the source of a training image or a correct answer image that is an input image when learning a neural network. In this embodiment, the learning image is a Bayer-array RAW image having R, Gr, Gb, and B color channels, and the possible values are 14 bits from 0 to 16383. Here, the process of converting a Bayer-array RAW image into a color image is called a de-Bayer process. The learning image is not limited to a Bayer-array RAW image, but may be an image in a color space such as RGB, XYZ, L*a*b*, or HSV after de-Bayer process, or an image having a luminance signal and a color difference signal such as YUV, YCbCr, or YPbPr. In addition, one learning image may be acquired and expanded in the RAM 106, or multiple learning images may be acquired together and expanded in the RAM 106.

In step S203, the CPU 107 uses the image processing unit 105 to create a correct answer image and a training image from the learning image expanded in the RAM 106, and expands them in the RAM 106. The correct answer image is an image that is the expected value of the output image of the neural network during learning. The training image is an image that is input to the neural network during learning. For example, in the case of a neural network intended for noise removal, an image with very little noise is considered to be the correct answer image. In this embodiment, the correct answer image is created by performing preprocessing on the learning image. Examples of preprocessing include image correction processing such as gamma correction and white balance correction.

Next, a training image is an image input to a neural network during learning. For example, in the case of a neural network for noise removal, an image containing more noise than a correct answer image is used as a training image. In this embodiment, a training image is created by performing degradation processing and preprocessing on a learning image. As an example of degradation processing, a noise pattern is added to an image. In addition, as a preprocessing performed to create a training image, correction processing such as gamma correction and white balance correction is performed on the assumption that an image after correction processing is input to the neural network. In addition, in order to give the neural network a correction processing function in addition to the image recovery function, preprocessing may be performed only on the correct answer image. Furthermore, different processing may be performed for the preprocessing for creating the correct answer image and the preprocessing for creating the training image. Alternatively, the learning image may be used as the correct answer image without performing preprocessing, or an image after only degradation processing is performed on the learning image may be used as the training image.

Note that, although gamma correction and white balance correction have been given as examples of pre-processing, pre-processing is not limited to these. Furthermore, degradation processing is not limited to adding noise patterns, and image reduction processing or blurring processing may also be performed depending on the purpose of the function realized by the neural network.

In this embodiment, a method of creating a correct image and a training image in step S203 from a learning image acquired in step S202 has been given as an example, but a correct image obtained by preprocessing the learning image and a training image obtained by degrading and preprocessing may be stored in the storage device 101. Also, with the same imaging device, angle of view, and subject, an image under conditions where no degradation or only slight degradation occurs and an image under conditions where significant degradation occurs may be captured, and these may be used as the correct image and the training image, respectively.

In step S204, the CPU 107 inputs the training image expanded in the RAM 106 in step S203 to the neural network expanded in the GPU 108 in step S201 or step S209. Then, the CPU 107 performs further neural network processing and expands the output image in the RAM 106. Here, the training image is defined as an input image for the neural network. The neural network processing performed in step S204 will be described using the flowchart in FIG. 5.

First, in step S401, the CPU 107 acquires the data and parameters related to the neural network structure stored in step S201 or step S209, and deploys the neural network on the GPU 108. Here, if the neural network has already been deployed on the GPU 108 in step S201 or step S209, this process is omitted.

In this embodiment, an example will be described in which an input image 608 is input to an input node 602 of a neural network 601 as shown in FIG. 4, and the result of calculation in the neural network 601 is output as an output image 609 from an output node 604 of the neural network 601. Also, in this embodiment, an example will be described in which the neural network is a U-Net. Here, a U-Net is a type of CNN that performs convolution calculations, pooling calculations, upsampling calculations, activation function calculations, and the like. The calculations performed by a U-Net will be described later.

In step S402, the CPU 107 inputs the input image of the neural network expanded in the RAM 106 in step S203 to the neural network expanded in the GPU 108 in step S401. In this embodiment, an image separated into R, Gr, Gb, and B color channels from the input image of the neural network, which is a RAW image with a Bayer array as shown in FIG. 6, is used as the input image 608 to be input to the neural network 601 in FIG. 4. Here, taking FIG. 4 as an example, the data of the input image 608 is input to the input node 602, and further input to the neurons of the input layer 605.

In step S403, the GPU 108 performs calculations on the data input to the input layer 605 in each neuron of the neural network 601, and outputs output data from the output layer 607.

As an example of the computation of the neural network 601, a convolution computation using a filter of 3 pixels vertical by 3 pixels horizontal will be described with reference to FIG. 7. In FIG. 7, the convolution computation refers to a calculation in which a product-sum computation is first performed on 9 pixels in an area of a filter 802 of 3 pixels vertical by 3 pixels horizontal of the input data 801 using weights 803, and then a bias 804 is added. An example of the convolution computation is shown in equation (3).

Here, Z(x, y) in formula (3) is the convolution operation result 805 at the coordinate position (x, y), X is the pixel value of the input data 801, W is the weight 803, and b is the bias 804.

Using Figure 7 as an example, first, a product-sum operation is performed on 9 pixels (3 pixels vertical x 3 pixels horizontal) centered on the pixel of interest X(x,y) of the input data 801 using a filter 802 with weights 803 (3 pixels vertical x 3 pixels horizontal) centered on W(0,0). Next, a bias 804 is further added to the product-sum operation result, and the result is used as the convolution operation result 805 for the pixel of interest X(x,y) of the input data 801.

Next, the convolution operation result 805 for the pixel of interest X(x, y) is input to the activation function 806, and the neuron output data is output. In addition, the Leaky ReLU function is used for the activation function 806. The formula for the Leaky ReLU function is shown in formula (4).

f(Z(x,y))=max(Z(x,y),Z(x,y)×a) ... (4)
Here, f(Z(x, y)) in formula (4) is the result of the activation function at coordinates (x, y), Z(x, y) is the convolution operation result 805 of formula (3), a is the coefficient of the Leaky ReLU function, and max is a function that outputs the maximum value among the arguments. Then, the convolution operation using formulas (3) and (4) is performed on all pixels of the input data 801, and the result is output data 807.

As described above, each neuron in the neural network 601 performs operations such as convolution, pooling, upsampling, and activation function calculations, and output data is output from the output layer 607.

In this embodiment, the neural network calculations have been described using U-Net as an example, but other calculations may be performed without being limited to this. Also, the filter 802 for the convolution calculation may have a size other than 3 pixels vertically and 3 pixels horizontally. Also, the activation function 806 may be something other than the Leaky ReLU function.

Next, in step S404, the CPU 107 acquires output data of the calculation performed by the neural network 601, and loads the output image of the neural network in the RAM 106. In this embodiment, as shown in FIG. 8, an output image 609, which is an image separated into the R, Gr, Gb, and B color channels, is output from the neural network 601 in FIG. 4, and the output image 609 is returned to a RAW image in a Bayer array.

Here, taking FIG. 4 as an example, the output data of the neurons in the output layer 607 is output to the output node 604, and further, the output image 609 is output. Then, the image obtained by converting the output image 609 back into a RAW image in a Bayer array is expanded in the RAM 106 as the output image of the neural network. Note that in this embodiment, the training image input to the neural network is an image in which the RAW image in the Bayer array is separated into the color channels R, Gr, Gb, and B. However, this embodiment is merely an example, and it is not necessary to separate into color channels, and multiple images may be input together. Furthermore, in this embodiment, the output of the neural network 601 is only the output from the neurons in the output layer 607, but in addition to this, the output of the neurons in the hidden layer 606 may be output as a feature map and expanded in the RAM 106.

Returning to the explanation of FIG. 2, in step S205, the image processing unit 105 performs image processing on the output image expanded in the RAM 106 in step S204, and expands the processed output image in the RAM 106. For example, the image processing performed on the output image includes correction processing such as gamma correction and white balance correction, assuming that an image that has been subjected to correction processing on the output image of a neural network is to be used. Here, gamma correction and white balance correction are given as examples of correction processing, but the correction processing is not limited to these.

In step S206, the CPU 107 calculates the error between the output image expanded in the RAM 106 in step S205 and the correct image expanded in the RAM 106 in step S203. In this embodiment, the pixel values of the output image and the correct image corresponding to the output image are input to a loss function, and the output result of the loss function is regarded as the error. As an example of a loss function, an equation for calculating the mean squared error is shown in equation (5).

In equation (5), M is the mean square error, m is the number of horizontal pixels in the image, n is the number of vertical pixels in the image, Y(x, y) is the pixel value of the pixel of interest in the output image, and J(x, y) is the pixel value of the pixel of interest in the correct image. Note that the loss function is not limited to equation (5). Other functions such as mean absolute error and cross entropy error may also be used as the loss function.

In step S207, the CPU 107 determines whether or not to use the output image expanded in the RAM 106 in step S204 to calculate an evaluation value for a pixel. For example, using the error calculated in step S206 as a criterion for determination, if the error is smaller than a certain value, the process proceeds to step S208, and a selection is made to calculate an evaluation value for the pixel. On the other hand, if the error is larger than the certain value, the process proceeds to step S209, and an evaluation value for the pixel is not calculated. Note that the criterion for determination is not limited to this, and evaluation values such as the number of learnings or SSIM (Structural Similarity Index Measure) may also be used.

In step S208, the CPU 107 uses the image processing unit 105 to calculate evaluation values for the pixels of the output image expanded in the RAM 106 in step S204. The evaluation values for the pixels are statistical distributions calculated from the pixel values of the output image. In this embodiment, an example will be described in which the evaluation values for the pixels are obtained by sequentially calculating an average histogram from the number of occurrences of pixel values in the RAW image. Also, an example will be described in which one output image is obtained in step S204.

First, the number of occurrences is counted for each pixel value from 0 to 16383, which are the possible 14-bit values of the output image. Next, the number of output images used in the calculations up to now and the currently calculated average histogram are obtained, which are expanded in RAM 106. Here, if the output image is the first one, the number of output images and the number of occurrences of all pixel values in the average histogram are set to 0. Next, the average histogram is calculated by sequentially calculating each pixel value from 0 to 16383 using equation (6). An example of the equation for sequential calculation is shown in equation (6).

Xn+1=(n×Xn+xn+1)/(n+1) ... (6)
Here, n in equation (6) represents the total number of images used to calculate the average value up to the time of calculation, Xn represents the nth average value, Xn+1 represents the (n+1)th average value, and xn+1 represents the number of times the pixel value appears in the (n+1)th image used.

Finally, the number of output images used in the calculations up to this point and the currently calculated average histogram are loaded into RAM 106. Here, the evaluation value for a pixel is not limited to the average histogram, and quantitative evaluation values of the image such as brightness, saturation, contrast, and sharpness may be calculated from the image that has been de-Bayered, and a statistical distribution may be calculated from the evaluation values of each image in the output images. Furthermore, the evaluation value for a pixel may be calculated from all pixel values of the RAW image, or may be calculated for each color channel by separating the RAW image into the R, Gr, Gb, and B color channels as shown in FIG. 6. Furthermore, the evaluation value for a pixel may be calculated from the feature map loaded into RAM 106 in step S204.

In step S209, the CPU 107 updates the parameters of the neural network deployed in the GPU 108 based on the error calculated in step S206. For example, the parameters are updated using an optimization method such as stochastic gradient descent so as to reduce the error calculated in step S206. The processing of steps S202 to S209 is then repeated to optimize the parameters of the neural network. Note that the optimization method for the parameters of the neural network is not limited to stochastic gradient descent, and other optimization methods such as Adam may be used. The updated parameters of the neural network may also be temporarily stored in the storage device 101.

In step S210, the CPU 107 determines whether the neural network learning end condition is met. For example, it checks whether the number of learning attempts has reached a preset number, and if so, it determines that the learning end condition is met, and proceeds to step S211 to end the learning. If the learning end condition is not met, the CPU 107 returns to step S202 and continues the learning. Note that the learning end condition is not limited to this. For example, the learning may be ended when the error calculated in step S206 falls below a threshold value.

In step S211, the CPU 107 stores the parameters of the neural network for which learning has been completed in the storage device 101. The destination for storing the parameters is not limited to this, and the ROM 104 or the RAM 106 may also be used.

In step S212, the CPU 107 uses the image processing unit 105 to create an image for quantization based on the pixel evaluation value calculated in step S208, and loads it in the RAM 106. Here, the image for quantization refers to an image that is the source of the input image at the time of quantization. For example, an average histogram of the output image as shown in FIG. 9 is calculated as the pixel evaluation value in step S208, and an example of creating 10 images for quantization based on the result will be described with reference to FIG. 9.

Here, FIG. 9 is a histogram graph with the horizontal axis representing pixel values and the vertical axis representing pixel counts. In this embodiment, an output image having a size of 128 vertical x 128 horizontal, and a total number of pixels of 16,384, will be described as an example.

First, one-dimensional array data having 16,384 pieces of data sorted in ascending order by pixel value is created as an average histogram from the statistical distribution data as shown in FIG. 9. Next, the data at the ends is excluded from the one-dimensional array data of the average histogram. For example, 192 pieces of data from the start position of the data and 192 pieces of data from the end position of the data are excluded to create one-dimensional array data having 16,000 pieces of data. Next, the number of pieces of data used to find one representative value is calculated. Here, a representative value is a value that represents the average histogram, and is array data having the same number of pieces of data as the number of images to be created for quantization. Therefore, in this embodiment, 10 representative values are calculated, so 1600 pieces of data, obtained by dividing 16,000 by 10, are used to find one representative value. Next, the representative value is calculated using formula (7).

Here, d[i] in formula (7) represents the value of the representative value at index i, n represents the number of data points used to find one representative value, and X represents the one-dimensional array data of the average histogram after excluding the data points at the ends. In this embodiment, n is 1600, and the average value is calculated for every 1600 data points, and a representative value d having 10 data points is calculated.

Next, a quantization image is created based on the representative value. For example, ten images for quantization are created in which the pixel values of all pixels are set to the representative value. As a result, a quantization image that simply represents the characteristics of the output image, such as reproducing the characteristics of the average histogram of the output image with a small number of images, can be created. Here, the number of pixels of the output image, the number of data to be excluded as edge data, and the number of images for quantization to be created are not limited to unique values. In addition, the data to be excluded from the average histogram may be determined based on pixel value conditions, such as excluding data with pixel values of 500 or less or 14,000 or more, or the excluded data may be eliminated. In addition, the method of calculating the representative value is not limited to the method of this embodiment, and a range of pixel values may be specified and a representative value may be calculated from within the range, such as calculating one representative value from pixel values of 0 to 500. In addition, the representative value may be calculated not only by the average value but also by other methods such as the median. In addition, if multiple evaluation values related to pixels are calculated in step S208, such as for each channel, the representative value may be calculated for each channel. Alternatively, an output image or another image may be obtained from the RAM 106, the representative value divided by the average pixel value of the obtained image may be used as a correction coefficient, and the obtained image may be multiplied by the correction coefficient to create an image having a value equivalent to the representative value. If the representative value is calculated for each channel, the above process may be performed for each channel to create an image for quantization.

Next, in step S213, the CPU 107 uses the image processing unit 105 to create an input image for quantization based on the image for quantization expanded in the RAM 106 in step S212, and expands it in the RAM 106. In this embodiment, an example will be described in which the input image for quantization is created by performing degradation processing and preprocessing on the image for quantization similar to that performed when creating a training image, which is an input image during neural network training. First, the image for quantization is subjected to a process similar to the degradation processing performed to create the training image in step S203. Next, the image for quantization that has been subjected to degradation processing is subjected to a process similar to the preprocessing performed to create the training image in step S203. Here, if preprocessing is not performed to create a training image during neural network training, it is not necessary to perform preprocessing on the image for quantization.

In step S214, the CPU 107 acquires the neural network parameters stored in the storage device 101 in step S211, and quantizes the parameters using the input image for quantization expanded in the RAM 106 in step S212.

The process of step S213 will be described with reference to the flowchart in FIG. 10, taking as an example a case where post-training quantization is performed using static quantization. In this embodiment, the description will be given on the premise that the bit precision of the weights, biases, and activation functions, which are parameters constituting the layers of the neural network, is converted from FLOAT32 to INT8 in quantization.

First, in step S501, the CPU 107 inputs an input image for quantization to a trained neural network. In step S502, the CPU 107 calculates the maximum and minimum values of the output data for each layer of the neural network. In step S503, the CPU 107 calculates the scale and offset of the output data for each layer of the neural network based on the maximum and minimum values. In step S504, the CPU 107 quantizes the weights and biases based on the scale and offset, and further quantizes the activation function by approximating it to a linear function with a number of digits or less that can be reproduced with the bit accuracy after quantization.

In step S505, the CPU 107 calculates the scale and offset of the output data for each layer of the neural network when the neural network parameters are quantized. In step S506, the CPU 107 evaluates the accuracy of matching between the scale and offset before quantization, and loads the quantized neural network parameters into the RAM 106.

In step S507, the CPU 107 determines whether the quantization end condition is satisfied. If the condition is not satisfied, the process returns to step S501, and the quantization continues using a new input image for quantization. On the other hand, if the quantization end condition is satisfied, the process proceeds to step S508. The quantization end condition may be when the number of quantizations reaches a specified value, or the quantization may be terminated at the user's discretion after checking the results after quantization.

Finally, in step S508, the CPU 107 updates the quantized parameters for each layer based on the results of step S506 to those for which the scale and offset matching accuracy is evaluated to be the highest.

In the present embodiment, the quantization is performed by the CPU 107, but the quantization may be performed by the GPU 108. Although the quantization after learning is performed by static quantization, the quantization method is not limited to this, and learning that takes quantization into consideration (QuantizationAwareTraining) may also be used. Quantization may be performed on one or more of the weights, biases, and activation functions. The bit precision for quantization is not limited to INT8, and may be any data format with a lower bit precision than the bit precision during learning of the neural network, such as FLOAT16, INT16, or INT4.

Returning to the explanation of FIG. 2, in step S215, the CPU 107 stores the neural network parameters quantized in step S213 in the storage device 101. In this embodiment, the explanation is given on the assumption that the parameters are stored in the storage device, but the parameters may be stored in other storage media.

As described above, in this embodiment, an input image for quantization is created based on evaluation values for pixels of an output image of a neural network. This makes it possible to create an input image for quantization that accurately reproduces the characteristics of the output image of a neural network with a small number of images (approximates the output image). Then, by performing quantization using the created input image for quantization, it becomes possible to suppress the variation in the decrease in calculation accuracy due to quantization in deep learning aimed at image restoration, and further suppress the decrease in calculation accuracy.

Second Embodiment
Next, the quantization process performed by the image processing device 100 according to the second embodiment of the present invention will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the quantization process according to the second embodiment. In the flowchart of Fig. 11, steps that perform the same processes as those in the flowchart of Fig. 2 are given the same step numbers as those in Fig. 2, and descriptions thereof will be omitted.

It is also assumed that the neural network has completed learning, and that data and parameters related to the structure are stored in the storage device 101. Here, a neural network that has completed learning refers to a neural network in which steps S202 to S206, S209, and S211 in FIG. 2 have each been performed at least once, and the parameters have been updated.

In step S1101, the CPU 107 acquires data and parameters relating to the structure of the trained neural network from the storage device 101 and deploys the neural network on the GPU 108. Note that if the data and parameters relating to the structure of the trained neural network are stored in another storage device such as the RAM 106 or ROM 104, they may be acquired from the other storage device such as the ROM 104 or RAM 106. Also, the structure of the neural network may be reconstructed to be the same as that at the time of training, without acquiring it from the storage device 101.

In step S1102, the CPU 107 acquires an image for inference from the storage device 101 and expands it in the RAM 106. The image for inference is an image that is the source of the input image during inference of the trained neural network. There may be one or more images for inference. Also, a part of the image for learning may be used as the image for inference. Alternatively, an output image or an image for quantization created by the method shown in the first embodiment may be used.

Next, in step S1103, the CPU 107 uses the image processing unit 105 to perform image processing on the inference image expanded in the RAM 106 in step S1102, and expands the image in the RAM 106 as an input image for the neural network. The image processing is the same as the processing for creating the training image in step S203.

In step S1104, the CPU 107 acquires the input image expanded in the RAM 106 in step S1103, inputs it to the neural network expanded in the GPU 108, and performs the same processing as in step S204. The output image of the neural network is expanded in the RAM 106.

In step S208, the CPU 107 calculates an evaluation value for the pixel based on the output image, as in FIG. 2.

In step S1105, the CPU 107 determines whether all the images for inference have been input to the neural network. If all the images for inference have been input to the neural network, the process proceeds to step S212, where an image for quantization is created from the evaluation values for the pixels. On the other hand, if there are images for inference that have not been input to the neural network, the process returns to step S1102, where the next image for inference is obtained and input to the neural network.

Steps S212 to S215 are the same as those in FIG. 2.

As described above, according to this embodiment, an input image for quantization is created using a method similar to that of the first embodiment, based on the output image obtained when an image for inference is input to a trained neural network. This makes it possible to create an input image for quantization that accurately reproduces the characteristics of the output image of the neural network with a small number of images (approximates the output image). Then, by performing quantization using the created input image for quantization, it is possible to suppress the variation in the decrease in calculation accuracy due to quantization in deep learning aimed at image recovery, and further suppress the decrease in calculation accuracy.

The image processing device 100 described in the above two embodiments may be included in the imaging device, and various types of image processing may be performed on image data captured and acquired by the imaging device.

The disclosure of this specification includes the following image processing device, method, program, and storage medium.

(Item 1)
an acquisition means for acquiring an evaluation value representing a feature of an output image obtained by inputting a training image used for learning the neural network into the neural network;
a generating means for generating an input image simply expressing the characteristics of the output image based on the evaluation value;
a reduction means for inputting the generated input image to the neural network and reducing the number of bits of parameters constituting a layer of the neural network;
An image processing device comprising:

(Item 2)
2. The image processing device according to item 1, wherein the reduction means inputs the input image to the trained neural network to reduce the number of bits.

(Item 3)
3. The image processing device according to item 1 or 2, wherein the evaluation value is an average histogram of the output image.

(Item 4)
4. The image processing device according to item 3, wherein the average histogram is calculated by sequential calculation.

(Item 5)
5. The image processing device according to claim 1, wherein the generating means generates the input image based on a representative value of the evaluation values.

(Item 6)
6. The image processing device according to item 5, wherein the representative values are data of the same number as the input image.

(Item 7)
7. The image processing device according to any one of items 1 to 6, characterized in that the training image and a correct answer image, which is an expected value of the output image of the neural network, are generated from a single original image.

(Item 8)
8. The image processing device according to item 7, wherein the training images are generated by performing noise addition processing or degradation processing on the original images.

(Item 9)
8. The image processing device according to item 7, wherein the correct image is generated by performing preprocessing on the original image.

(Item 10)
The image processing device according to any one of items 1 to 9, further comprising a calculation means for calculating an error between the output image and a correct image which is an expected value of the output image of the neural network, and a selection means for selecting whether or not to obtain the evaluation value from the output image based on the error.

(Item 11)
11. An imaging device comprising the image processing device according to any one of items 1 to 10.

(Item 12)
an acquisition step of acquiring an evaluation value representing a feature of an output image obtained by inputting a training image used for learning the neural network into the neural network;
a generating step of generating an input image simply expressing the characteristics of the output image based on the evaluation value;
a reduction step of inputting the generated input image to the neural network and reducing the number of bits of parameters constituting a layer of the neural network;
13. An image processing method comprising:

(Item 13)
Item 13. A program for causing a computer to execute each step of the image processing method according to item 12.

(Item 14)
13. A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to item 12.

Other Embodiments
The present invention can also be realized by a process in which a program for realizing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100: Image processing device, 101: Storage device, 102: Storage connection unit, 103: Storage drive unit, 104: ROM, 105: Image processing unit, 106: RAM, 107: CPU, 108: GPU, 109: Internal bus

Claims (14)

an acquisition means for acquiring an evaluation value representing a feature of an output image obtained by inputting a training image used for learning the neural network into the neural network;
a generating means for generating an input image simply expressing the characteristics of the output image based on the evaluation value;
a reduction means for inputting the generated input image to the neural network and reducing the number of bits of parameters constituting a layer of the neural network;
An image processing device comprising: The image processing device according to claim 1, characterized in that the reduction means reduces the number of bits by inputting the input image into the trained neural network. The image processing device according to claim 1, characterized in that the evaluation value is an average histogram of the output image. The image processing device according to claim 3, characterized in that the average histogram is calculated by sequential calculation. The image processing device according to claim 1, characterized in that the generating means generates the input image based on a representative value of the evaluation values. The image processing device according to claim 5, characterized in that the representative values are the same number of data as the input image. The image processing device according to claim 1, characterized in that the training image and the correct answer image, which is the expected value of the output image of the neural network, are generated from a single original image. The image processing device according to claim 7, characterized in that the training images are generated by performing noise addition processing or degradation processing on the original images. The image processing device according to claim 7, characterized in that the correct image is generated by performing preprocessing on the original image. The image processing device according to claim 1, further comprising a calculation means for calculating an error between the output image and a correct image, which is an expected value of the output image of the neural network, and a selection means for selecting whether or not to obtain the evaluation value from the output image based on the error. An imaging device comprising an image processing device according to any one of claims 1 to 10. an acquisition step of acquiring an evaluation value representing a feature of an output image obtained by inputting a training image used for learning the neural network into the neural network;
a generating step of generating an input image simply expressing the characteristics of the output image based on the evaluation value;
a reduction step of inputting the generated input image to the neural network and reducing the number of bits of parameters constituting a layer of the neural network;
13. An image processing method comprising: A program for causing a computer to execute each step of the image processing method according to claim 12. A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to claim 12.

Images