JP2023084092A - Image processing device, image processing method, generation method, and program - Google Patents

Abstract

To provide an image processing device capable of suppressing gradation degrading of a processed image even when a bit number expressing a pixel number in a neural network is smaller than a bit number expressing a bit number of an inputted image.SOLUTION: An image processing device according to the present disclosure includes: gradation compression means which compresses a gradation of a first image; processing means which outputs image data to which prescribed image processing has been applied by applying a neural network for executing the prescribed image processing to the image data in which the gradation has been compressed by the gradation compression means; and gradation expansion means which expands a gradation of the image data to which the prescribed image processing has been applied. The gradation compression means compresses the gradation by using a characteristic where larger gradation is assigned as the bit number expressing a pixel value in the neural network is smaller than the bit number expressing the pixel value of the first image data, and as brightness is lower.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing device, an image processing method, a generation method and a program.

Conventionally, an image processing technique using a neural network (deep neural network) having a plurality of intermediate layers is known. Images taken with an imaging device (such as a camera) may contain noise depending on the settings of the imaging device at the time of shooting and the shooting conditions. can be removed. Japanese Patent Application Laid-Open No. 2002-200001 discloses a technique for outputting an image that has undergone processing such as compression noise removal and upsampling using a neural network.

JP 2019-121252 A

By the way, in general, processing using a neural network such as a convolutional neural network (CNN) requires a large amount of computation. Conceivable.

In the technology proposed in Patent Document 1, for example, when an image sensor outputs an image in which pixel values are represented by a number of bits greater than 8 bits, it is not considered that the CNN performs processing with an accuracy of INT8. . That is, in the technique proposed in Patent Document 1, when a CNN that expresses pixel values with a smaller number of bits than the image to be processed processes the above image, the number of bits of the pixel values is reduced and the gradation is lost. I have a problem.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problem, and an object of the present invention is to enable appropriate processing even in a neural network with a limited number of corresponding bits.

In order to solve this problem, for example, the image processing apparatus of the present invention has the following configuration. That is, by applying a gradation compression means for compressing the gradation of the first image data and a neural network for performing predetermined image processing on the image data whose gradation is compressed by the gradation compression means, , processing means for outputting the image data on which the predetermined image processing has been performed, and gradation decompression means for decompressing the gradation of the image data on which the predetermined image processing has been performed; The number of bits representing the pixel value inside is smaller than the number of bits representing the pixel value of the first image data, and the gradation compression means uses the characteristic that the lower the brightness, the more gradation is assigned. and compressing the tone.

According to the present invention, even a neural network with a limited number of corresponding bits can perform appropriate processing.

1 is a block diagram showing a functional configuration example of an imaging device according to a first embodiment; FIG. 1 is a block diagram showing a functional configuration example of an image processing system according to a first embodiment; FIG. 3 is a flow chart showing operations of inference processing according to the first embodiment; Graph explaining gamma correction in operation of inference processing of Embodiment 1 Flowchart showing operation of inference processing of the second embodiment Graphs for explaining gamma correction in the inference processing operation of the second embodiment Flowchart showing the operation of the inference processing of the third embodiment FIG. 11 is a diagram conceptually showing region division according to the third embodiment; FIG. 11 is a diagram conceptually showing region selection of a region-divided image according to the third embodiment; 4 is a flowchart showing the operation of learning processing according to the first embodiment; 1 is a diagram conceptually explaining a neural network according to a first embodiment; FIG. Flowchart showing the operation of the learning process of the second embodiment Flowchart (1) showing operation of inference processing of Embodiment 4 Flowchart (2) showing operation of inference processing of Embodiment 4 FIG. 10 is a diagram for explaining EOTF and OETF according to Embodiment 4; Flowchart (1) showing operation of inference processing of Embodiment 5 Flowchart (2) showing operation of inference processing of Embodiment 5

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

<Configuration example of imaging device>
First, with reference to FIG. 1, an example of the functional configuration of an imaging device that executes inference processing, which will be described later, will be described. In the following description, a case in which an imaging device such as a digital camera executes inference processing will be described as an example. However, as long as an input image can be acquired and an inference process can be executed, an image processing apparatus that does not include an imaging means can implement this embodiment. The imaging device or image processing device may be an electronic device other than a digital camera as long as it is an electronic device capable of executing inference processing. In the following embodiments, the terms "first" and "second" are used to facilitate understanding, and are not necessarily used in the scope of claims. The terms "first" and "second" do not refer to the same thing.

The imaging device 100 includes, for example, a processor 106, a ROM 105, a RAM 107, an image processing section 104, an optical lens 101, an imaging device 102, a frame memory 103, a video output driving section 108, a display driving section 110, and a metadata extracting section 112. These units are connected to the internal bus 113 . Each unit connected to the internal bus 113 can exchange data with each other via the internal bus 113 .

The optical lens 101 includes a lens and a motor for driving the lens. The optical lens 101 operates based on a control signal, can optically enlarge or reduce an image, and can adjust a focal length or the like. Moreover, when it is desired to adjust the amount of incident light, the amount of light can be adjusted so as to achieve desired brightness by controlling the aperture area of the diaphragm. Light transmitted through the lens forms an image on the imaging element 102 .

A CCD sensor, a CMOS sensor, or the like is used as the imaging device 102, and serves to convert an optical signal into an electrical signal. The image sensor 102 is driven based on a control signal to reset charges in pixels and control readout timing. The image sensor 102 has a function of performing gain processing on a pixel signal read out as an electrical analog signal (voltage value) and converting the analog signal into a digital signal. It may be performed outside the imaging device 102 .

The image processing unit 104 performs various image processing on the image output from the image sensor 102 . The image processing unit 104 corrects, for example, the amount of light in the peripheral portion of the image caused by the characteristics of the optical lens 101, corrects variations in sensitivity of each pixel of the image sensor 102, corrects colors, corrects flicker, and the like. be able to. The image processing unit 104 also has a function of performing noise reduction processing using a neural network. Details of the noise reduction processing will be described later. Note that the image processing unit 104 may be implemented by executing a program by the processor 106 or another processor such as a GPU (not shown).

Frame memory 103 includes a volatile storage medium. The frame memory 103 is called a RAM (Random Access Memory), for example, and is an element that temporarily stores video signals and can be read out when necessary. Since the video signal has a huge amount of data, a high-speed and high-capacity signal is required. In recent years, for example, DDR4-SDRAM (Dual Data Rate 4-Synchronous Dynamic RAM) is used. Using this frame memory 103 enables various kinds of processing. The frame memory 103 is useful for performing image processing such as synthesizing temporally different images and cutting out only a necessary area.

The processor 106 may be composed of one or more processors, and the processor includes, for example, a CPU (Central Processing Unit). The processor 106 may include a CPU, a GPU (Graphics Processing Unit), and a special-purpose processor for high-speed processing of specific operations such as machine learning. The processor 106 functions as a control unit for controlling each function of the imaging device 100 . A ROM (Read Only Memory) and a RAM are connected to the processor 106 . The ROM 105 is a non-volatile storage medium, and stores programs for operating the processor 106, various adjustment parameters, and the like. In addition, the ROM 105 may contain information of a trained model for executing inference processing, such as weight parameters and hyperparameters of a deep neural network (also simply referred to as a neural network) that have been trained. A program read from the ROM 105 is developed in the volatile RAM 107 and executed. The RAM 107 may be slower and have a lower capacity than the frame memory 103 . The neural network, for example, is configured to output an image in which the noise of the input image is reduced, but is not limited to this, and performs some predetermined image processing on the input image and outputs the resulting image. It may be configured as

The metadata extraction unit 112 extracts metadata information such as lens drive conditions and sensor drive conditions. An image generated by the image processing unit 104 is output to the outside of the imaging device 100 via the video output driving unit 108 and the video terminal 109 . An interface for outputting images to the outside enables real-time images to be displayed on an external monitor or the like. The interface may be, for example, SDI (Serial Digital Interface), HDMI (registered trademark) (High Definition Multimedia Interface), DisplayPort (registered trademark), and other various interfaces. An image generated by the image processing unit 104 is displayed on a display device via the display driving unit 110 and the display unit 111 .

The display unit 111 is a display device that allows the user to view display content. The display unit 111 can display, for example, images processed by the image processing unit 104 , setting menus, and the like, so that the user can check the operation status of the imaging device 100 . The display unit 111 can use, for example, a small and low power consumption device such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) as a display device. In some cases, the display unit 111 also includes a resistive or capacitive thin film element called a touch panel. The processor 106 generates a character string for informing the user of the setting state of the imaging device 100 and a menu for setting the imaging device 100, and superimposes them on the image processed by the image processing unit 104 for display. Displayed in section 111 . In addition to the character information, shooting assist displays such as histogram, vector scope, waveform monitor, zebra, peaking, false color, etc. may be superimposed.

<Overview of image processing system>
Next, the image processing system of this embodiment will be described with reference to FIG. The image processing system is a system capable of executing learning processing, which will be described later. The image processing system is a system composed of an imaging device 200 , an image processing device 210 , a display device 220 and a storage device 230 . The configurations of the optical lens 201 and the imaging device 202 of the imaging device 200 are substantially the same as the optical lens 101 and the imaging device 102 of the imaging device 100 . Frame memory 203, image processing unit 204, ROM 205, processor 206, and RAM 207 of image processing apparatus 210 have substantially the same configuration as frame memory 103, image processing unit 104, ROM 105, processor 106, and RAM 107, respectively. The metadata extraction unit 208 and internal bus 218 of the image processing device 210 have substantially the same configurations as the metadata extraction unit 208 and internal bus 113 of the imaging device 100, respectively. Therefore, detailed description of the configuration that is substantially the same as the configuration of the imaging device 100 will be omitted.

A camera control unit 209 in the imaging device 200 performs drive control of the optical lens 201 and the imaging element 202 based on the communication signal output from the camera communication connection unit 212 of the image processing device 210 .

An image signal receiving unit 211 of the image processing device 210 is a receiving unit that receives an image signal output from the imaging device 202 of the imaging device 200 . The GPU 213 includes one or more GPUs, and can execute neural network learning processing according to instructions from the image processing unit 104 or the processor 106 . Since a huge amount of calculation is required when executing the learning process, the present embodiment uses a GPU, which has a higher processing capacity for image processing than the CPU. GPU 213 may also be used to generate images for display on display device 220 . At that time, an image generated under the control of the GPU 213 is displayed on the display device 220 via the display drive section 216 and the display device connection section 217 .

Storage device 230 can be used to store a large amount of image data as training images. The storage device 230 may also store network parameters (such as neural network weight parameters) and hyperparameters updated by the learning process. The image processing device 210 exchanges data with the storage device 230 via the storage drive unit 214 and storage connection unit 215 of the system.

In this embodiment, an example will be described in which the image processing system shown in FIG. 2 is used during learning processing, and the imaging apparatus 100 shown in FIG. 1 is used during inference processing. However, the use is not limited to such use, and the inference processing may be executed in the image processing system shown in FIG. 2, for example. Further, in the present embodiment, as an example, it is assumed that the training images are images of Bayer array. However, an image captured using a three-chip image sensor may be used, or an image captured by a vertical color separation type image sensor such as a FOVEON sensor may be used. In addition to the Bayer array, other arrays (honeycomb structure, X-Trans CMOS sensor filter array, etc.) are similar. In the case of a Bayer array image, 1ch of the Bayer array may be left as is, or may be separated for each color channel and used as a training image. Furthermore, in the present embodiment, the case where one training image is input to the neural network and one image is output from the neural network will be described as an example. may be used.

(Embodiment 1)
<Operation of Inference Processing in Imaging Apparatus>
Next, operation of the inference processing in the imaging device 100 will be described with reference to FIGS. 3 and 4. FIG. A series of operations shown in FIG. 3 are realized by controlling each unit of the imaging apparatus 100 by executing a program stored in the ROM 105 by the processor 106, for example. Also, the operation of the image processing unit 104 may be realized by executing a program stored in the ROM 105 by the processor 106 or another processor such as a GPU (not shown).

In step S<b>3001 , the processor 106 sets the neural network parameters (network parameters) recorded in the ROM 105 to the neural network in the image processing unit 104 . As will be described later, the network parameters are, for example, weights and biases that make up the neural network. The network parameters set in step S3001 are calculated in advance by learning processing of the image processing system shown in FIG. 2, for example.

In step S<b>3002 , the image sensor 102 acquires the first image and outputs the acquired image to the image processing unit 104 . In step S3003, the image processing unit 104 performs correction processing on the first image. The correction processing here is correction processing for reducing variations in the optical lens 101 and the image sensor 102, and includes, for example, correction of peripheral light amount and correction of sensitivity variations for each pixel. However, this step may not be performed if correction processing is not required.

In step S3004, the image processing unit 104 applies a digital gain to the corrected image. In the digital gain process, if the pixel value has an offset, the image processing unit 104 subtracts the offset from the pixel value and then multiplies the gain, and then applies the offset to the pixel value multiplied by the gain. add

In step S3005, the image processing unit 104 generates a second image by subtracting the offset from each pixel value of the image multiplied by the digital gain in step S3004. The offset here is the black level added by the image sensor 102 . In step S3006, the image processing unit 104 generates a third image by normalizing each pixel value of the second image. Note that in this embodiment, each pixel value of the first image acquired from the image sensor 102 in step S3002 is 14 bits, and each pixel value up to the second image generated in step S3004 is 14 bits. It is represented by data of the number of bits. Normalization here divides each pixel value by 2 to the 14th power in order to normalize each pixel value of 14 bits to the range of 0 to 1, and the calculation result is handled in a format such as float32 including decimal places. .

In step S3007, the image processing unit 104 generates a fourth image by applying gamma correction to each pixel value of the third image. The gamma correction here is applied according to equation (1) below. For example, the gamma correction in this embodiment has the characteristic that more gradations are assigned to lower brightness.

In step S3008, the image processing unit 104 generates a fifth image that is denormalized so that each pixel value of the fourth image becomes 8 bits. In the denormalization here, the image processing unit 104 multiplies each pixel value by 2 to the eighth power to denormalize to 8 bits. The calculation result is handled in a format such as INT8. That is, the pixel value of the fifth image is represented by 8 bits.

Here, FIG. 4 shows gamma correction characteristics with the horizontal axis representing pixel values before gamma correction and the vertical axis representing pixel values after gamma correction. When γ is 1 or more in Equation (1), each pixel value of the fourth image draws a gamma curve shown in FIG. That is, with such gamma correction, it is possible to obtain a gamma correction result in which many values are assigned to the pixel values in the low-luminance region before gamma correction. It becomes possible to maintain the gradation of the lower bits.

In step S3009, the image processing unit 104 inputs the fifth image to the neural network. The neural network here is a trained neural network trained to optimally remove noise from the gamma-corrected image in step S3007.

In step S3010, the image processing unit 104 generates a seventh image by normalizing each pixel value of the sixth image output from the neural network. In this normalization, the image processing unit 104 divides each 8-bit pixel value by 2 to the eighth power to normalize each 8-bit pixel value to a range of 0 to 1. FIG. The calculation result is handled in a format such as float32 including decimal places.

In step S3011, the image processing unit 104 generates an eighth image by applying degamma correction to each pixel value of the seventh image. The degamma correction here is applied, for example, according to Equation (2) below.

In step S3012, the image processing unit 104 generates a ninth image by denormalizing each pixel value of the eighth image to 14 bits. In the denormalization here, the image processing unit 104 multiplies each pixel value by 2 to the 14th power to denormalize to 14 bits. Calculation results are handled in 14-bit format. In the present embodiment, a case of canceling normalization to 14 bits will be described as an example.

In step S3013, the image processing unit 104 generates a tenth image by adding an offset to each pixel value of the ninth image. In this embodiment, gamma correction as gradation compression and degamma correction as gradation expansion are used as an example, but other methods may be used. Further, in the present embodiment, the case of applying a digital gain before inputting to the neural network has been described as an example. However, the digital gain may be applied after passing through the neural network. In this case, each neural network uses an optimally learned image before digital gain is applied.

<Operation of learning processing in image processing system>
Next, with reference to FIGS. 10 and 11, operation of learning processing in the image processing system (imaging device 200, image processing device 210, display device 220, storage device 230) will be described. 10, the processor 206 of the image processing apparatus 210 develops and executes a program stored in the ROM 205 in the RAM 207, and each unit of the image processing apparatus 210 (image processing unit 204, GPU 213, etc.) is realized by controlling Also, the operation of the image processing unit 204 may be realized by the GPU 213 executing a program stored in the ROM 205 .

In step S<b>9001 , the processor 206 acquires a training image (noise image) and a correct image (teacher image) from the storage device 230 . Here, the training images are images containing noise. A correct image is an image that contains the same subject as the training image and has no noise (or very little noise). A training image can be generated, for example, by adding noise in a simulation to a correct image that is less affected by noise. Alternatively, an image of the same subject as the correct image that is actually captured under conditions where noise may occur (for example, high sensitivity setting) may be used. In this case, for example, the training images are images shot with low sensitivity, and the correct images are images shot with high sensitivity or images shot with low illuminance, and the sensitivity correction is performed to obtain the same level of accuracy as the correct images. This is an image that has been corrected for brightness. Noise patterns and object structures (such as edges) that are not included in the learning processing operation cannot be accurately inferred in the later inference processing operation. Therefore, a plurality of training images and correct images are prepared, which are generated so as to include various noise patterns and subject structures. A single noise amount may be used, or a plurality of noise amounts may be mixed.

At step S9002, the processor 206 normalizes the training image and the correct image obtained at step S9001 by dividing by the upper limit value (saturation luminance value) of the signal, and gamma make corrections. At step S9003, the processor 206 selects at least one of the plurality of training images gamma-corrected at step S9002 and inputs the selected training image to the neural network of the image processing unit 204 to generate an output image. do. At this time, the noise amount of the training image used in the operation of the learning process may be the same as that of other training images, or may be changed.

Processing performed by the neural network will be described with reference to FIG. FIG. 11 schematically shows processing by a neural network. In the example shown in FIG. 11, a convolutional neural network (CNN) is described as an example, but this embodiment is not limited to CNN. A GAN (Generative Adversarial Network) may be used as a neural network that outputs an image. Alternatively, the neural network may have a skip connection or the like, or may be a recursive neural network such as an RNN (Recurrent Neural Network).

An input image 1001 shown in FIG. 11 represents an image to be input to the neural network or a feature map, which will be described later. Operation symbol 1002 represents a convolution operation. A convolution matrix 1003 is a filter that performs a convolution operation on the input image 1001 . A bias 1004 is added to the result output by the convolution operation between the input image 1001 and the convolution matrix 1003 . A feature map 1005 is the result of the convolution operation with the bias 1004 added. In FIG. 11, the numbers of neurons, intermediate layers, and channels are reduced for the sake of simplicity, but the number of neurons and layers, the number of connections between neurons, weights, and the like are not limited to these. . Also, when implementing in FPGA or the like, the connections and weights between neurons may be reduced. In the present embodiment, the learning processing operation and the inference processing operation are executed collectively for a plurality of color channels, but the learning processing operation and the inference processing operation may be executed individually for each color.

式（３）において、ｍａｘは、引数のうち最大値を出力する関数を表す。 In CNN, a feature map of the input image is obtained by convolving the input image with a certain filter. Note that the size of the filter is arbitrary. In the next layer, a different feature map is obtained by convolving the feature map of the previous layer with another filter. At each layer, an input signal is multiplied by a filter weight representing the strength of the connection and summed with the bias. By applying an activation function to this result, we obtain the output signal at each neuron. The weights and biases in each layer are called network parameters, and their values are updated by the operation of learning processing. Examples of commonly used activation functions include a sigmoid function and a ReLU function. In the present embodiment, a Leaky ReLU function according to the following equation (3) is used, but is not limited to this.

In Expression (3), max represents a function that outputs the maximum value among the arguments.

A CNN may have multiple layers for repeatedly performing convolution operations, followed by, for example, one or more fully connected layers, to which an output layer may be connected.

In step S9004, the image processing unit 204 performs image processing on each of the output image of the neural network and the correct image. The inference accuracy of noise reduction processing during inference can be improved by matching the conditions of the image processing performed in the inference processing operation and the image processing performed in the learning processing operation. Note that the image processing may be performed at any time before steps S9004 and S9005. For example, it may be performed on the input side of a neural network. When a plurality of patterns of noise amount are applied to the training image used in the operation of the learning process, noise can be effectively removed even if a captured image having a noise amount outside the learning is input during inference. If the number of training images is not sufficient, padding processing such as cropping, rotation, and reversal may be performed. In that case, it is necessary to apply the same processing to the correct image.

In step S9005, the image processing unit 204 calculates the error between the output image subjected to image processing in step S9004 and the correct image. The correct image also has an array of color components arranged in the same manner as the training image. The error is generally calculated using the mean square error of each pixel or the sum of the absolute values of the differences of each pixel, but other indices may be used. In step S9006, the image processing unit 204 updates each parameter of the neural network using error backpropagation so that the error calculated in step S9005 becomes smaller. However, this embodiment is not limited to this. The update amount of each parameter may be fixed or may be varied.

Next, in step S9007, the processor 206 determines whether or not a predetermined end condition is satisfied.If the condition is not satisfied, the process returns to step S9001 to proceed with learning again. On the other hand, if the predetermined termination condition is satisfied, the process proceeds to step S9008. The predetermined termination condition may be when the number of times of learning reaches a predetermined value, or when the error is less than or equal to a predetermined value. Alternatively, the process may be terminated when the error is almost no longer reduced, or at the discretion of the user. Next, in step S<b>9008 , the processor 206 causes the storage device 230 to store information regarding the network parameters updated by learning, the structure of the neural network, and the like. A storage device 230 may be used to store the output network parameters. In the present embodiment, the description is based on the premise that the data is stored in the storage device, but other storage media may be used.

In step S9009, the processor 206 quantizes the parameters of the neural network learned by the FP32 to INT8. The bit width and data type of data are not limited to these, and parameters of FP16 may be used, or INT4 may be quantized. At step S9010, the processor 206 stores the quantized network parameters in the parameter storage area. The processor 206 ends the operation of the learning process with the above operations. Through this learning process, a trained neural network can be obtained.

As for processing other than noise reduction, learning processing operations can be executed by similarly preparing a pair of a training image and a correct image in a simulation. For example, in super-resolution, training images can be prepared by downsampling correct images. At this time, the correct image and the training image may or may not be matched in size. In the case of deblurring or deblurring, a training image can be prepared by applying a blurring function to a correct image. In the case of white balance correction, an image that has not been properly adjusted or corrected may be used as a training image with respect to a correct image that has been photographed with an appropriate white balance. The same applies to color correction such as color matrix correction. With loss interpolation, a training image can be obtained by missing a correct image. In the case of demosaicing, correct images may be prepared using a three-chip imaging device or the like, and training images may be prepared by re-sampling the correct images using a Bayer array or the like. Furthermore, in inference of color components, a training image can be prepared by reducing the color components with respect to the correct image. As for dehaze, a training image can be prepared by adding scattered light obtained by simulating a physical phenomenon to a correct image without haze or the like. In addition, in the case where a plurality of frames such as a moving image continue, noise can be removed more effectively by collectively inputting a desired number of frames in the depth direction to the neural network.

As described above, in this embodiment, in a configuration in which the number of bits representing pixel values inside the neural network is smaller than the number of bits representing pixel values of image data to be processed, first, the gradation of the image data is adjusted. Compress. Specifically, the gradation of the image data is compressed so that the lower the brightness, the more gradation is assigned. Then, a neural network that performs predetermined image processing is applied to the image data whose gradation is compressed to generate an output image, and the image data output from the neural network is processed to expand the gradation. implement. By doing so, even a neural network with a limited number of corresponding bits can perform appropriate processing. In other words, it is possible to use a neural network with a smaller computational load while suppressing a decrease in image gradation.

(Embodiment 2)
Next, Embodiment 2 will be described. In the first embodiment, gamma correction with predetermined characteristics is applied for gradation compression. Different from form 1. However, the configuration of the imaging device 100 and the functional configuration example of the image processing system may be substantially the same as those of the first embodiment. Therefore, substantially the same configurations and processes are denoted by the same reference numerals, and descriptions thereof are omitted, and differences are mainly described.

<Operation of Inference Processing in Imaging Apparatus>
The operation of inference processing in the imaging device 100 will be described below with reference to FIGS. 5 and 6. FIG. A series of operations shown in FIG. 5 are realized by controlling each unit of the imaging apparatus 100 by executing a program stored in the ROM 105 by the processor 106, for example. Further, the operation of the image processing unit 104 may be realized by executing a program stored in the ROM 105 by the processor 106 or another processor such as a GPU (not shown). First, the processor 106 or the image processing unit 104 performs the same processing as in the first embodiment in steps S3002 to S3005 to generate a second image.

In step S6001, the image processing unit 104 calculates the brightness of the second image from which the offset has been removed in step S3005. In this embodiment, an example of calculating the average value of the pixel values of the second image as the brightness will be described, but the brightness may be calculated from a value obtained by converting each pixel value into luminance.

In step S6002, the processor 106 refers to the first lookup table showing the relationship between the average value of each pixel value recorded in the ROM 105 and the γ value for gamma correction. Based on the first lookup table, the processor 106 sets a γ value for gamma correction corresponding to the average value of each pixel value in the image processing unit 104 . In step S6003, the processor 106 refers to the second lookup table showing the relationship between the average value of each pixel value recorded in the ROM 105 and the γ value for degamma correction. Based on the second lookup table, the processor 106 sets a γ value for degamma correction corresponding to the average value of each pixel value in the image processing unit 104 . Note that in this embodiment, the case of setting the γ value (characteristic) for degamma correction according to the average value of each pixel value is described as an example. A characteristic for degamma correction may be set.

In step S6004, the processor 106 refers to the third lookup table showing the relationship between the average value of each pixel value recorded in the ROM 105 and the parameters of the neural network. Based on the third lookup table, the processor 106 sets parameters of the neural network corresponding to the average value of each pixel value to the neural network in the image processing unit 104 . In the present embodiment, a case of acquiring network parameters corresponding to the average value of each pixel value is described as an example. good. For example, by referring to a lookup table in which neural network parameters are associated with γ values for gamma correction that differ in stages, the image processing unit 104 obtains neural network parameters corresponding to the γ values set in step S6002. can be set to

The processor 106 further executes steps S3006 to S3013 in the same manner as in the first embodiment, performs degamma processing and the like on the eighth image generated by the neural network, and then generates a tenth image. After generating the tenth image, the processor 106 ends the process.

Here, with reference to FIG. 6, the characteristics of gamma correction applied in this embodiment will be described. In FIG. 6, the horizontal axis indicates pixel values before gamma correction, and the vertical axis indicates pixel values after gamma correction. FIG. 6 shows gamma curves when γ is 2.6, 2.4, 2.2, 2.0, 1.8, 1.6, and 1.4 in the above equation (1). there is In this embodiment, in the operations of steps S6001 to S6004, a specific gamma curve corresponding to the average value of each pixel value is associated. For example, processor 106 selects a gamma curve (characteristic) of γ=2.6 when each pixel value of the image is below a predetermined low luminance threshold. By doing so, it is possible to obtain a gamma correction result in which many gradations are assigned to pixel values in the low-luminance region before gamma correction. It becomes possible to maintain the gradation of pixels in the luminance range. Also, the processor 106 selects a gamma curve of γ=1.4 when each pixel value of the image is higher than a predetermined threshold value for high brightness. By doing so, it is possible to obtain a gamma correction result in which more gradations are assigned to the pixel values in the high-luminance region before gamma correction compared to other gamma curves. Therefore, it is possible to maintain the gradation of pixels in the high-luminance region when denormalization to 8 bits is performed in step S3008.

<Operation of learning processing in image processing system>
Next, with reference to FIG. 12, operation of learning processing according to the second embodiment in the image processing system (imaging device 200, image processing device 210, display device 220, storage device 230) will be described. In the series of operations shown in FIG. 12, the processor 206 of the image processing apparatus 210 develops and executes a program stored in the ROM 205 in the RAM 207, and each unit of the image processing apparatus 210 (image processing unit 204, GPU 213, etc.) is executed. Realized under control. Also, the operation of the image processing unit 204 may be realized by the GPU 213 executing a program stored in the ROM 205 . First, the processor 206 or the image processing unit 204 performs processing from step S9001 to step S9008 as in the first embodiment.

In step S10009, the processor 206 determines whether the network parameters of the neural network (eg, corresponding to the mean value of each stepped pixel value) for all conditions have been obtained. Inference accuracy of noise reduction processing at the time of inference can be improved by matching the conditions of the operation of switching network parameters and the operation of learning processing in S6004 of the inference processing. Therefore, when image processing is performed under multiple conditions (or conditions are switched) during inference, it is advantageous to have network parameters for each condition. If the processor 206 determines that network parameters for all conditions have been acquired, it moves to step S9009. On the other hand, if the processor 206 determines that the network parameters for all the conditions have not been acquired, it proceeds to step S10010 and changes the conditions. The processor 206 then returns the process to step S9001 and performs the above process again. Network parameters are stored in the parameter storage area for each condition. A parameter storage area may be the ROM 205 or the RAM 207 . Moreover, the network parameters stored in the parameter storage area may be stored in the storage device 230 as necessary. Processor 206 also executes steps S9009 and S9010 in the same manner as in FIG.

As described above, in the present embodiment, among a plurality of gamma correction characteristics that differ in stages, the gamma correction characteristics corresponding to the brightness of the image are used to compress the gradation. Image processing (such as noise reduction processing) is performed using different network parameters according to the brightness of the image among the network parameters of a plurality of neural networks learned in advance. Furthermore, the gradation of the image data is expanded using the characteristic corresponding to the brightness of the image among the plurality of degamma correction characteristics for expanding the gradation. Also, in the learning process, optimal network parameters are acquired and stored for each condition (for example, each image brightness). By doing so, it is possible to obtain a neural network whose inference accuracy is less likely to be affected by image processing that is affected by changes in image brightness and other conditions.

In the above example, a neural network for reducing image noise has been described as an example. However, for processing other than noise reduction, learning processing operations can be executed by similarly preparing pairs of training images and correct images through simulation. In super-resolution, training images can be prepared by downsampling the correct images. At this time, the correct image and the training image may or may not be matched in size. For deblurring and deblurring, a training image can be generated by applying a blurring function to the correct image. In the case of white balance correction, an image that has not been properly adjusted or corrected may be used as a training image with respect to a correct image that has been photographed with an appropriate white balance. The same applies to color correction such as color matrix correction. With loss interpolation, a training image can be obtained by missing a correct image. In the case of demosaicing, a training image may be generated by preparing a correct image using a three-chip imaging device or the like and resampling the correct image using a Bayer array or the like. In inference of color components, training images can be generated by reducing the color components of correct images. As for dehaze, a training image can be generated by adding scattered light obtained by simulating a physical phenomenon to a correct image without haze or the like. In addition, in the case where a plurality of frames such as a moving image continue, noise can be removed more effectively by collectively inputting a desired number of frames in the depth direction to the neural network.

In this embodiment, the case of uniquely selecting the γ value corresponding to the brightness using the lookup table has been described as an example. However, if the γ value is greatly changed, the luminance change of the output image becomes large and the image may become difficult to see. Therefore, instead of using a lookup table to uniquely select a γ value corresponding to the brightness, the current γ value is changed to a nearby γ value recorded in the ROM 205 according to the brightness. You can do it. That is, among a plurality of stepwise different γ values (characteristics) of gamma correction contained in a lookup table as shown in FIG. 2.4 or 2.0). In this case, the γ value is changed to the target γ value over time without a large change at once. Also, in this case, by referring to a lookup table that associates neural network parameters with each γ value, it is possible to set the corresponding neural network parameters when the above neighboring γ values are set. is. Further, with respect to degamma correction, degamma correction may be performed using characteristics of degamma correction corresponding to γ values in the vicinity of the above-described degamma correction, among stepwise different degamma correction specifications. Further, in the present embodiment, digital gain is applied before input to the neural network, but digital gain may be applied after processing by the neural network. In this case, each neural network prepares an optimally learned image before digital gain is applied.

(Embodiment 3)
Next, Embodiment 3 will be described. In the second embodiment, the brightness of the second image is obtained, and the gamma correction value is set based on the obtained brightness. Embodiment 3 differs from Embodiment 2 in that the brightness is obtained for each region of the second image and the gamma correction value is set based on the brightness of the region. However, the configuration of the imaging device 100 and the functional configuration example of the image processing system may be substantially the same as those of the first embodiment. Therefore, substantially the same configurations and processes are denoted by the same reference numerals, and descriptions thereof are omitted, and differences are mainly described.

<Operation of Inference Processing in Imaging Apparatus 100>
The operation of inference processing in the imaging device 100 will be described with reference to FIG. 7 . A series of operations shown in FIG. 7 are realized by controlling each unit of the imaging apparatus 100 by executing a program stored in the ROM 105 by the processor 106, for example. Further, the operation of the image processing unit 104 may be realized by executing a program stored in the ROM 105 by the processor 106 or another processor such as a GPU (not shown). First, the processor 106 or the image processing unit 104 performs the processing from step S3002 to step S3005 in the same manner as in the second embodiment (FIG. 5) to generate a second image.

In step S<b>7001 , the processor 106 acquires the coordinate information for region division for calculating the brightness of each region recorded in the ROM 105 . Then, the processor 106 sets the coordinates of the divided regions for the second image in the image processing unit 104 based on the coordinate information for dividing the regions. The coordinate information for area division is, for example, the starting point coordinates and the ending point coordinates of each area based on the coordinate origin when the upper left corner pixel of the image is the coordinate origin of the image (X, Y)=(0, 0). (X, Y). Alternatively, the coordinate information for region division may be composed of the starting point coordinates (X, Y) of each region and the width and height of the region. Alternatively, the processor 106 may calculate coordinate information for area division from information on the width and height of the second image and information on the number of divisions in the X and Y directions.

For example, in this embodiment, as shown in FIG. 8, the second image is divided into four regions in the X direction and the Y direction, respectively, so that a total of 16 regions (regions 801 to 816) are obtained. explain. FIG. 8 schematically shows an example of an image including a dark sea and light from a lighthouse or an electric light, which is captured by the image capturing apparatus 100 for harbor surveillance, for example. When gamma correction characteristics are set using the brightness of the entire image, it is affected by the light from a lighthouse or an electric light, but it is desirable to obtain a gradation that corresponds to the dark sea. For this reason, in the present embodiment, the processing of steps S7002 to S7004 is performed to acquire the brightness of the dark sea area.

In step S7002, the image processing unit 104 calculates the brightness of each region of the second image based on the coordinates of the divided regions set in step S7001. In the present embodiment, an example will be described in which the brightness is calculated as the average value of the pixel values in each region of the second image. In step S7003, the processor 106 selects the area of the second image (according to the average value of the pixel values for each area calculated in step S7002) based on the area selection conditions recorded in the ROM 105. Set to 104. As for the area selection condition, the user may select one or more arbitrary areas, or one or more arbitrary areas may be selected based on the number of areas to be used and the brightness priority of the average value of each pixel value. You may Alternatively, one or more regions in which the average value of each pixel value is lower than a predetermined threshold may be selected. For example, let us consider a case where the number of regions to be used is eight and a region with a dark average pixel value is prioritized as a region selection condition. In this case, processor 106 selects the eight regions shown in FIG. 9 (regions 804, 808, 809, 811, 812, 813, 815 and 816) as regions of the second image to be used in step S7004.

In this embodiment, the area of the second image to be used in step S7004 is selected according to the above-described area selection conditions, but selection of the area is not limited to this. For example, when the imaging apparatus 100 is used as a surveillance camera, the image area may be selected as follows. For example, the area of the second image to be used may be selected for each hour based on the time information of the previous day and earlier and the brightness information for each area. Also in that case, the brightness of each area to be used is calculated, and if the deviation from the brightness of the same area before the previous day is greater than a predetermined value, it is determined that the brightness is different from that before the previous day, and is used. You can change the area. In other words, an area is selected in which the difference from the brightness of the same area before the previous day is equal to or less than a predetermined value. In addition, the brightness of each area to be used is calculated, and if the difference in the brightness change from the same area calculated last time is greater than a predetermined value from the brightness change in the same area and at the same time period on the previous day or earlier, , the area to be used may be changed based on the determination that there is a special condition different from that of the day before. For example, in the time period from sunset to nighttime, if the same area in the same time period before the previous day is dark, but the brightness in the same area in the same time period on the current day rises by a predetermined threshold or more, the illumination is turned off. It is conceivable that the light is turned on or that the light that is turned on moves. Through the above-described processing, it can be determined that such an area is not suitable as an area for selecting gamma correction characteristics.

In step S7004, the image processing unit 104 calculates the brightness of each area of the second image set in step S7003. The brightness may be calculated by dividing the sum of the average values of the pixel values for each area to be used by the number of areas to be used. Alternatively, the average value of each pixel value for each region calculated in step S7002 is stored in the RAM 107, only the average value of each pixel value in the region to be used is read, and the sum is divided by the number of regions to be used. good.

After that, the image processing unit 104 executes the operations from step S6002 to step S3013 as described with reference to FIG. 5, and ends this series of processes after the process of step S3013.

As described above, in this embodiment, the brightness is calculated for each predetermined region in the image, and the gamma correction corresponding to the calculated brightness and the corresponding neural network are applied. By doing so, in addition to the effects of the above-described embodiments, when processing an image with a large difference in brightness for each area, many gradations are assigned to dark areas in the image.

In the above description, the processing for calculating the brightness of an image or an area of an image within the device was explained as an example. You may make it acquire the value calculated by.

(Embodiment 4)
Next, Embodiment 4 will be described. In the fourth embodiment, image processing is performed using a neural network after tone correction is applied to the image according to the setting of the image to be output. Note that the configuration of the imaging device 100 and the functional configuration example of the image processing system may be substantially the same as those in the first embodiment. Therefore, substantially the same configurations and processes are denoted by the same reference numerals, and descriptions thereof are omitted, and differences are mainly described.

<Operation of Inference Processing in Imaging Apparatus 100>
13A and 13B, the operation of inference processing in the imaging device 100 will be described. A series of operations shown in FIGS. 13A and 13B are realized by controlling each unit of the imaging apparatus 100 by executing a program stored in the ROM 105 by the processor 106, for example. Further, the operation of the image processing unit 104 may be realized by executing a program stored in the ROM 105 by the processor 106 or another processor such as a GPU (not shown).

First, in step S13001, the processor 106 determines whether the image output mode set in the imaging apparatus 100 is HDR (High Dynamic Range) mode or SDR (Standard Dynamic Range) mode. The processor 106 determines whether the image output mode is the HDR mode or the SDR mode, for example, by referring to the setting values stored in the RAM 107 . If the processor 106 determines that the set image output mode is the HDR mode, the process proceeds to step S13002; otherwise, the process proceeds to step S13003. In this embodiment, an example will be described in which the setting of the image to be output is a setting indicating whether the image output from the imaging device 100 is HDR or SDR. However, the setting of the image to be output is not limited to this. For example, the setting may indicate whether the image output by this series of processes is HDR or SDR.

In step S<b>13002 , the processor 106 sets the parameters of the first neural network recorded in the ROM 105 to the neural network in the image processing unit 104 . Here, the first neural network is a neural network optimized for input corresponding to an HDR image having, for example, 1024 levels of gradation. In step S<b>13003 , the processor 106 sets the parameters of the second neural network recorded in the ROM 105 to the neural network in the image processing unit 104 . Here, the second neural network is a neural network optimized for input corresponding to an SDR image having fewer gradations than HDR (for example, 256 levels). In the present embodiment, an example is described in which a neural network is set according to the setting of the number of gradations of an image to be output. However, the neural network may be set according to the setting of the maximum value (or upper limit value) of the brightness of the image to be output. In this embodiment, a case of setting a neural network associated with the output mode of each image will be described as an example. However, if both HDR and SDR can be processed by one neural network, the determination process in step S13001 does not have to be performed. Subsequently, the processing from step S3002 to step S3005 is executed in the same manner as in the above embodiment.

In step S13004, the processor 106 determines whether the set image output mode is the HDR mode or the SDR mode, as in step S13001. If the processor 106 determines that the image output mode is the HDR mode, the process proceeds to step S13005; otherwise, the process proceeds to step S13007.

In step S13005, the image processing unit 104 generates a third image by normalizing each pixel value of the second image. Note that in the example of the present embodiment, for example, each pixel value of the first image acquired from the image sensor 102 in step S3002 is 14 bits, whereas each pixel value of the image generated by digital gain is applied in step S3004. A case where pixel values are handled in 18 bits will be described as an example. The normalization in this step is a process of associating each 18-bit pixel value with a range from 0 to 1, and the image processing unit 104 divides each pixel value by 2 to the 18th power. The calculation result is handled in a format such as float32 including decimal places.

In step S13006, the image processing unit 104 generates a fourth image by multiplying each pixel value of the third image by OETF (Opto-Electronic Transfer Function) of the PQ curve. That is, the image processing unit 104 compresses the gradation of the third image by applying the OETF of the PQ curve to each pixel value of the third image. The OETF will be described later with reference to FIG. 14B, but the OETF has the characteristic that more gradations are assigned as the brightness is lower. In addition, OETF has the characteristic of assigning more gradations in a predetermined low-luminance region than the characteristic of gamma correction applied to an image in SDR mode (step S13008 will be described later). In other words, in setting the image output mode, the image processing unit 104 uses the characteristic that the greater the number of bits representing the pixel values of the image to be processed, the more gradations are assigned in a predetermined low-luminance region. .

The PQ curve is defined in ITU-R (Radiocommunication Sector of ITU) BT. 2100 compliant EOTF (Electro-Optical Transfer Function).

The PQ curve will be explained with reference to FIG. FIG. 14A shows an example of the PQ curve (EOTF). The EOTF corresponds to a function that converts the gradation value (luminance gradation value), which is an image signal, into the luminance of the light output. Specifically, the EOTF in FIG. 14(A) is represented by Equation 4 below. The _pin is an EOTF input value, and is a value obtained by normalizing the gradation value (R value, G value, B value, etc.) to 0.0 to 1.0. p _in =1.0 corresponds to the upper limit of the gradation value (upper limit according to the number of bits), and p _in =0.0 corresponds to the lower limit of the gradation value. For example, when the number of bits of the gradation value is 10 bits, the upper limit of the gradation value is 1023 and the lower limit of the gradation value is 0. p _out is an EOTF output value, which is a gradation value obtained by normalizing a gradation value (R value, G value, B value, etc.) proportional to luminance to 0.0 to 1.0. For example, p _out =0.0 corresponds to 0 nits and p _out =1.0 corresponds to 100000 nits. max[x,y] is a function that outputs the larger of x and y.

FIG. 14(B) shows an example of an OETF having characteristics opposite to those of the EOTF of FIG. 14(A). OETF corresponds to a function that converts luminance into a gradation value of an image signal. Specifically, the OETF in FIG. 14B is represented by Equation 5 below. q _in is an input value of OETF, and is a gradation value obtained by normalizing a gradation value (R value, G value, B value, etc.) proportional to luminance to 0.0 to 1.0. For example, q _in =0.0 corresponds to 0 nits and q _in =1.0 corresponds to 10000 nits. q _out is an output value of OETF, which is a value obtained by normalizing the gradation value (R value, G value, B value, etc.) to 0.0 to 1.0. q _out =1.0 corresponds to the upper limit of the gradation value (upper limit according to the number of bits), and q _out =0.0 corresponds to the lower limit of the gradation value. For example, when the number of bits of the gradation value is 10 bits, the upper limit of the gradation value is 1023 and the lower limit of the gradation value is 0.

In step S13007, the image processing unit 104 performs processing for clipping each pixel value of the second image generated in step S3005 by a predetermined value (for example, a pixel value equal to or greater than a predetermined value is set as a predetermined value). Generate a fifth image with normalized values. The predetermined value is, for example, the upper limit of the dynamic range sufficient for SDR. For example, the image processing unit 104 clips at 16383 of 14 bits. In the normalization of this step, the image processing unit 104 divides each pixel value by 2 to the 14th power in order to normalize each pixel value of 14 bits to the range of 0 to 1. FIG. The calculation result is handled in a format such as float32 including decimal places. In step S13008, the image processing unit 104 generates a sixth image by gamma-correcting each pixel value of the fifth image. The gamma correction here is the same processing as in step S3007 described above.

In step S13009, the image processing unit 104 generates a seventh image by denormalizing each pixel value of the fourth image generated in step S13006 or the sixth image generated in step S13008 to 8 bits. The cancellation of normalization in this step is the same processing as in step S3008 described above. In step S13010, the image processing unit 104 inputs the seventh image to the neural network. The neural network used in this step is the first or second neural network set in step S13002 or S13003, and is a neural network that removes noise from the image. That is, in the present embodiment, a neural network is applied using different parameters among the plurality of previously learned neural network parameters according to the setting of the image output mode.

In step S13011, the image processing unit 104 generates a ninth image by normalizing each pixel value of the eighth image output from the neural network. The normalization in this step is the same processing as in step S3010. In step S13012, the processor 106 determines whether the set image output mode is the HDR mode or the SDR mode, as in step S13001. If the processor 106 determines that the image output mode is the HDR mode, the process proceeds to step S13013; otherwise, the process proceeds to step S13015.

Since the image output mode is the HDR mode, in step 13013, the image processing unit 104 multiplies each pixel value of the ninth image by the EOTF of the PQ curve to generate the tenth image. Then, in step S13014, the image processing unit 104 generates an eleventh image by denormalizing each pixel value of the tenth image to 18 bits. In the denormalization in this step, the image processing unit 104 denormalizes to 18 bits, so each pixel value is multiplied by 2 to the 18th power. Calculation results are handled in 18 bits.

Since the image output mode is the SDR mode, in step S13015, the image processing unit 104 applies degamma correction to each pixel value of the ninth image to generate the twelfth image. The degamma correction process in this step is the same process as in step S3011. In step S13016, the image processing unit 104 generates a 13th image by denormalizing each pixel value of the 12th image to 14 bits. The cancellation of normalization in this step is the same processing as in step S3012. As described above, in this embodiment, the gradation of image data is expanded using different characteristics among a plurality of characteristics for expanding gradation (the characteristics of EOTF and degamma correction) according to the setting of the image output mode. .

In step S13017, the image processing unit 104 generates a 14th image by α-blending the 11th image generated in step S13014 or the 13th image generated in step S13016 with the second image. Alpha-blending is to synthesize two images based on a weight (alpha value) set for each pixel. The image processing unit 104 performs α-blending according to the pixel values of the input image. The image processing unit 104 multiplies the 14th image by the coefficient (1-α), multiplies the 11th image or the 13th image by the coefficient α, and then adds the multiplied results. The α value at this time is linearly transformed between 0 and 1 according to the magnitude of the pixel value. However, it is not always necessary to perform this α-blending.

In step S13018, the image processing unit 104 generates a fifteenth image by adding an offset to each pixel value of the fourteenth image. After generating the fifteenth image, the processor 106 terminates this series of processes. Note that, in this embodiment, OETF and gamma correction of the PQ curve are applied in order to perform gradation compression. In addition, EOTF and degamma correction of the PQ curve are applied in order to extend the gradation. However, other transfer functions and transformation characteristics may be used for tone compression and tone decompression.

As described above, in the present embodiment, the tone correction corresponding to the setting of the image to be output is applied to the image, and then the image processing is performed by the neural network. By doing so, neural network processing with a limited number of bits can be performed while maintaining the necessary dynamic range according to the setting of the image to be output (for example, HDR or SDR). Also, an image (for example, a second image) before processing using a neural network is used, and an image after processing using a neural network is alpha-blended. By doing so, it becomes possible to restore information lost by bit reduction.

In this embodiment, the case where the setting of the image to be output is HDR or SDR (that is, the number of gradations of the image to be output is different) has been described as an example. However, the setting of the image to be output may be a characteristic (for example, OETF/EOTF, γ value) that compresses or expands the gradation of the image. Alternatively, the setting of the image to be output may be the number of bits representing the pixel value of the image to be output, or the upper limit value of the pixel value.

Further, in the present embodiment, the case where the characteristics used for tone compression, the parameters of the neural network, and the characteristics used for tone expansion are controlled according to the setting of the image to be output has been described as an example. However, the properties used for tone compression, the parameters of the neural network, and the properties used for tone expansion may be controlled based on other information. For example, the characteristics used for tone compression may be controlled based on the settings of the image input to the neural network. The setting of the image input to the neural network may be, for example, the number of bits representing the pixel values of the image, the number of gradations of the image, or the upper limit of the pixel values of the image.

(Fifth embodiment)
Furthermore, Embodiment 5 will be described. In the fifth embodiment, after clipping input image data, image processing using gradation processing and a neural network is performed. Note that the configuration of the imaging device 100 and the functional configuration example of the image processing system may be substantially the same as those in the first embodiment. Therefore, substantially the same configurations and processes are denoted by the same reference numerals, and descriptions thereof are omitted, and differences are mainly described.

<Operation of Inference Processing in Imaging Apparatus 100>
The inference processing performed by the imaging device 100 will be described with reference to FIGS. 15A and 15B. A series of operations shown in FIGS. 15A and 15B are realized by controlling each unit of the imaging apparatus 100 by executing a program stored in the ROM 105 by the processor 106, for example. Further, the operation of the image processing unit 104 may be realized by executing a program stored in the ROM 105 by the processor 106 or another processor such as a GPU (not shown).

First, the processing from step S3001 to step S3005 is executed in the same manner as in the above-described embodiment to set parameters of the neural network and generate a second image by subtracting the offset from the pixel values of the image.

Next, in step S15001, the image processing unit 104 generates a third image by clipping the pixel values of the second image by a predetermined value (for example, setting pixel values equal to or greater than a predetermined value to a predetermined value). In this embodiment, the case where an image clipped at 1023 is generated with the upper limit of the pixel value set to 10 bits will be described as an example. Any number of bits may be used to clip pixel values.

In step S15002, the image processing unit 104 generates a fourth image by normalizing each pixel value of the third image. In the normalization of this step, the image processing unit 104 divides each pixel value by 2 to the 10th power in order to normalize each pixel value of 10 bits to the range of 0 to 1. FIG. The calculation result is handled in a format such as float32 including decimal places. In step S15003, the image processing unit 104 generates a fifth image by gamma-correcting each pixel value of the fourth image. The gamma correction processing in this step is the same processing as in step S3007 described above.

In step S15004, the image processing unit 104 generates a sixth image by denormalizing each pixel value of the fifth image to 8 bits. The cancellation of normalization in this step is the same processing as in step S3008. In step S15005, the image processing unit 104 inputs the sixth image to the neural network. This neural network is a neural network that removes noise that is optimally trained on the gamma-corrected image in step S15003.

In step S15006, the image processing unit 104 generates an eighth image by normalizing each pixel value of the seventh image output from the neural network. The normalization here is the same processing as in step S3010. In step S15007, the image processing unit 104 generates a ninth image by applying degamma correction to each pixel value of the eighth image. The degamma correction process in this step is the same process as in step S3011. In step S15008, the image processing unit 104 generates a tenth image by denormalizing each pixel value of the ninth image to 10 bits. In the denormalization process of this step, the image processing unit 104 denormalizes to 10 bits, so each pixel value is multiplied by 2 to the 10th power. Calculation results are handled in 10 bits.

In step S15009, the image processing unit 104 generates an eleventh image by α-blending the tenth image generated in step S15008 and the second image. The image processing unit 104 performs α-blending according to the input pixel values. The image processing unit 104 multiplies the 14th image by the coefficient (1-α), multiplies the 11th image or the 13th image by the coefficient α, and then adds the multiplication results. The α value at this time is linearly transformed between 0 and 1 according to the magnitude of the pixel value. However, it is not always necessary to perform this α-blending.

In step S15010, the image processing unit 104 generates a twelfth image by adding an offset to each pixel value of the eleventh image. After generating the twelfth image, the processor 106 ends this series of processes.

As described above, in this embodiment, the data input to the neural network is clipped. By doing so, it is possible to generate an image in which much of the information in the dark part remains. Furthermore, the clipped and lost bright area information is alpha blended with the image before processing using the neural network. By doing so, it is possible to restore the information that has been clipped and lost.

The present invention also includes the execution of a software program that implements the functions of the above-described embodiments. Therefore, in order to implement the functional processing of the present invention in a computer, the program code itself supplied and installed in the computer also implements the present invention. In other words, the present invention also includes the computer program itself for realizing the functional processing of the present invention. In that case, as long as it has the function of a program, the form of the program, such as an object code, a program executed by an interpreter, or script data supplied to the OS, does not matter.

The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical/magneto-optical storage medium, or a nonvolatile semiconductor memory. As a program supply method, a method of storing a computer program forming the present invention in a server on a computer network, and a connected client computer downloading and programming the computer program is also conceivable.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

(Disclosure of this Specification)
The disclosure of this specification includes the following image processing apparatus, image processing method, generation method, and program.

(Item 1)
gradation compression means for compressing the gradation of the first image data;
processing means for applying a neural network for performing predetermined image processing to image data whose gradation has been compressed by said gradation compression means, thereby outputting image data on which said predetermined image processing has been performed;
a gradation decompression means for decompressing the gradation of the image data on which the predetermined image processing has been performed;
the number of bits representing pixel values inside the neural network is smaller than the number of bits representing pixel values of the first image data;
The image processing apparatus is disclosed, wherein the gradation compression means compresses the gradation using a characteristic that more gradation is assigned as the brightness is lower.

(Item 2)
further comprising acquisition means for acquiring the brightness of the first image data;
The gradation compression means compresses the first image data using a different characteristic among a plurality of characteristics in which more gradations are assigned as the brightness is lower, according to the brightness acquired by the acquisition means. The image processing device according to item 1 is disclosed, which compresses the gradation of .

(Item 3)
The processing means applies the neural network using different parameters among a plurality of pre-learned parameters of the neural network according to the brightness acquired by the acquisition means. An image processing device according to item 2 is disclosed.

(Item 4)
The gradation expansion means expands the gradation of the image data using a different one of a plurality of gradation expansion characteristics according to the brightness acquired by the acquisition means. An image processing device according to item 2 or 3 is disclosed.

(Item 5)
The acquiring means acquires the brightness of the first image data captured at a first time and the brightness of the second image data captured at a second time after the first time. death,
The gradation compression means selects a second characteristic adjacent to the first characteristic corresponding to the brightness of the first image data, among the plurality of characteristics that differ in stages corresponding to the brightness of the image data. The image processing apparatus according to item 2, characterized in that the gradation of the second image data is compressed using a .

(Item 6)
The processing means uses a parameter associated with the second characteristic among the plurality of neural network parameters associated with the plurality of characteristics that differ in stages corresponding to the brightness of the image data. The image processing device according to item 5 is disclosed, wherein the neural network is applied.

(Item 7)
The gradation expansion means expands the gradation of the image data using a characteristic corresponding to the second characteristic among a plurality of characteristics that differ in stages for expanding the gradation. An image processing device according to item 5 or 6 is disclosed.

(Item 8)
the acquiring means acquires brightness of a selected area from among a plurality of areas of the first image data;
The gradation compression means compresses the gradation of the first image data using a third characteristic corresponding to the brightness of the selected area among the plurality of characteristics. An image processing device according to item 2 is disclosed.

(Item 9)
The processing means uses a parameter corresponding to the third characteristic with respect to the brightness of the selected region, among the plurality of parameters of the neural network associated with the plurality of characteristics, to operate the neural network. The image processing device according to item 8 is disclosed, characterized in that:

(Item 10)
10. The image processing apparatus according to item 8 or 9, wherein the selected area is an area in which brightness of each area is lower than a predetermined threshold among the plurality of areas of the first image data. is disclosed.

(Item 11)
An item characterized in that the selected area is an area, among the plurality of areas of the first image data, whose difference in brightness from the same area in the same time zone on the previous day or earlier is equal to or less than a predetermined value. An image processing apparatus according to any one of 8 to 10 is disclosed.

(Item 12)
The gradation compression means compresses the gradation of the first image data using a different characteristic among a plurality of characteristics in which a larger number of gradations are assigned as brightness decreases, according to a predetermined setting. An image processing apparatus according to item 1 is disclosed, characterized by:

(Item 13)
The gradation compression means uses a characteristic that the larger the number of bits representing the pixel value of the image data to be processed, the more gradation is assigned to the predetermined low-luminance region. Disclosed is an image processing apparatus according to item 12, characterized.

(Item 14)
Item 12 or 13, wherein the processing means applies the neural network using different parameters among a plurality of pre-learned parameters of the neural network according to the predetermined setting. An image processing apparatus as described is disclosed.

(Item 15)
Items 12 to 14, wherein the gradation expansion means expands the gradation of the image data using a different one of a plurality of characteristics for expanding gradation in accordance with the predetermined setting. An image processing apparatus according to any one of the items is disclosed.

(Item 16)
The image processing apparatus according to any one of items 12 to 15 is disclosed, wherein the predetermined settings are settings for image data output from the image processing apparatus.

(Item 17)
The settings for the image data output from the image processing apparatus include the characteristics used for tone compression, the characteristics used for tone expansion, the number of tones of the output image data, and the pixel values of the output image data. 17. The image processing device according to item 16 is disclosed, characterized in that the number of bits representing .

(Item 18)
16. The image processing apparatus according to any one of items 12 to 15, wherein the predetermined settings are settings for image data to be input to the neural network.

(Item 19)
The settings for the image data input to the neural network include the upper limit of the pixel value of the image data input to the neural network, the number of gradations of the image data, and the number of bits representing the pixel value of the image data. 18. An image processing apparatus according to item 18, characterized by including any of

(Item 20)
20. The image processing apparatus according to any one of items 12 to 19, further comprising synthesizing means for synthesizing the image data decompressed by the tone decompression means and the first image data. disclosed.

(Item 21)
21. The image processing apparatus according to item 20, wherein the first image data includes image data clipped using a predetermined upper limit of pixel values.

(Item 22)
An image processing device for learning a neural network,
a gradation compression means for compressing the gradation of the image data of the training image and the image data of the correct image;
a processing means for outputting image data on which the predetermined image processing has been performed by applying the neural network for performing predetermined image processing to the image data obtained by compressing the gradation of the image data of the training image; ,
changing means for changing parameters of the neural network based on an error between the image data on which the predetermined image processing has been performed and the image data obtained by compressing the gradation of the image data of the correct image;
The number of bits representing pixel values inside the neural network is smaller than the number of bits representing pixel values of the image data of the training image;
The image processing apparatus is disclosed, wherein the gradation compression means compresses the gradation using a characteristic that more gradation is assigned as the brightness is lower.

(Item 23)
a gradation compression step of compressing the gradation of the first image data;
a processing step of applying a neural network that performs predetermined image processing to the image data whose gradation has been compressed in the gradation compression step, thereby outputting image data that has been subjected to the predetermined image processing;
a gradation decompression step of decompressing the gradation of the image data on which the predetermined image processing has been performed;
the number of bits representing pixel values inside the neural network is smaller than the number of bits representing pixel values of the first image data;
An image processing method is disclosed, wherein, in the gradation compression step, the gradation is compressed using a characteristic that more gradation is assigned as the brightness is lower.

(Item 24)
A method for generating a trained neural network in which each step is executed in an image processing device,
a gradation compression step of compressing the gradation of the image data of the training image and the image data of the correct image;
a processing step of applying the neural network for performing predetermined image processing to the image data obtained by compressing the gradation of the image data of the training image, thereby outputting image data on which the predetermined image processing has been performed; ,
a changing step of changing parameters of the neural network based on an error between the image data on which the predetermined image processing has been performed and the image data obtained by compressing the gradation of the image data of the correct image,
The number of bits representing pixel values inside the neural network is smaller than the number of bits representing pixel values of the image data of the training image;
A generation method is disclosed in which, in the gradation compression step, the gradation is compressed using the characteristic that more gradation is assigned as the brightness is lower.

(Item 25)
A program for causing a computer to function as each means of the image processing apparatus according to any one of items 1 to 21 is disclosed.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

DESCRIPTION OF SYMBOLS 100... Imaging device, 102... Imaging element, 104... Image processing apparatus, 105... ROM, 106... Processor, 107... RAM

Claims (25)

gradation compression means for compressing the gradation of the first image data;
processing means for applying a neural network for performing predetermined image processing to image data whose gradation has been compressed by said gradation compression means, thereby outputting image data on which said predetermined image processing has been performed;
a gradation decompression means for decompressing the gradation of the image data on which the predetermined image processing has been performed;
the number of bits representing pixel values inside the neural network is smaller than the number of bits representing pixel values of the first image data;
The image processing apparatus according to claim 1, wherein the gradation compression means compresses the gradation using a characteristic that more gradation is assigned as the brightness is lower. further comprising acquisition means for acquiring the brightness of the first image data;
The gradation compression means compresses the first image data using a different characteristic among a plurality of characteristics in which more gradations are assigned as the brightness is lower, according to the brightness acquired by the acquisition means. 2. The image processing apparatus according to claim 1, wherein the gradation of is compressed. The processing means applies the neural network using different parameters among a plurality of pre-learned parameters of the neural network according to the brightness acquired by the acquisition means. The image processing apparatus according to claim 2. The gradation expansion means expands the gradation of the image data using a different one of a plurality of gradation expansion characteristics according to the brightness acquired by the acquisition means. The image processing apparatus according to claim 2. The acquiring means acquires the brightness of the first image data captured at a first time and the brightness of the second image data captured at a second time after the first time. death,
The gradation compression means selects a second characteristic adjacent to the first characteristic corresponding to the brightness of the first image data, among the plurality of characteristics that differ in stages corresponding to the brightness of the image data. 3. The image processing apparatus according to claim 2, wherein the gradation of said second image data is compressed using a . The processing means uses a parameter associated with the second characteristic among the plurality of neural network parameters associated with the plurality of characteristics that differ in stages corresponding to the brightness of the image data. 6. The image processing apparatus according to claim 5, wherein said neural network is applied. The gradation expansion means expands the gradation of the image data using a characteristic corresponding to the second characteristic among a plurality of characteristics that differ in stages for expanding the gradation. The image processing apparatus according to claim 5. the acquiring means acquires brightness of a selected area from among a plurality of areas of the first image data;
The gradation compression means compresses the gradation of the first image data using a third characteristic corresponding to the brightness of the selected area among the plurality of characteristics. 3. The image processing apparatus according to claim 2. The processing means uses a parameter corresponding to the third characteristic with respect to the brightness of the selected region, among the plurality of parameters of the neural network associated with the plurality of characteristics, to operate the neural network. 9. The image processing apparatus according to claim 8, wherein: 9. The image processing apparatus according to claim 8, wherein the selected area is an area in which the brightness of each area is lower than a predetermined threshold among the plurality of areas of the first image data. wherein the selected area is an area, of the plurality of areas of the first image data, whose difference in brightness from the same area in the same time zone on the previous day or earlier is equal to or less than a predetermined value. Item 9. The image processing apparatus according to item 8. The gradation compression means compresses the gradation of the first image data using a different characteristic among a plurality of characteristics in which a larger number of gradations are assigned as brightness decreases, according to a predetermined setting. 2. The image processing apparatus according to claim 1, characterized by: The gradation compression means uses a characteristic that the larger the number of bits representing the pixel value of the image data to be processed, the more gradation is assigned to the predetermined low-luminance region. 13. The image processing device according to claim 12. 13. The method according to claim 12, wherein said processing means applies said neural network using different parameters among a plurality of pre-learned parameters of said neural network according to said predetermined setting. image processing device. 13. The method according to claim 12, wherein said gradation expansion means expands the gradation of image data using a different one of a plurality of characteristics for expanding gradation in accordance with said predetermined setting. image processing device. 13. The image processing apparatus according to claim 12, wherein said predetermined setting is for image data output from said image processing apparatus. The settings for the image data output from the image processing apparatus include the characteristics used for tone compression, the characteristics used for tone expansion, the number of tones of the output image data, and the pixel values of the output image data. 17. The image processing apparatus according to claim 16, wherein the number of bits representing 13. The image processing apparatus according to claim 12, wherein said predetermined setting is a setting for image data input to said neural network. The settings for the image data input to the neural network include the upper limit of the pixel value of the image data input to the neural network, the number of gradations of the image data, and the number of bits representing the pixel value of the image data. 19. The image processing apparatus according to claim 18, comprising any one of: 13. The image processing apparatus according to claim 12, further comprising synthesizing means for synthesizing the image data decompressed by said tone decompression means and said first image data. 21. The image processing apparatus according to claim 20, wherein said first image data includes image data clipped using a predetermined upper limit of pixel values. An image processing device for learning a neural network,
a gradation compression means for compressing the gradation of the image data of the training image and the image data of the correct image;
a processing means for outputting image data on which the predetermined image processing has been performed by applying the neural network for performing predetermined image processing to the image data obtained by compressing the gradation of the image data of the training image; ,
changing means for changing parameters of the neural network based on an error between the image data on which the predetermined image processing has been performed and the image data obtained by compressing the gradation of the image data of the correct image;
The number of bits representing pixel values inside the neural network is smaller than the number of bits representing pixel values of the image data of the training image;
The image processing apparatus according to claim 1, wherein the gradation compression means compresses the gradation using a characteristic that more gradation is assigned as the brightness is lower. a gradation compression step of compressing the gradation of the first image data;
a processing step of applying a neural network that performs predetermined image processing to the image data whose gradation has been compressed in the gradation compression step, thereby outputting image data that has been subjected to the predetermined image processing;
a gradation decompression step of decompressing the gradation of the image data on which the predetermined image processing has been performed;
the number of bits representing pixel values inside the neural network is smaller than the number of bits representing pixel values of the first image data;
The image processing method, wherein in the gradation compression step, the gradation is compressed using a characteristic that more gradation is assigned as the brightness is lower. A method for generating a trained neural network in which each step is executed in an image processing device,
a gradation compression step of compressing the gradation of the image data of the training image and the image data of the correct image;
a processing step of applying the neural network for performing predetermined image processing to the image data obtained by compressing the gradation of the image data of the training image, thereby outputting image data on which the predetermined image processing has been performed; ,
a changing step of changing parameters of the neural network based on an error between the image data on which the predetermined image processing has been performed and the image data obtained by compressing the gradation of the image data of the correct image,
The number of bits representing pixel values inside the neural network is smaller than the number of bits representing pixel values of the image data of the training image;
The generating method, wherein in the gradation compression step, the gradation is compressed using a characteristic that more gradation is assigned as the brightness is lower. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 21.

Images