JP2021090129A - Image processing device, imaging apparatus, image processing method and program - Google Patents

Abstract

To restore an image with good image quality from a coded image regardless of an imaging condition.SOLUTION: An image processing device includes determining means that determines an inference parameter and an acquiring means that acquires an image capturing condition in order to acquire an image in which the degradation due to encoding is restored by performing inference based on an inference parameter on a decoded image obtained by decoding an encoded image. The determining means determines the inference parameter according to the image capturing condition acquired by the acquiring means.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing device, an imaging device, an image processing method and a program.

Recent imaging devices can record imaging data (RAW images that have not been developed) immediately after being acquired by the imaging sensor. Since the RAW image is not developed, the color information is not lost and the RAW image is recorded while maintaining abundant gradations, so that the RAW image can be edited with a high degree of freedom. However, since the amount of data in a RAW image is enormous, there is a problem that the free area of the recording medium is squeezed. Therefore, it is desirable to perform compression coding on the RAW image and record it with a small amount of data.

On the other hand, deep learning technology using neural networks is applied in a wide range of fields. Deep learning technology is used in the field of image processing for the purpose of improving the image quality of images. Here, if the RAW image is encoded, the RAW image is deteriorated. Therefore, by applying the deep learning technique to the deteriorated RAW image, the deteriorated image can be restored. As a result, the image quality of the RAW image can be ensured, and it is possible to avoid pressure on the free space of the recording medium.

As a related technique, the technique of Patent Document 1 has been proposed. The technique of Patent Document 1 aims to improve the noise removal performance by adjusting at least one internal parameter of the intermediate layer of the neural network at the time of processing after learning.

2018-206382A

The technique of Patent Document 1 determines the method of adjusting the internal parameters only by the amount of noise, and does not consider imaging conditions other than noise. In general, in an imaging device, noise fluctuates depending on the imaging sensitivity, and brightness also fluctuates depending on the exposure. The depth of field also varies depending on the aperture and focal length. Here, it is assumed that a noise-removed image is acquired by a neural network to which an inference parameter learned is applied to an image in which noise is added to an image captured with an appropriate exposure as a learning image. Since the above inference parameter is a parameter optimized for the condition of proper exposure, it is difficult to secure high noise removal performance when the image to be inferred is underexposed or overexposed. Therefore, it is desirable that the inference parameter is not a learned parameter adjusted only by the amount of noise, but a learned parameter corresponding to various imaging conditions. This also applies to the restoration of image quality deterioration using a neural network.

An object of the present invention is to restore an encoded image to an image with good image quality regardless of the imaging conditions.

In order to achieve the above object, the image processing apparatus of the present invention acquires an image in which the deterioration due to the coding is restored by executing inference based on the inference parameters on the decoded image obtained by decoding the encoded image. In order to do so, the inference parameter is provided with a determination means for determining the inference parameter and an acquisition means for acquiring the imaging condition of the image, and the determination means has the inference parameter according to the imaging condition acquired by the acquisition means. Is characterized by determining.

According to the present invention, when restoring an encoded image, it is possible to restore an image with good image quality regardless of the imaging conditions.

It is a block diagram which shows the structural example of an image processing apparatus. It is a figure explaining the learning method of a neural network. It is a flowchart which shows the flow of the method of determining an inference parameter in 1st Embodiment. It is a flowchart which shows the flow of the method of determining an inference parameter in 2nd Embodiment. It is a flowchart which shows the flow of the method of determining an inference parameter in 3rd Embodiment. It is a flowchart which shows the flow of the method of determining an inference parameter in 4th Embodiment. It is a figure explaining the method of embedding PSNR in the metadata of the coded data. It is a flowchart which shows the flow of the method of determining an inference parameter in 5th Embodiment. It is a figure which shows an example of the file format of the coded data.

Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the respective embodiments.

<First Embodiment>
FIG. 1 is a block diagram showing a configuration example of the image processing device 100. The image processing device 100 includes a RAW decoding unit 101, a metadata acquisition unit 102, a deterioration restoration processing unit 103, and an inference parameter determination unit 104. The image processing device 100 will be described as being built in an imaging device (hereinafter, a camera) having an imaging unit (imaging means), but the image processing device 100 is an arbitrary device (for example, an information processing device, a decoding device). ) May be applied. The image processing device 100 may have a processor and a memory. In this case, the processing of each embodiment may be realized by the processor executing the program stored in the memory. Further, the image processing device 100 may be realized by a predetermined programming circuit. The inference processing by the deterioration restoration processing unit 103, which will be described later, may be executed by, for example, the graphics processing unit.

In FIG. 1, the coded data, the decoded RAW image, and the deteriorated restored RAW image are stored in the storage units 105A, 105B, and 105C, respectively. The storage unit 105A, the storage unit 105B, and the storage unit 105C may be an integrated storage unit or may be separate storage units. Further, each storage unit may be realized by, for example, the above-mentioned memory, a buffer, a predetermined storage device, or the like.

The RAW decoding unit 101 generates a decoded RAW image by acquiring and decoding the encoded data. The coded data is, for example, data obtained by encoding a RAW image (captured image) acquired by an imaging device. The RAW decoding unit 101 outputs the decoded RAW image to the deterioration restoration processing unit 103. The decoded RAW image corresponds to the decoded image. The decoded RAW image generated by the RAW decoding unit 101 is a RAW image in which deterioration due to coding has occurred, and the image quality is deteriorated.

The metadata acquisition unit 102 is a first acquisition means that acquires metadata (meta information) from the encoded data and outputs the acquired metadata to the inference parameter determination unit 104. The metadata acquisition unit 102 may acquire information about the imaging unit that captured the image. In this case, the metadata acquisition unit 102 also functions as a second acquisition means. FIG. 9 is a diagram showing an example of a file format of coded data. As shown in FIG. 9, the coded data is composed of a header unit 901, a metadata unit 902, and a payload unit 903. The header portion 901 includes an identification code or the like indicating that the coded data is RAW format data. The metadata unit 902 includes parameters representing imaging conditions (shooting information) such as an exposure compensation value, an ISO sensitivity, an aperture value, and a focal length. The metadata unit 902 may further include imaging conditions such as the number of AF focus points at the time of imaging and information unique to the camera used for imaging such as sensor information. The payload unit 903 contains compressed data of the encoded image.

The deterioration restoration processing unit 103 shown in FIG. 1 is composed of a neural network. The deterioration restoration processing unit 103 functions as an inference means that performs inference processing for the purpose of image restoration on the decoded RAW image using the inference parameters output by the inference parameter determination unit 104. The deterioration restoration processing unit 103 generates an deterioration restoration RAW image in which the deterioration generated when the RAW image is encoded is restored by performing inference processing using a neural network on the decoded RAW image.

It is assumed that the neural network of each embodiment is a CNN (Convolutional Neural Network). CNN is a neural network having a convolutional layer and a pooling layer, and a fully connected layer is connected to the output side. The inference parameters of each embodiment correspond to the weights and biases of each edge connecting the nodes of each layer in the fully connected layer, and correspond to the weights and biases of the kernel (filter) in the convolution layer. Hereinafter, the parameters updated by the learning (machine learning) of the neural network are collectively referred to as inference parameters. The inference parameter is a learned inference parameter.

The inference parameter determination unit 104 is a determination means for determining the inference parameter to be used in the deterioration restoration processing unit 103 based on the metadata (meta information) acquired by the metadata acquisition unit 102. The inference parameter determination unit 104 has a storage area 104A. Inference parameters are stored in the storage area 104A for each imaging condition.

Next, the learning method of the neural network will be described. Depending on the imaging conditions of the camera, even if the subject is the same, the quality of the image changes greatly depending on the brightness, the amount of noise, the occurrence of blur, and the like. Therefore, even if an image is input to a neural network to which the same inference parameters are applied, it is difficult to always obtain a constant image restoration effect. For example, it is assumed that the inference parameters are generated by learning using a learning image captured under a single condition of proper exposure. In this case, if the image to be inferred is an image acquired under the same imaging conditions as at the time of learning, a high image restoration effect can be obtained at proper exposure. On the other hand, an image captured with overexposure (an image captured with an exposure higher than the proper exposure) has a brighter subject than an image with a proper exposure, and an image captured with an underexposure (an image captured with an exposure smaller than the proper exposure) The subject is darker than the image with proper exposure. Therefore, since the properties of the image change significantly from the viewpoint of brightness, the image restoration effect is reduced when the same inference parameters are applied.

Here, an approach is conceivable in which learning images captured under the above three imaging conditions are mixed and learned to generate inference parameters. In this case, the image restoration effect on the proper exposure is lower than that in the case of generating the inference parameter using the learning image captured under the single condition of the proper exposure described above. Therefore, in order to maximize the image restoration effect, a plurality of inference parameters according to the imaging conditions are generated, and each generated inference parameter is applied according to the imaging conditions when the RAW image to be restored is imaged. It is necessary.

FIG. 2 is a diagram illustrating a learning method of a neural network. In this embodiment, an exposure compensation value is applied as an imaging condition. Then, the inference parameters for the exposure compensation value are learned separately for proper exposure, underexposure, and overexposure. The RAW coding unit 201 acquires an uncompressed RAW image, compresses and encodes it, and generates coded data. The RAW coding unit 201 outputs the generated coded data to the RAW decoding unit 101. Here, as shown in FIG. 2, an uncompressed RAW image is stored in the storage unit 105D. The storage unit 105D will be described as being integrally configured with the storage units 105A, 105B and 105C, but may be provided separately. The storage unit 105D stores an uncompressed RAW image (uncompressed RAW image).

The image quality comparison unit 202 calculates MSE (Mean Squared Error) by comparing the decoded RAW image whose image quality has deteriorated with the uncompressed RAW image whose image quality has not deteriorated. The image quality comparison unit 202 outputs the calculated MSE to the inference parameter update unit 203. MSE is an index value indicating the performance of the neural network, and the smaller the value of MSE, the higher the reproducibility of the decoded RAW image to the uncompressed RAW image. The inference parameter update unit 203 updates the inference parameters according to the MSE output by the image quality comparison unit 202. The inference parameter update unit 203 has a storage area 203A for storing inference parameters for each imaging condition. The inference parameter update unit 203 individually updates each inference parameter stored in the storage area 203A according to the imaging conditions acquired by the metadata acquisition unit 102. The inference parameter update unit 203 outputs the updated inference parameter to the deterioration restoration processing unit 103 according to the imaging conditions. The deterioration restoration processing unit 103 performs restoration processing of the code-degraded composite RAW image by using the inference parameters updated every time the learning is performed. The inference parameter update unit 203 updates the inference parameters by using a stochastic gradient descent method or the like so that the MSE becomes smaller each time learning is performed.

The inference parameter update unit 203 functions as a learning means for individually repeating learning for each of the three imaging conditions of proper exposure, underexposure, and overexposure. As a result, the inference parameters for each imaging condition stored in the storage area 203A are optimized. In the above example, the case where MSE is applied to compare the image quality of the decoded RAW image and the uncompressed RAW image has been described, but any method other than MSE may be applied to the comparison of the two images. For example, a method of evaluating the loss function or the like may be applied. The index value in the image quality comparison may be an index indicating image quality, and is not limited to MSE. Further, as the method for updating the inference parameters, any method for autonomously adjusting the error can be applied. For example, an error backpropagation method or the like may be applied to update the inference parameters.

Next, a method of determining the inference parameter performed by the inference parameter determination unit 104 will be described. FIG. 3 is a flowchart showing the flow of the method of determining the inference parameter in the first embodiment. The inference parameter determination unit 104 acquires the exposure compensation value acquired from the coded data by the metadata acquisition unit 102 (S301). The inference parameter determination unit 104 determines whether the acquired exposure compensation value is smaller than "0" (S302). If the exposure compensation value is smaller than "0", it is determined as Yes in S302. In this case, the inference parameter determination unit 104 determines the inference parameter used in the restoration process performed by the deterioration restoration processing unit 103 as the learned inference parameter for underexposure (first inference parameter) (S303).

When the exposure compensation value is "0" or more, it is determined as No in S302. In this case, the inference parameter determination unit 104 determines whether the exposure compensation value is “0” (S304). When the exposure compensation value is "0", it is determined as Yes in S304. In this case, since the exposure compensation value is "0", the inference parameter determination unit 104 uses the inference parameters used in the restoration process performed by the deterioration restoration processing unit 103 as the learned proper exposure inference parameters (second inference parameter). ) (S305). If the exposure compensation value is not "0", it is determined as No in S304. In this case, since the exposure compensation value is "0" or more, the inference parameter used in the restoration process performed by the deterioration restoration processing unit 103 is determined as the learned inference parameter for overexposure (third inference parameter) (S306). ). The learned inference parameters for overexposure correspond to the third inference parameter. The inference parameter determination unit 104 outputs the inference parameter determined by any one of S303, S305 or S306 to the deterioration restoration processing unit 103 (S307).

Therefore, the inference parameter determination unit 104 learns and generates three inference parameters, an inference parameter for underexposure, an inference parameter for proper exposure, and an inference parameter for overexposure. Then, each inference parameter learned is stored in the storage area 203A. The inference parameter determination unit 104 determines the inference parameter to be used for the restoration process from the three inference parameters according to the exposure compensation value (metadata of the coded data) acquired by the metadata acquisition unit 102. The inference parameter determination unit 104 applies the inference parameter determined according to the exposure compensation value to the neural network of the deterioration restoration processing unit 103. As a result, the restoration process can be performed using the neural network to which the appropriate inference parameters are applied according to the imaging conditions indicated by the metadata of the coded data. Therefore, even if the brightness at the time of imaging is different, it is possible to restore the deterioration due to coding with high accuracy. That is, when restoring the encoded image, it is possible to restore the image with good image quality regardless of the imaging conditions (brightness).

In the first embodiment, an example in which three inference parameters, an inference parameter for underexposure, an inference parameter for proper exposure, and an inference parameter for overexposure are generated by learning has been described. That is, learning of the above three inference parameters is performed, and restoration processing is performed using a neural network to which any one of the inference parameters is applied. The number of inference parameters is not limited to three, and may be two or four or more. Further, although the deterioration restoration processing unit 103 has described an example in which the neural network is configured, the deterioration restoration processing unit 103 may be an arbitrary learning model using inference parameters learned in advance for each imaging condition. .. Further, the deterioration restoration processing unit 103 may perform not only the image quality restoration processing of the RAW image whose image quality has deteriorated due to compression coding, but also, for example, super-resolution processing on the RAW image whose image quality has deteriorated by reducing the image size. .. These points are the same in each of the following embodiments.

Here, in the present embodiment, the inference parameter update unit 203 determines the inference parameter applied to the neural network of the deterioration restoration processing unit 103. This eliminates the need to manually adjust the inference parameters of the neural network (designer). Further, the designer does not need to grasp the operation contents of each node of the neural network. In particular, as the configuration of a neural network becomes more sophisticated and complicated, the number of nodes to be grasped inevitably increases. In the present embodiment, even if the configuration of the neural network becomes sophisticated and complicated, the inference parameter update unit 203 applies the inference parameters to the neural network, so that the workload by the manual (designer) does not increase. ..

<Second Embodiment>
Next, the second embodiment will be described. The image processing device 100 of the second embodiment is an example of restoring coding deterioration in consideration of noise caused by imaging conditions. Generally, in a RAW image captured with high sensitivity, a noise component (high frequency component) is amplified as well as a pixel value, so that the spatial frequency is increased. The spatial frequency distribution differs greatly between high-sensitivity images and low-sensitivity images. Therefore, even if a high-sensitivity image and a low-sensitivity image are input to a neural network to which the same inference parameters are applied, it is difficult to obtain a high restoration effect. In order to improve the image restoration effect, it is desirable to perform individual learning using learning images according to the imaging conditions that affect noise and optimize the inference parameters. The image processing apparatus of the second embodiment restores the coding deterioration by applying inference parameters optimized according to the ISO value representing the imaging sensitivity.

Since the configuration of the image processing device 100 and the learning method of the neural network of the second embodiment are the same as those of the first embodiment, the description thereof will be omitted. The image processing apparatus 100 of the second embodiment uses an ISO value as an imaging condition. In the second embodiment, learning is performed individually under two conditions of high sensitivity having an ISO value higher than a predetermined threshold value and low sensitivity having an ISO value equal to or lower than a predetermined threshold value, and inference parameters are generated. Hereinafter, a predetermined threshold value in each embodiment can be arbitrarily set.

Here, the relationship between the ISO value representing the imaging sensitivity and noise will be described. As for the ISO value, the noise component is amplified as the value increases, but the generation of noise decreases as the sensor size increases and as the number of pixels decreases. This is because as the distance between the pixels of the image sensor (hereinafter, referred to as pixel pitch) becomes wider, the mutual interference of electric signals decreases, and as a result, the possibility of noise generation decreases. For example, even if the ISO value is such that noise is not noticeable in a full-size sensor, noise may be noticeable in a type 1 sensor. This is because the pixel pitch of the 1-inch sensor is narrow and the sensitivity that can be used regularly is relatively low compared to the full-size sensor. Therefore, when the sensor size of the learning image and the sensor size of the RAW image to be restored are different, it is necessary to scale the ISO threshold value for determining the magnitude of the noise amount according to the pixel pitch.

FIG. 4 is a flowchart showing the flow of the method of determining the inference parameter in the second embodiment. In the second embodiment, the learning image used when learning is performed is an image taken in full size, and the RAW image to be restored is an image taken by a type 1 sensor. The inference parameter determination unit 104 acquires the ISO value and the sensor information acquired from the coded data by the metadata acquisition unit 102 (S401). It is assumed that the sensor information includes the sensor size and the number of sensor pixels. The inference parameter determination unit 104 calculates the pixel pitch of the RAW image to be restored (S402). The pixel pitch is represented by the square root of the value obtained by dividing the sensor size by the number of sensor pixels. For example, when the number of sensor pixels is 12 million pixels, the sensor size of the 1-inch sensor is "13.2 [mm] x 8.8 [mm]", so the pixel pitch is 3.11 [μm]. Become. If the metadata contains information about the pixel pitch, the processing of S402 can be omitted.

The inference parameter determination unit 104 updates the threshold value T based on the pixel pitch calculated in S402 (S403). The threshold value T is the threshold value of the ISO value used for determining the magnitude of the noise amount in S404 described later, and is the normal sensitivity when the full-size sensor used at the time of learning is used. The inference parameter determination unit 104 updates the threshold value T according to the sensor of the RAW image to be restored. Assuming that the threshold value before the update is T1 and the threshold value after the update is T2, the relationship between T2 and T1 is expressed by the following equation.
"T2 = T1 x (pixel pitch of RAW image to be restored / pixel pitch of learning image) ² "

The pixel pitch of the learning image in the second embodiment is an image captured by using a full-size sensor. Assuming that the number of sensor pixels is 12 million pixels, the sensor size is "36 [mm] x 24 [mm]", so that the pixel pitch of the learning image is 8.49 [μm]. Since the pixel pitch of the RAW image to be restored is 3.11 [μm], the updated threshold value T2 is T1 × 0.13. In this way, the normal sensitivity of the full-size sensor is scaled to the normal sensitivity of the type 1 sensor, and the threshold value T is updated.

The inference parameter determination unit 104 compares the ISO value acquired in S401 with the threshold value T updated in S403, and determines whether the ISO value is smaller than the threshold value T (predetermined value) (S404). As a result of comparison, when the ISO value is smaller than the threshold value T, it is determined as Yes in S404. In this case, the inference parameter determination unit 104 determines the inference parameter used in the restoration process performed by the deterioration restoration processing unit 103 as the learned low-sensitivity inference parameter (fourth inference parameter) (S405). When the ISO value is equal to or greater than the threshold value T (greater than or equal to a predetermined value), S404 determines No. In this case, the inference parameter determination unit 104 determines the inference parameter used in the restoration process performed by the deterioration restoration processing unit 103 as the high-sensitivity inference parameter (fifth inference parameter) (S406). Then, the inference parameter determination unit 104 outputs the inference parameter determined in either S405 or S406 to the deterioration restoration processing unit 103 (S407).

As described above, in the second embodiment, two inference parameters, a low-sensitivity inference parameter and a high-sensitivity inference parameter, are generated by learning. Each learned inference parameter is stored in the storage area 203A. The inference parameter determination unit 104 selects and determines the inference parameter to be used for the restoration process from the two inference parameters according to the ISO value acquired by the metadata acquisition unit 102. The inference parameter determination unit 104 applies the inference parameter determined according to the ISO value to the neural network of the deterioration restoration processing unit 103. As a result, the restoration process can be performed using the neural network to which the appropriate inference parameters are applied according to the ISO value indicated by the metadata of the coded data. Therefore, even if the amount of noise at the time of imaging is different, it is possible to restore the deterioration due to coding with high accuracy. That is, when restoring the encoded image, it is possible to restore the image with good image quality regardless of the imaging conditions (ISO sensitivity). The number of inference parameters applied to the deterioration restoration processing unit 103 may be three or more. The second embodiment may be applied alone or together with the first embodiment.

<Third Embodiment>
Next, the third embodiment will be described. The image processing apparatus 100 of the third embodiment restores the coding deterioration in consideration of the blur generated outside the range of the depth of field. Generally, when the distance to the subject is the same, the depth of field becomes shallower when the focal length is long and the aperture value is set small, so that blurring is likely to occur. Even if the subject is the same, the spatial frequency distribution is different between the image in which blurring occurs and the image in which blurring does not occur. Therefore, it is difficult to obtain a high restoration effect even if the image is input to a neural network to which the same inference parameters as the blurred image and the non-blurred image are applied. In order to improve the image restoration effect, it is desirable to individually learn using the learning image according to the imaging conditions that affect the blur and optimize the inference parameters. The image processing apparatus 100 of the third embodiment applies inference parameters optimized based on the aperture value, the focal length, and the number of AF focal points to the neural network to restore the coding deterioration.

Since the configuration of the image processing device 100 and the learning method of the neural network of the third embodiment are the same as those of the first embodiment and the second embodiment, the description thereof will be omitted. The image processing apparatus 100 of the third embodiment calculates the blur evaluation value based on the aperture value and the focal length. Then, the image processing device 100 individually learns based on the calculated blur evaluation value and the number of AF focal points under two conditions, that is, a large blur area and a small blur area, and generates an inference parameter.

Here, the blur evaluation value will be described. The blur evaluation value is an evaluation value indicating whether or not the imaging condition is such that blur is likely to occur. The larger the blur evaluation value, the more likely the imaging condition is to cause blurring, and the smaller the blurring evaluation value, the less likely the blurring is to occur. The blur evaluation value can be calculated by "B = D / F", where B is the blur evaluation value, D is the focal length, and F is the aperture value. This is a simple calculation formula that utilizes the property that the longer the focal length and the smaller the aperture value, the more likely it is that blurring will occur.

However, the above-mentioned blur evaluation value does not accurately calculate the area where blur occurs. Even if the conditions that are likely to cause blurring are set from the aperture value and focal length, if the entire screen is in focus, the inference parameter for the large blurring area will be applied to the neural network. There is a possibility that the restoration effect of the will be reduced. Therefore, assuming actual shooting, it is desirable to estimate the accurate blurred area using other imaging conditions. Therefore, in the third embodiment, the AF in-focus number is used. The AF in-focus number represents the number of points in focus on the subject when the image is taken with autofocus. When the number of in-focus points is larger than a predetermined threshold value, it is considered that the in-focus area is large and the area where blurring occurs is small. On the other hand, when the number of focal points is equal to or less than a predetermined threshold value, it is considered that the focusing area is narrow and the blurring area is large. Therefore, by using not only the blur evaluation value but also the number of AF focal points included in the imaging conditions, it is possible to estimate the area where the blur occurs more accurately.

FIG. 5 is a flowchart showing the flow of the method of determining the inference parameter in the third embodiment. The inference parameter determination unit 104 acquires the aperture value, focal length, and AF focal length acquired from the encoded data from the metadata acquisition unit 102 (S501). The inference parameter determination unit 104 calculates the blur evaluation value based on the aperture value and the focal length acquired in S501 (S502). The inference parameter determination unit 104 compares the blur evaluation value calculated in S502 with the predetermined threshold value T1 and determines whether the blur evaluation value is smaller than the predetermined threshold value T1 (S).
It is determined whether it is smaller (S503).

As a result of the comparison, when the blur evaluation value is smaller than the predetermined threshold value T1, it is determined as Yes in S503. In this case, the inference parameter determination unit 104 determines the inference parameter used in the restoration process performed by the deterioration restoration processing unit 103 as the learned defocused area small inference parameter. The learned inference parameter for reducing the blurred area corresponds to the sixth inference parameter used when the area where the blur occurs is smaller than the predetermined area. When the blur evaluation value is equal to or higher than the threshold value T1, it is determined as No in S503. In this case, the inference parameter determination unit 104 compares the number of AF focus points with the predetermined threshold value T2, and determines whether the number of AF focus points is larger than the predetermined threshold value T2 (S505).

As a result of comparison, when the number of AF focal points is larger than the predetermined threshold value T2, it is determined as Yes in S505. In this case, since the blurring area is considered to be small, the flow shifts to S504. On the other hand, when the number of AF focus points is equal to or less than the predetermined threshold value T2, it is determined as No in S505. In this case, the area where blurring occurs is considered to be large. Therefore, the inference parameter determination unit 104 determines the inference parameter used in the restoration process performed by the deterioration restoration processing unit 103 as the learned blurred area large-use inference parameter. The learned inference parameter for increasing the blurred area corresponds to the seventh inference parameter used when the area in which the blur occurs is equal to or larger than a predetermined area.

Therefore, in the third embodiment, two inference parameters, an inference parameter for increasing the area where blurring occurs and an inference parameter for reducing the area where blurring occurs, are generated. The inference parameter determination unit 104 determines the inference parameters used for the restoration process of the coding deterioration according to the aperture value, the focal length, and the number of AF focal points, and applies the determined inference parameters to the neural network. This makes it possible to restore the coding deterioration with high accuracy even when the blurring area is different. That is, when restoring the encoded image, it is possible to restore the image with good image quality regardless of the imaging conditions (blurring area).

The number of inference parameters applied to the deterioration restoration processing unit 103 may be three or more. The third embodiment may be applied alone or together with the first and second embodiments. Further, the third embodiment may be applied together with either the first embodiment and the second embodiment.

<Fourth Embodiment>
Next, the fourth embodiment will be described. The image processing apparatus 100 of the fourth embodiment restores the coding deterioration in consideration of the combination of a plurality of imaging conditions. Here, if the inference parameters are generated so as to cover all the combinations of the imaging conditions of the camera, the learning time for generating each inference parameter becomes very long. The image processing device 100 of the fourth embodiment does not perform learning for each combination of all imaging conditions of the camera, but learns by narrowing down the combination of imaging conditions to generate inference parameters. As a result, it is possible to restore the coding deterioration with high accuracy. The imaging conditions used in the fourth embodiment are an exposure compensation value, an ISO value, an aperture value, and a focal length. These imaging conditions are the imaging conditions described in the first embodiment, the second embodiment, and the third embodiment. Since the image processing device 100 of the fourth embodiment has the same configuration as the image processing device 100 of the first embodiment, the second embodiment, and the third embodiment, the description thereof will be omitted. Further, since the learning method of the neural network is the same, the description thereof will be omitted.

As described above, since there are a wide variety of combinations of imaging conditions, it takes a huge amount of learning time to generate inference parameters so as to cover all combinations of imaging conditions. However, some combinations of imaging conditions are redundant. Here, it is assumed that the conditions are high sensitivity and the area where blurring occurs is small. Random noise is applied to the entire screen of an image captured with high sensitivity. In this case, most of the information on the screen is buried in random noise, so that there is almost no image information related to blurring.

That is, the imaging condition of high sensitivity has a relatively higher influence on the image quality than the imaging condition of small blurring area. Therefore, it is not necessary to separately generate inference parameters under the condition of high sensitivity and small blurring area. In this case, the high-sensitivity inference parameters of the second embodiment may be generated. Further, the imaging condition with a small blurring area has a low influence on the image quality and does not extremely change the spatial frequency in the screen. Therefore, it is not necessary to generate an inference parameter for reducing the blurring area of the third embodiment. As described above, by excluding the imaging conditions having a low influence on the image quality (imaging conditions having an influence on the image quality lower than a predetermined degree), it is not necessary to generate unnecessary inference parameters.

The image processing apparatus 100 of the fourth embodiment generates inference parameters in consideration of the degree of influence of each imaging condition on the image quality. Since the imaging condition having a high influence on the image quality has a high degree of difficulty in restoration, the image processing device 100 generates an optimized inference parameter using the learning image of the imaging condition having a high influence on the image quality. On the other hand, since the difficulty of restoration is low under the imaging conditions having a low influence on the image quality, the image processing device 100 is versatile by using a learning image in which a plurality of other imaging conditions having a low influence on the image quality are mixed. Inference parameters (general-purpose inference parameters) are generated. The generated inference parameters are three types of inference parameters, an inference parameter for underexposure, an inference parameter for high sensitivity, and an inference parameter for large blurring area, and a general-purpose inference parameter. The general-purpose inference parameter is an inference parameter generated by using an image that does not satisfy any of the conditions of underexposure, high sensitivity, and large blurring area as a learning image.

All of the above three imaging conditions of underexposure, high sensitivity, and large blurring area have different degrees of influence on image quality. High-sensitivity imaging conditions affect the entire screen and increase brightness and spatial frequency. Underexposure imaging conditions affect the entire screen and reduce brightness. Imaging conditions with a large area of blurring affect the local area in the screen and reduce the spatial frequency. This is because it is unlikely that blurring will occur in the entire screen unless a mistake occurs in focusing at the time of imaging. As described above, from the viewpoints of brightness, spatial frequency, and affected area, the degree of influence of each imaging condition on the image quality increases in the order of high sensitivity, underexposure, and large blurred area. Therefore, in the fourth embodiment, when the inference parameter is determined, it is preferentially selected in order from the imaging condition having the highest degree of influence on the image quality.

FIG. 6 is a flowchart showing the flow of the method of determining the inference parameter in the fourth embodiment. The description of the parts that overlap with the first to third embodiments will be omitted. The inference parameter determination unit 104 acquires the exposure compensation value, ISO value, aperture value, focal length, and AF focus number acquired from the coded data by the metadata acquisition unit 102 (S601). The inference parameter determination unit 104 calculates the sensor pixel pitch and the blur evaluation value (S602). The calculation of the sensor pixel pitch corresponds to S402 of the second embodiment, and the calculation of the blur evaluation value corresponds to S502 of the third embodiment. The inference parameter determination unit 104 updates the threshold value T0 (S603). The threshold value T0 corresponds to the threshold value T of the second embodiment. Further, the processing of S603 corresponds to S403 of the second embodiment.

The inference parameter determination unit 104 determines whether the ISO value of the RAW image to be restored is smaller than the threshold value T0 updated in S603 (S604). S604 corresponds to S404 of the second embodiment. When the ISO value is equal to or higher than the threshold value T0, it is determined as No in S604. In this case, the inference parameter determination unit 104 determines the inference parameter used in the restoration process performed by the deterioration restoration processing unit 103 as the high-sensitivity inference parameter (S605). S605 corresponds to S406 of the second embodiment. If the ISO value is smaller than the threshold value T0, S604 determines Yes. In this case, the inference parameter determination unit 104 determines whether the exposure compensation value acquired in S601 is smaller than "0" (S606). The process of S606 corresponds to S302 of the first embodiment.

If the exposure compensation value is smaller than "0", it is determined as Yes in S606. In this case, the inference parameter determination unit 104 determines the inference parameter used in the restoration process performed by the deterioration restoration processing unit 103 as the underexposure inference parameter (S607). S607 corresponds to S303 of the first embodiment. When the exposure compensation value is "0" or more, it is determined as No in S606. In this case, the inference parameter determination unit 104 calculates the blur evaluation value based on the aperture value and the focal length acquired in S601 (S608). The inference parameter determination unit 104 determines whether the blur evaluation value calculated in S608 is smaller than the predetermined threshold value T1 (S609). S609 corresponds to S503 of the third embodiment. When the blur evaluation value is equal to or higher than the threshold value T1, it is determined as No in S610. In this case, the inference parameter determination unit 104 determines whether the number of AF focal points acquired in S601 is greater than the predetermined threshold value T2 (S610). S610 corresponds to S505 of the third embodiment.

When the number of AF focus points is equal to or greater than the predetermined threshold value T2, No is determined in S610. In this case, the inference parameter determination unit 104 determines the inference parameter used in the restoration process performed by the deterioration restoration processing unit 103 as the defocused area large-use inference parameter (S611). S611 corresponds to S506 of the third embodiment. When the blur evaluation value is smaller than the predetermined threshold value T1 and the number of AF focal points is larger than the predetermined threshold value T2, the inference parameter determination unit 104 uses the inference parameter used in the restoration process performed by the deterioration restoration processing unit 103 as a general-purpose inference parameter. (S612). The inference parameter determination unit 104 outputs the inference parameter determined in S605, S607, S611 or S612 (S613).

As described above, in the fourth embodiment, the inference parameter determination unit 104 determines that the inference parameter optimized for each imaging condition is applied to the neural network under the imaging condition having a high influence on the image quality. On the other hand, for imaging conditions that have a low effect on image quality, it is decided to apply general-purpose inference parameters to the neural network. Then, the inference parameter determination unit 104 applies the corresponding inference parameters to the neural network in order from the imaging conditions having a high influence on the image quality (imaging conditions having a high influence on the image quality of a predetermined degree or more). decide. As a result, deterioration of the image restoration effect can be suppressed, and it is not necessary to perform learning for each combination of imaging conditions to generate inference parameters. Therefore, it is possible to significantly reduce the learning time of the inference parameters. In the fourth embodiment, imaging conditions other than the four imaging conditions of exposure compensation value, ISO value, aperture value, and focal length may be added.

<Fifth Embodiment>
In the first to fourth embodiments described above, the coding deterioration is restored by applying the inference parameters according to the imaging conditions indicated by the metadata included in the coded data. In the fifth embodiment, the PSNR (Peak Signal to Noise Ratio) of the RAW image to be restored is newly recorded as the metadata of the encoded data. Then, the inference parameters optimized according to the PSNR are applied to restore the coding deterioration. PSNR will be described. In the coding field, an index value called PSNR is used as a measure of image reproducibility. The PSNR indicates that the larger the value, the higher the image reproducibility, and the smaller the value, the lower the image reproducibility. That is, it is highly possible that the image quality of an image having a low PSNR is significantly deteriorated due to data loss in the coding process. Therefore, in order to enhance the image restoration effect, it is desirable to optimize the inference parameters by individually learning using the learning image according to the PSNR indicating the degree of image quality deterioration.

Since the configuration of the image processing device 100 and the learning method of the neural network of the fifth embodiment are the same as those of the first to fourth embodiments, the description thereof will be omitted. The image processing apparatus 100 of the fifth embodiment uses PSNR as the imaging information, and when the PSNR is lower than a predetermined threshold value, the image restoration is performed by applying the inference parameters generated by using the learning image having a low PSNR. Do. Further, when the PSNR is equal to or higher than a predetermined threshold value, it is considered that the image reproducibility is high and the image quality is less deteriorated. In this case, the image processing apparatus 100 of the fifth embodiment does not perform image restoration using inference parameters. FIG. 7 is a diagram illustrating a method of embedding PSNR in the metadata of the coded data.

Each part shown in FIG. 7 is included in the image processing apparatus 100. The storage unit 704A stores an uncompressed RAW image. The storage unit 704B stores the coded data. The storage unit 704C stores the decoded RAW image. Each storage unit shown in FIG. 7 may be integrally configured with each storage unit described in the first embodiment, or may be provided separately. The RAW coding unit 701 compresses and encodes an uncompressed RAW image to generate coded data. The RAW coding unit 701 corresponds to the RAW coding unit 201 of the first embodiment. The RAW decoding unit 702 decodes the encoded data to generate a decoded RAW image. The RAW decoding unit 702 corresponds to the RAW decoding unit 101 of the first embodiment. The PSNR calculation unit 703 acquires the decoded RAW image and the uncompressed RAW image and calculates the PSNR. Assuming that the maximum pixel value that the current image can take is MAX and the maximum square error between the evaluation target image and the current image is MSE, the PSNR is represented by the following "Equation 1".

算出されたＰＳＮＲは、符号化データのメタデータ部９０２に記録される。

The calculated PSNR is recorded in the metadata unit 902 of the coded data.

FIG. 8 is a flowchart showing the flow of the method of determining the inference parameter in the fifth embodiment. The inference parameter determination unit 104 acquires the PSNR acquired from the coded data from the metadata acquisition unit 102 (S801). The inference parameter determination unit 104 determines whether the PSNR acquired in S801 is lower than the predetermined threshold value T3 (S802). If the PSNR is lower than the predetermined threshold value T3, S802 determines Yes. In this case, the inference parameter determination unit 104 determines the inference parameter used in the restoration process performed by the deterioration restoration processing unit 103 as the inference parameter for low PSNR (S803).

The low PSNR inference parameter is an inference parameter generated by individually learning using a learning image captured under the condition of low PSNR. When the PSNR is equal to or higher than the predetermined threshold value T3, it is determined as No in S802. In this case, since the PSNR value is high, the low PSNR inference parameter is not applied to the neural network of the deterioration restoration processing unit 103.

That is, the inference parameter determination unit 104 controls whether to apply the inference parameter to the neural network of the deterioration restoration processing unit 103. Thereby, even if the degree of coding deterioration is different, it is possible to enhance the restoration effect of the coding deterioration. The number of patterns of inference parameters for PSNR may be arbitrary. Moreover, although the example in which PSNR is used as an index value for quantifying the image quality has been described, other index values may be used.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and modifications can be made within the scope of the gist thereof. In the above-described embodiment, the deterioration / restoration processing unit 103 is included in the image processing device 100, but an external device may be used instead of the image processing device 100 to execute the processing of the deterioration / restoration processing unit 103. In that case, the multiple RAW image decoded by the RAW composite unit 101 and the inference parameter determined by the inference parameter determination unit 104 are sent to an external device such as a cloud server via the communication unit (not shown) of the image processing device 100. To send. Then, the external device may execute the processing of the deterioration restoration processing unit 103 from the compound RAW image and the inference parameter, and transmit the RAW image restored to the deterioration to the image processing device 100. Further, the image processing apparatus 100 of the present embodiment acquires encoded data and performs decoding processing by the RAW composite unit 101 to acquire a decoded RAW image. However, the image processing device 100 does not have the RAW composite unit 101, and is from an external device. The multiple RAW image and the metadata of the image may be acquired, and the deterioration restoration processing unit 103 and the inference parameter determination unit 104 may perform the restoration processing to acquire the deterioration restoration RAW image.

The present invention supplies a program that realizes one or more functions of each of the above-described embodiments to a system or device via a network or storage medium, and one or more processors of the computer of the system or device transmits the program. It can also be realized by the process of reading and executing. The present invention can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Image processing device 101 RAW decoding unit 102 Metadata acquisition unit 103 Deterioration restoration processing unit 104 Inference parameter determination unit 201 RAW coding unit 202 Image quality comparison unit 203 Inference parameter update unit 703 PSNR calculation unit

Claims (20)

Determining means for determining the inference parameters in order to obtain an image in which the deterioration due to the encoding is restored by performing inference based on the inference parameters on the decoded image obtained by decoding the encoded image.
An acquisition means for acquiring the imaging conditions of the image is provided.
The determination means is an image processing apparatus characterized in that the inference parameters are determined according to the imaging conditions acquired by the acquisition means. The image processing apparatus according to claim 1, wherein the determination means determines the inference parameter based on the result of comparing the imaging condition with the threshold value. Further, it has a second acquisition means for acquiring information about the imaging means that captured the image.
The image processing apparatus according to claim 2, wherein the determination means determines the threshold value according to information about the imaging means. Any one of claims 1 to 3, further comprising an inference parameter determined by the determination means and an inference means for inferring an image in which deterioration due to coding is restored based on the decoded image. The image processing apparatus according to item 1. The image processing apparatus according to any one of claims 1 to 4, further comprising a compound means for decoding the encoded image. The imaging condition includes an exposure compensation value and includes an exposure compensation value.
The plurality of inference parameters include a first inference parameter for underexposure, a second inference parameter for proper exposure, and a third inference parameter for overexposure.
Claims 1 to 1, wherein the determination means determines any one of the first inference parameter, the second inference parameter, and the third inference parameter according to the exposure compensation value. The image processing apparatus according to any one of 5. The imaging conditions include ISO values.
The plurality of inference parameters include a fourth inference parameter used when the ISO value is smaller than the predetermined value and a fifth inference parameter used when the ISO value is greater than or equal to the predetermined value.
The determination means according to any one of claims 1 to 6, wherein one of the fourth inference parameter and the fifth inference parameter is determined according to the ISO value included in the imaging condition. The image processing apparatus according to any one item. The image processing apparatus according to claim 7, wherein the predetermined value is updated based on a pixel pitch calculated by the number of pixels of the image sensor and the size of the image sensor. The imaging conditions include the aperture value, the focal length, and the number of AF focal points.
The plurality of inference parameters include a sixth inference parameter used when the blurred area is smaller than a predetermined area and a seventh inference parameter used when the blurred area is equal to or larger than the predetermined area. Including
The determining means is characterized in that it determines one of the sixth inference parameter and the seventh inference parameter according to the aperture value, the focal length, and the number of AF focal points included in the imaging condition. The image processing apparatus according to any one of claims 1 to 8. The image processing apparatus according to any one of claims 1 to 9, wherein the determination means determines corresponding inference parameters in order from an imaging condition having a high degree of influence on image quality. The image processing apparatus according to claim 10, wherein inference parameters for imaging conditions whose degree of influence on image quality is lower than a predetermined degree are not individually generated. The image processing according to claim 11, wherein the inference parameter for an imaging condition whose degree of influence on the image quality is lower than the predetermined degree is learned using a learning image in which a plurality of imaging conditions are mixed. apparatus. The determining means includes an inference parameter for underexposure, an inference parameter used when the ISO value is a predetermined value or more, and an area where blurring occurs is a predetermined area or more according to the imaging condition. The image processing apparatus according to claim 12, wherein one of the inference parameters used and the inference parameters learned using the learning image in which the plurality of imaging conditions are mixed is determined. The determination means is in the order of inference parameters used when the ISO value is equal to or greater than a predetermined value, inference parameters for underexposure, and inference parameters used when the area where the blur occurs is equal to or greater than a predetermined area. The image processing apparatus according to claim 13, wherein the inference parameters are preferentially determined. The determination means according to any one of claims 1 to 14, wherein the determination means determines whether or not to apply the inference parameter to the learning model according to an index value for quantifying the image quality. Image processing device. The inference is made by a neural network
The image processing apparatus according to any one of claims 1 to 15, wherein the inference parameter is a weight and a bias obtained by learning the neural network. The image processing apparatus according to any one of claims 1 to 16, wherein the encoded image is a RAW image. Imaging unit and
The image processing apparatus according to any one of claims 1 to 17.
An imaging device characterized by comprising. A step of determining the inference parameters in order to obtain an image in which the deterioration due to the encoding is restored by performing inference based on the inference parameters on the decoded image obtained by decoding the encoded image.
It is provided with an acquisition step of acquiring the imaging conditions of the image.
An image processing method characterized in that the inference parameters are determined according to the acquired imaging conditions. A program for causing a computer to execute the image processing apparatus according to any one of claims 1 to 17.

Images