JP2021082212A - Image processing system, image processing method, and program - Google Patents

Abstract

To reduce noises on images while suppressing a decrease of sharpness.SOLUTION: A noise-reduced image with reduced noises is generated from an input image. A high frequency emphasis image in which the high-frequency component of the input image is emphasized is generated. On each pixel of high-frequency emphasis image, a series of processing of limiting the pixel value to a range according to the brightness of a subject corresponding to the pixel is performed. An output image is generated based on the processed high-frequency emphasis image and noise-reduced image.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing apparatus, an image processing method, and a program, and particularly to noise reduction processing for an image.

The digital image obtained by imaging with an imaging device or the like contains noise. Noise reduction processing for images is known to reduce such noise. In Patent Document 1, a patch set is generated from a captured image, noise reduction processing is performed on all patches belonging to the patch set, and noise reduction processing is performed to synthesize patches to reduce noise in the captured image. The method of doing so is disclosed.

Japanese Unexamined Patent Publication No. 2013-0266669

In the conventional noise reduction processing, high frequency components may be lost in the processing process. Therefore, the user receives the impression that the sharpness of the image is reduced by the noise reduction processing.

An object of the present invention is to reduce image noise while suppressing a decrease in sharpness.

In order to achieve the object of the present invention, for example, the image processing apparatus of the present invention has the following configurations. That is,
A noise reduction means that generates a noise reduction image with noise reduced from the input image,
An extraction means for generating a high-frequency emphasized image in which the high-frequency component of the input image is emphasized, and
A correction means for limiting the pixel value of each pixel of the high-frequency emphasized image to a range corresponding to the brightness of the subject corresponding to the pixel.
A generation means for generating an output image based on the high-frequency enhanced image after processing by the correction means and the noise reduction image, and
To be equipped.

Image noise can be reduced while suppressing a decrease in sharpness.

The figure which shows the hardware configuration example of the image processing apparatus which concerns on one Embodiment. The figure which shows the functional structure example of the image processing apparatus which concerns on one Embodiment. The figure which shows the functional structure example of the image processing apparatus which concerns on one Embodiment. The flowchart of the image processing method which concerns on one Embodiment. The flowchart of the image processing method which concerns on one Embodiment. The figure which shows the functional structure example of the image processing apparatus which concerns on one Embodiment. The figure explaining an example of the smoothing process. The flowchart of the image processing method which concerns on one Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

[Embodiment 1]
The image processing apparatus according to the first embodiment corrects the image obtained by performing noise reduction processing on the input image by using the high frequency enhanced image in which the high frequency component of the input image is emphasized. In the first embodiment, in particular, for each pixel of the high-frequency emphasized image, a process of limiting the pixel value to a range corresponding to the brightness of the subject corresponding to the pixel is performed. With such a configuration, it is possible to reduce image noise while suppressing a decrease in sharpness.

The image processing apparatus according to the present embodiment can be realized by a computer including a processor and a memory. FIG. 1 is a block diagram showing a hardware configuration example of the image processing apparatus according to the first embodiment. The image processing device shown in FIG. 1 includes a CPU 101, a RAM 102, a storage unit 103, a general-purpose I / F (interface) 104, and a display unit 108. The configurations are connected to each other via the main bus 109. Further, the image processing device 105, the input device 106, and the external memory 107 are connected to the image processing device via the general-purpose I / F 110.

The CPU 101 controls each configuration according to the input image and a program that realizes the processing described later. The storage unit 103 is a storage medium such as an HDD or SSD. The storage unit 103 stores a computer program for the CPU 101 to execute each process. The RAM 102 functions as a buffer memory for temporarily holding the input image data or the like, a work area of the CPU 101, or the like. A processor such as the CPU 101 interprets and executes a program stored in a memory such as the RAM 102 or the storage unit 103, and performs an operation based on an instruction to realize the functions of each unit shown in FIG. 2 and the like described later. can do.

The image pickup device 105 is a device that captures an image of a camera or the like, and the obtained captured image can be input to an information processing device. The input device 106 is a device for receiving instructions or data, for example, a device such as a mouse or a keyboard used by a user to give an instruction to an image processing device. The external memory 107 is a storage medium for storing data, such as a memory card. The display unit 108 is hardware having a function of displaying information, such as a liquid crystal display or an organic EL display. The display unit 108 can display an image and can display a user interface (UI) used by the user to input a desired instruction. The image processing device according to the present embodiment can execute the processing described later on the image stored in the RAM 102 according to the instruction from the user input via the user interface and the input device 106. The image after the noise reduction processing obtained by the processing described later is stored in the RAM 102 again. The image after the noise reduction processing stored in the RAM 102 can be output to the display unit 108 or the external memory 107 in response to an instruction from the user.

FIG. 2 is a block diagram showing a logical configuration example of the image processing device according to the present embodiment. The image processing apparatus according to the present embodiment includes a noise reduction unit 210, an extraction unit 220, and an image generation unit 230.

The noise reduction unit 210 acquires the input image 200 and generates a noise reduction image in which noise is reduced from the input image 200. In the present embodiment, the input image 200 is an image in which a plurality of types of pixels representing different types of color information are arranged on one plane, and is specifically a Raw format image. A Raw format image is a one-plane image. The plane refers to the number of types of information that each pixel has. A one-plane image has one type of information in each pixel, and a three-plane image has three types of information in each pixel. For example, an RGB image is a 3-plane image and an RGBα image is a 4-plane image. In the present embodiment, the input image 200 is an image of the Bayer array, that is, an image obtained by using the image sensor of the Bayer array, but the input image 200 is not limited to the image of the Bayer array. Further, in the present embodiment, it is not essential that the input image 200 is a Raw format image.

The noise reduction processing method used by the noise reduction unit 210 for noise reduction is not particularly limited. For example, the method described in Patent Document 1 may be adopted. In the present embodiment, the noise reduction unit 210 performs noise reduction processing by using the following method. First, the noise reduction unit 210 sets a region consisting of a plurality of pixels in the input image 200 as a patch of interest, and generates a similar patch group similar to the patch of interest for each patch of interest. Next, the noise reduction unit 210 calculates the average value of each pixel of the similar patch group, and covariates by taking the sum of the products of any two pixel values of each similar patch group for all the similar patch groups. Calculate the matrix. The noise reduction unit 210 obtains an eigenvalue and an eigenvector from the average value and the covariance matrix. Further, the noise reduction unit 210 generates a basis matrix corresponding to the similar patch group based on the eigenvalues and the eigenvectors. The noise reduction unit 210 removes noise from each similar patch by performing projection processing on similar patches based on the average value of each pixel and the basis matrix, and obtains a noise removal patch group. Then, the noise reduction unit 210 generates a noise reduction image by synthesizing the noise removal patch group. Such a synthesis process is called aggregation or the like. Specifically, the noise reduction unit 210 returns each of the noise removal patch groups to the original patch position in the input image, and performs a weighted average based on averaging or similarity for pixels in which a plurality of patches overlap, thereby performing synthesis processing. It can be performed.

The extraction unit 220 generates a high-frequency emphasized image in which the high-frequency component of the input image 200 is emphasized. This high frequency enhanced image can contain edge components that are easily lost by noise reduction processing, and may also contain certain noise components. Therefore, the high-frequency emphasized image can be called an edge noise mixed image. In the present embodiment, the extraction unit 220 extracts a high frequency emphasized image based on the input image 200 and the noise reduction image.

As shown in FIG. 3A, the extraction unit 220 according to the present embodiment includes a difference image generation unit 222 and a difference image processing unit 223.

The difference image generation unit 222 generates a difference image between the input image 200 and the noise reduction image. The difference image generation unit 222 can obtain a difference image by subtracting the pixel value of the pixel of the difference image from the pixel value of the pixel of the input image 200 at the same pixel position. The difference image shows the difference between the input image 200 and the image obtained by performing noise reduction processing on the input image 200. Since the high frequency component (edge component and noise component) is attenuated by the noise reduction processing, the difference image shows the high frequency component (edge component and noise component) of the input image 200. The difference image thus obtained corresponds to a high-frequency enhanced image in which the high-frequency component of the input image 200 is emphasized. By using a high-frequency enhanced image containing such an edge component and correcting the image after the noise reduction processing as described later, the sharpness of the obtained output image can be improved.

The difference image processing unit 223 performs processing for limiting the pixel value of each pixel of the high-frequency emphasized image to a range corresponding to the brightness of the subject corresponding to the pixel. When imaging is performed using a sensor, noise depending on the amount of light is superimposed on the captured image. Therefore, in the bright part where the amount of noise is large, there is a possibility that noise having a large amplitude is mixed in the difference image. The difference image processing unit 223 has a function of coping with such noise generated depending on the brightness of the subject.

As shown in FIG. 3B, the difference image processing unit 223 includes a clip value calculation unit 223a and a clip processing unit 223b. The clip value calculation unit 223a calculates the clip value ClipValue according to the brightness of the subject. As described above, the amount of noise is considered to depend on the amount of light incident on the sensor, that is, the brightness of the subject. Further, the brightness of the subject can be obtained by referring to the noise reduction image or the input image 200. In the present embodiment, the clip value calculation unit 223a applies the equations (1) and (2) to the pixel values I of the corresponding pixels of the noise reduction image to obtain the clip value ClipValue for each pixel applied to the difference image. Obtainable.
σ = √ (K × (I-I0))… (1)
ClipValue = R × σ… (2)

In the formula (1), K and I0 represent the noise characteristics of the sensor that captured the input image 200. The parameters representing these noise characteristics can be estimated in advance by performing work (processing) such as photographing and analyzing a chart for noise evaluation. σ represents the noise standard deviation for each pixel. R is a parameter for adjusting the clip value and can be determined in advance.

The clip processing unit 223b performs a process of limiting the pixel value of each pixel of the high-frequency emphasized image to a range corresponding to the brightness of the subject corresponding to the pixel. The specific method for limiting the pixel value is not particularly limited, but for example, the clip processing unit 223b has a pixel value that is out of the range corresponding to the brightness of the subject corresponding to the pixel for each pixel of the high-frequency emphasized image. Can be changed to an upper limit value or a lower limit value that defines a range. In the present embodiment, the clip processing unit 223b obtains a post-clip image dc by clipping the difference image based on the clip value ClipValue obtained by the clip value calculation unit 223a. The post-clip image thus obtained also corresponds to a high-frequency enhanced image in which the high-frequency component of the input image 200 is emphasized. In the present embodiment, the post-clip image dc thus obtained is input to the image generation unit 230 as a high-frequency component image. The clip processing unit 223b can perform processing according to the equation (3). In the formula (3), d represents the pixel value of the difference image.

In this way, the clip processing unit 223b performs non-linear processing that limits the pixel value of the clipped image within the range from −ClipValue to +ClipValue, which is determined according to the brightness of the subject. With such a configuration, it is possible to suppress noise having a large amplitude exceeding this range, and to suppress the occurrence of point-like noise (salt-and-pep noise) after adding the high-frequency emphasized image to the noise reduction image. In particular, in the present embodiment, noise in a dark region can be effectively suppressed by reducing this range as the subject becomes darker.

The image generation unit 230 generates an output image based on the high-frequency enhanced image and the noise reduction image processed by the difference image processing unit 223. As shown in FIG. 3C, the image generation unit 230 includes a blend processing unit 232. Further, the image generation unit 230 may include a demosaic processing unit 231.

The blend processing unit 232 corrects the noise reduction image by using the high frequency component image. In one embodiment, the blend processing unit 232 can generate a blend image which is a corrected noise reduction image by adding the high frequency component image and the noise reduction image.

On the other hand, in the present embodiment, the blend processing unit 232 multiplies the high-frequency component image by a gain value according to the brightness of the subject, and then adds the image to the noise-reduced image. In the present embodiment, as shown in FIG. 3D, the blend processing unit 232 includes a gain calculation unit 232a, a gain multiplication unit 232b, and an addition processing unit 232c.

The gain calculation unit 232a determines the brightness of the subject corresponding to each pixel. The brightness of the subject can be obtained by referring to the noise reduction image or the input image 200. The gain calculation unit 232a further calculates the gain value based on the brightness of the subject corresponding to each of the determined pixels. In the present embodiment, the gain calculation unit 232a acquires the brightness value of the corresponding pixel of the noise reduction image corresponding to each pixel as the brightness of the subject, and based on this value, obtains the gain value for each pixel. Can be calculated. This gain value is determined for each pixel so that the greater the brightness of the subject, the smaller the gain value, and the smaller the brightness, the larger the gain value. For example, this gain value may be preset for each pixel value of the noise reduction image.

The gain multiplication unit 232b corrects the high frequency component image by using the gain value calculated by the gain calculation unit 232a. Specifically, the gain multiplication unit 232b can multiply the pixel value of each pixel of the high-frequency component image by the gain value corresponding to this pixel. That is, in the present embodiment, a larger gain is applied to a darker subject portion of the high frequency component image.

The gain multiplication unit 232b can also use the gain set for each of the other pixels. For example, the gain value may be a predetermined coefficient, a coefficient depending on the image height, or a coefficient depending on the pixel value.

The addition processing unit 232c generates a blended image by adding the high frequency component image corrected by the gain multiplication unit 232b to the noise reduction image. In a natural image, an image of a dark subject tends to have low contrast. Noise reduction processing for images with low contrast is not easy, and if sufficient noise reduction processing is performed, the sharpness of the image tends to be significantly reduced. As in the present embodiment, by adding the high frequency component image corrected by using the gain according to the brightness of the subject to the noise reduction image, the high frequency component of the darker subject is strongly recovered and the sharpness is improved. be able to.

By the above processing, the sharpness of the noise-reduced image can be improved. The image generation unit 230 may output the blended image obtained by the blending processing unit 232 correcting the noise reduction image as an output image. In this case, the demosaic processing unit 231 is unnecessary. On the other hand, for example, when the input image is an image in Raw format, the demosaic processing unit 231 may perform demosaic processing on the blended image. In this case, the image generation unit 230 can output the demosaic image after the demosaic processing as an output image.

The demosaic processing unit 231 performs demosaic processing on the blended image which is the noise reduction image corrected by the blending processing unit 232. The demosaic process is a process of converting an image having one plane into a demosaic image having a plurality of planes each representing one type of color information. For example, the demosaic processing unit 231 can convert a Raw format image into a YUV image, an L * a * b * image, an RGB image, a YMCK image, or the like.

Hereinafter, a specific processing example will be described when the demosaic processing unit 231 converts the RAW format blended image into the YUV format. As described above, since the input image 200 is a Bayer image, the blended image is also a Bayer image. The demosaic processing unit 231 determines the color of the pixel of interest other than the color of the pixel of interest by linear interpolation using eight peripheral pixels. For example, when the pixel of interest is R (red), it is necessary to complement G (green) and B (blue). When performing G complementation, linear interpolation is performed using the four G pixels above, below, left, and right of the pixel of interest. When performing B interpolation, linear interpolation is performed using four B pixels of diagonally upper right, diagonally lower right, diagonally upper left, and diagonally lower left. As a result of such linear interpolation, RGB pixel values are obtained for the pixel of interest, that is, an RGB image is generated. The demosaic processing unit 231 converts the RGB image thus obtained into a YUV image. This conversion can be performed by multiplying the pixel value of each pixel by a coefficient according to the equations (4) to (6).
Y = 0.2126 × R + 0.7152 × G + 0.0722 × B …… (4)
U = −0.1145272 × R − 0.385428 × G + 0.5 × B …… (5)
V = 0.5 × R − 0.454153 × G − 0.045547 × B …… (6)

Hereinafter, the flow of the image processing method according to the present embodiment will be described with reference to the flowcharts of FIGS. 4 and 5. In step S510, the noise reduction unit 210 reads the input image 200. In step S520, the noise reduction unit 210 obtains a noise reduction image by reducing the noise of the input image 200 as described above.

In step S530, the extraction unit 220 generates a high-frequency emphasized image in which the high-frequency component of the input image 200 is emphasized as described above. FIG. 5A shows a detailed processing flow of step S530. In step S531, the difference image generation unit 222 obtains a difference image based on the input image 200 and the noise reduction image as described above. In step S532, the clip value calculation unit 223a calculates the clip value based on the noise reduction image as described above. In step S533, the clip processing unit 223b obtains a high-frequency component image by performing clip processing based on the clip value on the difference image as described above.

In step S540, the image generation unit 230 obtains an output image from the noise reduction image and the high frequency component image. FIG. 5B shows a detailed processing flow of step S540. In step S541, the gain calculation unit 232a calculates the gain value based on the noise reduction image as described above. In step S542, the gain multiplication unit 232b multiplies the high frequency component image by the gain value as described above. In step S543, the addition processing unit 232c obtains a blended image by adding the noise reduction image and the high frequency component image multiplied by the gain value as described above. In step S544, the demosaic processing unit 231 obtains an output image by performing demosaic processing on the blended image.

As described above, in the present embodiment, the noise reduction image is corrected by using the high frequency component image in which the ratio of the edge components is relatively improved by the clip processing, thereby reducing the noise while maintaining the sharpness of the input image. can do. Further, even when noise having a standard deviation different for each pixel is superimposed, good image quality is realized by performing clipping based on the expected amount of noise.

In the present embodiment, the difference image processing unit 223 performs processing for limiting the pixel value of each pixel of the high-frequency emphasized image to a range corresponding to the brightness of the subject corresponding to the pixel, thereby limiting the brightness of the subject. Addressed the noise that depends on. Further, the blend processing unit 232 coped with the decrease in sharpness of the image according to the brightness of the subject by multiplying the high-frequency component image by the gain value according to the brightness of the subject. On the other hand, it is not essential to adopt both of these. It is also possible to adopt another method of correcting each pixel of the high-frequency emphasized image according to the brightness of the subject corresponding to the pixel. In this case, non-linear processing can be adopted so as to improve the sharpness or aesthetics of the output image.

[Embodiment 2]
In the second embodiment, the high frequency emphasized image is blended with the demosaic image obtained by demosacing the Raw format image. The high frequency component may be lost due to the complementation performed in the demosaic processing, but due to such a configuration, the high frequency contained in the output image is compared with the case where the demosaic processing is performed on the image to which the high frequency component is added. The amount of ingredients can be increased. Further, in the second embodiment, a process of increasing the ratio of the edge component of the high-frequency emphasized image is performed. When most of the noise can be removed by the noise reduction processing, the edge component of the high-frequency emphasized image is relatively small compared to the noise, but by increasing the ratio of the edge component of the high-frequency emphasized image, the output image is sharp. The rate of improvement in sex can be increased. However, it is not essential to adopt both of these.

The configuration of the image processing apparatus according to the second embodiment is similar to that of the first embodiment, but the processing and configuration of the extraction unit 220 and the image generation unit 230 are different. In the following, the differences from the first embodiment will be mainly described, and the description of the parts common to the first embodiment may be omitted.

In the present embodiment, as shown in FIG. 6A, the extraction unit 220 includes a smoothing processing unit 221 in addition to the difference image generation unit 222 and the difference image processing unit 223. The smoothing processing unit 221 performs smoothing processing on the noise-reduced image. For example, the smoothing processing unit 221 can perform smoothing of a noise-reduced image by performing filter processing. Further, the smoothing processing unit 221 may perform smoothing processing using another method such as a two-dimensional Fourier transform-high frequency mask-two-dimensional inverse Fourier transform. Here, the smoothing processing unit 221 may adaptively attenuate the edge component or the texture component by performing a non-linear processing that attenuates the high frequency component by utilizing a range filter, a bilateral filter, or the like.

A specific processing example of the smoothing processing unit 221 will be described below. The smoothing processing unit 221 sets the pixel of interest (i, j) in the noise reduction image, and the pixel of the same color as the pixel of interest and the pixel value of the pixel of interest in the range of 5 × 5 pixels centered on the pixel of interest. Based on the above, a weighted average value A (i, j) is obtained according to the equation (1). In the formula (7), I (i, j) indicates the pixel value of the pixel (i, j). 7 (A), (B), and (C) show the pixel of interest and the pixel of interest of the noise reduction image, which is a Bayer image, for each of the cases where the pixel of interest is an R pixel, a G pixel, and a B pixel. The positional relationship between the pixels of the same color and the pixels of the same color is shown. By repeating this process while sequentially setting all the pixels of the noise reduction image as the pixel of interest, a smoothed image having the weighted average value A (i, j) in the pixels (i, j) can be obtained.

The difference image generation unit 222 performs the same processing as in the first embodiment except that the difference image of the input image 200 and the noise reduction image after the smoothing process is generated. The difference image processing unit 223 also performs the same processing as in the first embodiment. By performing the smoothing process on the noise reduction image in this way, at least a part of the edge component is reduced from the image after the noise reduction process, so that the edge component of the input image 200 of the difference image is increased. Can be done. Therefore, the ratio of the edge component in the high-frequency component image is improved, and the sharpness of the output image can be improved.

In the present embodiment, as shown in FIG. 6B, the image generation unit 230 includes a demosaic processing unit 231 and a mix processing unit 233. The demosaic processing unit 231 performs the same processing as in the first embodiment except that the noise reduction image is demosaic processed.

The mix processing unit 233 generates an output image by correcting the demosaic image obtained by the demosaic processing by the demosaic processing unit 231 using the high frequency emphasized image. Here, the mix processing unit 233 can correct each plane of the demosaic image by a different method. For example, the mix processing unit 233 can synthesize a high-frequency emphasized image with each plane of the demosaic image at a mixing ratio set for each plane. Here, the mixing ratio for some planes can be set to 0. That is, the mix processing unit 233 can selectively add the high-frequency-enhanced image to a part of the planes selected from the two or more planes of the demosaic image, and the high-frequency-enhanced image to the remaining planes. Does not have to be added.

In the present embodiment, the mix processing unit 233 generates an output image by adding the high frequency component image only to the Y plane of the image after demosaication. Here, the U plane and the V plane are not corrected. By adding the high-frequency emphasized image only to the Y plane in this way, it is possible to maintain good sharpness while reducing noise. Considering human visual characteristics, the adverse effect of the presence of noise components on aesthetics is small in the Y plane that represents luminance information, but large in the UV plane that represents color difference information. Further, the positive effect of the sharpness of the signal (the large edge component) on the aesthetics is large in the Y plane and small in the UV plane. Therefore, by adding the high-frequency emphasized image containing the edge component and the noise component only to the Y plane, it is possible to increase the positive influence of the edge component on the image while suppressing the adverse effect of the noise component on the image.

By the way, due to the difference in sensitivity of each of the RGB pixels, the high-frequency component image may have a checkered light-dark pattern that follows the Bayer arrangement. Therefore, the mix processing unit 233 can correct each pixel of the high-frequency enhanced image such as the high-frequency component image according to the type of pixel, and then correct the image after demosaic using the high-frequency enhanced image. .. As described above, in the present embodiment, the input image is a Raw format image. Each pixel of a Raw format image is classified into a plurality of types of pixels (for example, R, G, B), and the pixel values of the pixels of different types are image pixels having different spectral sensitivity characteristics (for example, R, G, B). Obtained by B). Therefore, the mix processing unit 233 can correct the high-frequency component image by multiplying the pixel value of each pixel of the high-frequency component image by the gain value set for each pixel. In the present embodiment, different gain values are used for each of the R pixel, the G pixel, and the B pixel. For example, as the gain value, the white balance coefficient for each of the R pixel, the G pixel, and the B pixel can be used. By correcting the high-frequency component image in this way, it is possible to suppress the light-dark pattern that follows the Bayer array. Further, the mix processing unit 233 sets both the gain value calculated in the same manner as the gain calculation unit 232a of the first embodiment and the gain value set for each pixel as the pixel value of each pixel of the high-frequency component image. May be multiplied by.

Hereinafter, the flow of the image processing method according to the present embodiment will be described with reference to the flowcharts of FIGS. 4, 5 and 8. The processing of steps S510, S520, S532, and S533 is the same as that of the first embodiment. In step S531, the smoothing processing unit 221 performs smoothing processing on the noise reduction image as described above, and the difference image generation unit 222 subtracts the noise reduction image after the smoothing processing from the input image 200 to make a difference. Generate an image. In step S540, the image generation unit 230 obtains an output image from the noise reduction image and the high frequency component image. FIG. 8 shows a detailed processing flow of step S540 in this embodiment. In step S546, the demosaic processing unit 231 obtains a demosaiced image by demosaic processing on the noise reduction image as described above. In step S547, the mix processing unit 233 obtains an output image by correcting the Y plane of the image after demosaication using the high frequency component image as described above. By correcting the demosaic image using the high-frequency component image in this way, the sharpness of the output image can be improved.

(Other Examples)
In the above-described embodiment, each function of the image processing apparatus shown in FIG. 2 and the like is realized by a computer. However, some or all of the functions of the image processing apparatus may be realized by dedicated hardware or an image processing circuit. Further, the image processing device according to the embodiment of the present invention may be composed of, for example, a plurality of information processing devices connected via a network.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

210: Noise reduction unit, 220: Extraction unit, 222: Difference image generation unit, 223: Difference image processing unit, 223a: Clip value calculation unit, 223b: Clip processing unit, 230: Image generation unit

Claims (14)

A noise reduction means that generates a noise reduction image with noise reduced from the input image,
An extraction means for generating a high-frequency emphasized image in which the high-frequency component of the input image is emphasized, and
A correction means for limiting the pixel value of each pixel of the high-frequency emphasized image to a range corresponding to the brightness of the subject corresponding to the pixel.
A generation means for generating an output image based on the high-frequency enhanced image after processing by the correction means and the noise reduction image, and
An image processing device comprising. The correction means is characterized in that, for each pixel of the high-frequency enhanced image, a pixel value that is out of the range corresponding to the brightness of the subject corresponding to the pixel is changed to an upper limit value or a lower limit value that defines the range. The image processing apparatus according to claim 1. The correction means is characterized in that, for each pixel of the high-frequency enhanced image, a range corresponding to the brightness of the subject corresponding to the pixel is set based on the corresponding pixel value of the input image or the noise reduction image. The image processing apparatus according to claim 1 or 2. The generation means according to any one of claims 1 to 3, wherein a larger gain is applied to a pixel value of a pixel whose corresponding subject is darker in the high-frequency enhanced image. Image processing device. The image processing according to claim 4, wherein the generation means sets the gain for each pixel of the high-frequency enhanced image based on the corresponding pixel value of the input image or the noise reduction image. apparatus. The image processing apparatus according to any one of claims 1 to 5, wherein the extraction means generates the high-frequency enhanced image from a difference image between the input image and the noise reduction image. Any one of claims 1 to 5, wherein the extraction means generates the high-frequency enhanced image from a difference image between the input image and an image obtained by filtering the noise-reduced image. The image processing apparatus according to the section. The extraction means is characterized in that the high frequency enhanced image is generated from a difference image between the input image and an image obtained by a nonlinear process for reducing a high frequency component with respect to the noise reduction image, according to claim 1. The image processing apparatus according to any one of 5. The image processing apparatus according to any one of claims 6 to 8, wherein the extraction means generates the high-frequency enhanced image by extracting a high-frequency component of the difference image. The input image is an image in which a plurality of types of pixels representing different types of color information are arranged in one plane.
The generation means corrects the noise reduction image using the high-frequency enhanced image processed by the correction means, and performs demosaic processing on the corrected image, whereby a plurality of colors representing one type of color information are exhibited. The image processing apparatus according to any one of claims 1 to 9, wherein the output image having a plane is generated. The image processing apparatus according to any one of claims 1 to 10, wherein the input image is a Raw format image. The image processing apparatus according to any one of claims 1 to 11, wherein the input image is a Bayer image. The process of generating a noise-reduced image with reduced noise from the input image,
A step of generating a high frequency enhanced image in which the high frequency component of the input image is emphasized, and
A step of limiting the pixel value of each pixel of the high-frequency emphasized image to a range corresponding to the brightness of the subject corresponding to the pixel, and
A generation step of generating an output image based on the high-frequency emphasized image after the processing for limiting the pixel value and the noise reduction image, and
An image processing method characterized by having. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 12.

Images