JP6891014B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing technique for reducing color noise.

High-quality captured images are desired for image pickup devices such as digital cameras, but in dark places or in environments where a sufficient S / N ratio cannot be obtained, such as at night, noise of color components, which is low-frequency random noise. Image quality is greatly degraded by (color noise). Particularly in recent years, there has been a high demand for high-sensitivity shooting, and it is desired that a high-quality shot image can be obtained with less noise even in a dark place or at night. Therefore, in order to suppress color noise, a smoothing process using an averaging filter, a Gaussian filter, or the like and a color noise reduction process using an order statistic filter such as a median filter are performed. However, when smoothing processing or an order statistic filter is used, in order to sufficiently reduce noise in a large range (low frequency noise), it is necessary to design a large number of taps on the filter, which causes an increase in the circuit scale. It ends up. In this regard, a method has been proposed in which the same effect can be obtained without increasing the number of taps by performing the filtering process after reducing the input image (see Patent Document 1). In the method of Patent Document 1, first, a plurality of low-resolution images whose resolutions are gradually lowered are generated from the input images, and color noise suppression processing is performed on each of the input images and the low-resolution images. After that, for low-resolution images, the resolution is returned and weighted composition is performed based on the composition ratio coefficient. The composition ratio coefficient at this time is determined so that the composition ratio of images from the lower layers is high when it is determined that the pixel of interest belongs to a region with little color change.

Japanese Unexamined Patent Publication No. 2012-104968

In the case of the method of Patent Document 1, color noise near the edge may remain. For example, in the high-contrast edge portion, since there are few pixels having a pixel value close to the pixel value of interest around the pixel of interest, color noise cannot be sufficiently reduced even if the number of taps of the filter is increased. Then, if tuning is performed so as to enhance the smoothing effect as much as possible for such a region near the edge, problems such as color loss and deterioration of color resolution (edge blurring) occur. Therefore, an object of the present invention is to sufficiently reduce color noise near edges while maintaining color sharpness.

The image processing apparatus according to the present invention is a reduction processing means for generating reduced image data of a plurality of layers having reduced resolution by performing reduction processing on the input image data, and attention in the input image data and the reduced image data. An image of another layer with respect to the correction means for correcting the pixel value of the pixel to a value with reduced color noise and the image data excluding the reduced image data having the lowest resolution among the corrected image data. The composite ratio derivation means for deriving the composite ratio with the data, and the color noise reduction degree by the correction for each pixel for the image data excluding the reduced image data having the lowest resolution among the corrected image data. Based on the color noise reduction degree calculation means calculated in 1 and the composite ratio and the color noise reduction degree, the image data of interest among the corrected image data and the image data of the next lower layer thereof. Synthetic means for synthesizing
Wherein the combining means includes enlargement processing means for enlarging the image data, and the color difference signal of the target pixel of the image data to be the target, the target by the enlargement processing means an image data of the one next lower layer The color similarity determination means for determining the color similarity of the image data enlarged to the same resolution as the image data to be performed with the color difference signal of the pixel of interest, and the image data of interest and the image data of the next lower layer thereof. When the color noise reduction degree is smaller than a predetermined threshold value and the color similarity determination means determines that the colors are similar to each other, the image data of the next lower layer is preferentially used. and determines, when determining to use the image data of preferentially said one of a hierarchy emption decision to change the synthesis ratio so as to increase the weight of the image data of the hierarchy under the one The means, the pixel value of the image data of interest, and the pixel value of the image data one layer below it and enlarged to the same resolution as the image data of interest by the enlargement processing means. A pixel value synthesizing means for synthesizing based on the synthesizing ratio,
To have a, characterized in that.

According to the present invention, it is possible to sufficiently reduce color noise near edges while maintaining color sharpness.

It is a figure which shows an example of the hardware composition of an image processing apparatus. (A) is a block diagram showing an example of a logical configuration related to noise reduction processing of an image processing apparatus, and (b) is a block diagram showing an internal configuration of a color noise reduction processing unit. It is a sequence diagram which showed the flow of the whole process realized by the logical structure shown in FIG. 2 (b). It is a figure which shows the specific example of the filter used for the low-pass filter processing. It is a figure which shows the specific example of how to apply a low-pass filter to a pixel of interest. It is a flowchart which shows the flow of each process in a color noise reduction process. It is a flowchart which shows the flow of the resolution reduction processing. It is a flowchart which shows the flow of the expansion synthesis processing which concerns on Example 1. FIG. It is a block diagram which shows the internal structure of the color noise reduction processing part in Example 2. FIG. It is a figure explaining the similarity determination of the hue using the formula (11). It is a flowchart which shows the flow of the expansion synthesis processing which concerns on Example 2.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The configuration shown in the following examples is only an example, and the present invention is not limited to the illustrated configuration.

FIG. 1 is a diagram showing an example of a hardware configuration of an image processing device according to this embodiment. The image processing device 100 is, for example, a PC or the like, and includes a CPU 101, a RAM 102, an HDD 103, a general-purpose interface (I / F) 104, a monitor 108, and a main bus 109. Then, the image pickup device 105 such as a camera, the input device 106 such as a mouse and a keyboard, and the external memory 107 such as a memory card are connected to the main bus 109 by the general-purpose I / F 104.

The CPU 101 realizes the following various processes by operating various software (computer programs) stored in the HDD 103. First, the CPU 101 starts an image processing application stored in the HDD 103, expands it into the RAM 102, and displays a user interface (UI) on the monitor 108. Subsequently, various data stored in the HDD 103 and the external memory 107, image data acquired by the image pickup device 105, user instructions from the input device 106, and the like are transferred to the RAM 102. Further, according to the processing in the image processing application, the data stored in the RAM 102 is arithmetically processed based on the command from the CPU 101. The result of the arithmetic processing is displayed on the monitor 108 or stored in the HDD 103 or the external memory 107. The image data stored in the HDD 103 or the external memory 107 may be transferred to the RAM 102. Further, the image data transmitted from the server via a network (not shown) may be transferred to the RAM 102.

In this embodiment, in the image processing apparatus 100 having the above configuration, an embodiment in which image data is input to an image processing application and image data with reduced color noise is output based on a command from the CPU 101 will be described. It shall be.

(Logical configuration of image processing device)
FIG. 2A is a block diagram showing an example of a logical configuration related to noise reduction processing of the image processing apparatus 100. The image processing device 100 includes a signal distribution unit 201, a color noise reduction processing unit 202, a luminance noise reduction processing unit 203, and a signal conversion processing unit 204.

The color signal (color signal represented in the RGB color space) of the color image data input from the image pickup apparatus 105, the HDD 103, or the external memory 107 is first input to the signal conversion processing unit 201. Here, the input color image data may be intermediate image data generated by the image pickup apparatus 105. For example, it may be RAW image data which is the original image data in which the image captured by the image pickup element is digitized, or RAW image data after the defect pixel correction which corrected the defect pixel. Further, it may be RAW image data after various correction processes such as false color suppression and low-pass filter processing have been performed. When inputting the color signal of the RAW image data, the RGB color signal may be generated by using a known demosaic technique. Further, in the case of Bayer array RAW image data, a method of treating 2 × 2 pixels (RGGB) as one pixel can be considered. The description will be made on the premise that the color image data in this embodiment is in an RGB color space, but the description is not limited to this, and the data may be converted into, for example, an L * a * b * color space.

The signal conversion processing unit 201 generates a luminance signal representing the luminance component (Y) and a luminance signal representing the color difference component (Cr and Cb) from the RGB color signal of the input color image data by a known conversion formula. Here, the spatial frequency of the luminance signal noise (hereinafter, luminance noise) and the spatial frequency of the color difference signal noise (hereinafter, color noise) are different from each other, and the spatial frequency of the color difference signal noise is lower. By generating a YCrCb color signal from the RGB color signal related to the input color image data by this processing, optimum noise reduction processing is performed for each of the brightness signal representing the brightness component and the color difference signal representing the color difference component. be able to. The input RGB color signal and the YCrCb color signal generated by the signal conversion process are sent to the color noise reduction processing unit 202 and the luminance noise reduction processing unit 203.

The color noise reduction processing unit 202 performs processing for reducing the noise of the color difference signal based on the RGB color signal and the YCrCb color signal. The details of the color noise reduction processing will be described later. The luminance noise reduction processing unit 203 performs processing for reducing the noise of the luminance signal. In the luminance noise reduction processing unit 203, general noise reduction processing may be used.

The signal integration processing unit 204 performs a process of integrating the color signal whose color noise is reduced by the color noise reduction processing unit 202 and the color signal whose brightness noise is reduced by the brightness noise reduction processing unit 203. The signal integration processing unit 204 outputs color image data in which both color noise and luminance noise are reduced, which is generated as a result of the signal integration processing. The color image data after integration is output to the monitor 108, HDD 103, or the like. In addition, for example, it may be output to an external memory 107 connected to a general-purpose I / F 104, an external server (not shown), a printer, or the like.

(Details of color noise reduction processing unit)
Subsequently, the color noise reduction processing unit 202 will be described in detail. FIG. 2B is a block diagram showing an internal configuration of the color noise reduction processing unit 202 according to this embodiment. As shown in FIG. 2B, the color noise reduction processing unit 202 includes a reduction processing unit 210, a correction unit 220, a composition ratio derivation unit 230, a color noise reduction degree calculation unit 240, and an expansion composition processing unit 250. To. Then, FIG. 3 is a sequence diagram showing the flow of the entire process realized by the logical configuration shown in FIG. 2 (b). Hereinafter, each part of the color noise reduction processing unit 202 will be described with reference to FIGS. 2B and 3.

<Reduction processing unit>
The reduction processing unit 210 performs a process of reducing the resolution of the input image data (reduction processes 302a to 302d in FIG. 3) to generate the reduced image data. In the example shown in FIG. 3, the resolution of the input image is reduced to 1/2 times, reduced to 1/4 times, reduced to 302b, reduced to 1/8 times, reduced to 302c, and 1/16 times, respectively. The reduction process 302d is performed to generate four types of reduced image data having different resolutions. The type of reduced image data is not limited to this example, and may be reduced to 1/8 times or further reduced to 1/32 times. Generally, when reducing image data, if low-pass filter processing is not performed, aliasing noise will occur, and the pattern of the aliasing noise will appear in the image data that is finally output. Therefore, when performing the reduction processing, for example, an algorithm including an average pixel method or other low-pass filtering processing is used, or a bilinear method or the like is applied after performing low-pass filtering processing in advance. The filter used for the low-pass filter processing is determined based on the reduction ratio. For example, when the reduction ratio is 1/2, the filter as shown in FIG. 4 is used. The size and coefficient of the filter are not limited to the filter shown in FIG. Hereinafter, the input image data and the plurality of reduced image data generated from the input image data will be collectively referred to as “multi-resolution image data”. The reduced image data generated by the reduction process is stored in the RAM 102 for use in the correction process described later.

<Correction part>
The correction unit 220 corrects the color signal of the pixel of interest so as to reduce color noise with respect to the above-mentioned multi-resolution image data based on preset correction parameters (correction processes 303a to 303e in FIG. 3). To do. Here, the correction parameter means a predetermined reference area for the pixel of interest and each threshold value for determining whether or not to average. It is desirable to change these to appropriate values according to the amount of noise. Therefore, for example, it is determined in advance according to the shooting sensitivity, or it is adaptively determined based on the information of the pixel of interest and its peripheral pixels. In the correction process, the value of the color signal (here, the RGB signal) of the pixel of interest of the input color image data is corrected for each color component using, for example, the following equations (1) to (3).

In the above equations (1) to (3), R [i], G [i], and B [i] represent pixels satisfying the following equation (4) within a predetermined reference region for the pixel of interest. That is, for the pixel of interest, the smoothing process is performed using only the pixels that satisfy the condition of the following equation (4) in the predetermined reference region. A predetermined reference area includes a pixel of interest and an adjacent pixel surrounding the pixel of interest, and may be, for example, a 3 × 3 pixel area or a larger 10 × 10 pixel area. The method of determining the reference region will be described in the description of the color noise reduction degree calculation unit 240. Further, M is the number of pixels satisfying the following equation (4) in the reference region for the pixel of interest. Since the pixel of interest itself is always included, M ≧ 1.

In the above formula (4), ∧ represents the logical product (the AND) (hereinafter, the same), Th _r represents the threshold for the R signal, Th _g is a threshold for the G signal, Th _b is a threshold value for B signals, respectively .. Then, R _diff , G _diff , and B _diff represent the difference between each pixel value in the reference region and the pixel value to be compared, and are represented by the following equations (5) to (7), respectively.

_{As the pixel values R comp} , G _comp , and B _comp to be compared in the above equations (5) to (7), the pixel values of the pixel of interest may be used as they are. Alternatively, the result of applying a low-pass filter to the pixel value of the pixel of interest may be used as the pixel value of the pixel of interest. FIG. 5 is a diagram showing a specific example of how to apply a low-pass filter to a pixel of interest. In the example of FIG. 5, the pixel value of interest and the four pixels adjacent to each other in the vertical and horizontal directions thereof are used, and a low-pass filter is applied to the pixel value of the pixel of interest as _{"comparing pixel values R comp} , G _comp , and B _comp". The result is derived. If the effect is similar to that of the low-pass filter, the weighting and the number of adjacent pixels to be used may be freely determined.

The correction process shown here is an example, and any other method may be used as long as it is a method for correcting the color signal value so as to reduce the color noise. The multi-resolution image data whose color noise has been reduced by the correction processing is stored in the RAM 102 for use in the composition ratio derivation processing and the color noise reduction degree calculation processing described later.

<Composite ratio derivation unit>
The composite ratio derivation unit 230 derives a composite ratio with the lower layer image data for each image data included in the multi-resolution image data whose color noise has been reduced by the correction process (composite ratio derivation process 304a in FIG. 3). ~ 304d) is performed. However, as is clear from the sequence diagram of FIG. 3, the image data of the lowest layer (reduced image data reduced to 1/16 times in this embodiment) does not have image data lower than itself. It is out of the scope of this process. Here, the composition ratio is determined according to the degree of abrupt change in the color signal. In the case of this embodiment, the composition ratio can be derived by performing edge detection using the RGB signal of the pixel of interest as an input. Specifically, for example, among the results of applying the filter in the horizontal direction and the vertical direction using a spatial filter having a filter coefficient [-1,2,-1], the one considered to have a high degree of color edge is selected. It is mapped to the coefficient K between 0 and 1 by a predetermined function. A coefficient K having a value closer to 1 is output as a filter application result in which the degree of color edge is considered to be higher, and a coefficient K having a value closer to 0 is output as a filter application result in which the degree of color edge is considered to be lower. Then, this coefficient K becomes the synthesis ratio.

A luminance signal or a color difference signal may be used for edge detection. Further, the filter coefficient and the determination method are not limited to those described above. Any method may be used as long as a large value composition ratio can be obtained when the degree of color edge is considered to be high and a small value composition ratio can be obtained when the degree of color edge is considered to be low. The derived synthesis ratio data is stored in the RAM 102 for use in the expansion synthesis process described later.

<Color noise reduction calculation unit>
The color noise reduction degree calculation unit 240 calculates the color noise reduction degree for each pixel of each image data excluding the lowest layer among the multiple resolution image data processed by the correction unit 220 (color in FIG. 3). Noise reduction degree calculation processing 305a to 305d) is performed. The degree of color noise reduction is a ratio obtained by dividing the number of pixels actually referred to in the correction process (smoothing process) by the total number of pixels constituting the reference area in the correction process, and is as follows. It is obtained for each pixel using the equation (8). In this case, the pixel actually referred to in the smoothing process is a pixel satisfying the condition of the above-mentioned equation (4), and is hereinafter referred to as a "real reference pixel".

Here, the degree of color noise reduction calculated by the above formula (8) does not indicate the amount of pure color noise reduction. It shows the result of color noise reduction (color noise reduction effect) for the purpose of reducing color noise. That is, the reference area used in the correction process for each pixel of interest corresponds to the "target", and the actual number of reference pixels corresponds to the "noise reduction effect". Therefore, the color noise reduction degree obtained by the above equation (8) greatly changes depending on how the reference region is set. For example, when the pixel of interest is a pixel belonging to the edge portion, the reference region is limited to the edge region in order to prevent the harmful effects of color loss and blurring. In this case, since the size of the reference area is small, the number of pixels included in the reference area is small. Therefore, even if the number of actual reference pixels is small, the denominator is also small, and the calculated color noise reduction degree is high. On the other hand, it is assumed that the reference area is set as large as possible in order to reduce the residual noise at the edge portion as much as possible. In this case, since the size of the reference area becomes large, the number of pixels included in the reference area is large. Therefore, even if the number of actual reference pixels is large, the degree of color noise reduction calculated is low because the denominator is also large. As described above, even if the actual number of reference pixels is the same, the calculated color noise reduction degree changes depending on the size of the reference area. The degree of color noise reduction based on the above equation (8) tends to be low in situations where residual color noise is desired to be reduced as much as possible, and high in situations where color loss is desired to be prevented as much as possible.

In this embodiment, an index showing the effect obtained for the aim of the color noise to be reduced is used as the degree of color noise reduction, but an index showing a pure noise reduction amount may be used. The calculated color noise reduction degree information is stored in the RAM 102 for use in the priority use determination process described later.

<Expansion composition processing unit>
The magnifying and compositing processing unit 250 uses the compositing ratio derived by the compositing ratio derivation unit 230 and the color noise reduction degree calculation unit 240 to combine two types of image data having different resolutions among the multi-resolution image data processed by the correction unit 220. The compositing process is performed based on the derived color noise reduction degree. The enlargement synthesis processing unit 250 is further composed of an enlargement processing unit 251, a priority use determination unit 252, and a pixel value synthesis unit 253. Hereinafter, each part constituting the expansion synthesis processing unit 250 will be described.

≪Enlargement processing unit≫
The enlargement processing unit 251 processes the image data of the highest layer in the multi-resolution image data (input image data itself. The image data of each layer excluding the input image 301 in FIG. 3 is enlarged twice by, for example, the bilinear method (FIG. 3). The enlargement processing 306a to 306d) in 3 is performed. At this time, the target of the enlargement processing is the image data processed by the correction unit 220 in the lowermost layer, and the pixel value composition processing described later in the other lower layers. In the example of the sequence diagram of FIG. 3, the target of the enlargement processing 306a is the image data of the lowermost layer to which the correction processing 303e has been applied. The image data after the composition is subjected to the pixel value composition process 308a described later. The target of the enlargement process 306c is the image data after the composition subjected to the pixel value composition process 308b described later. The target of the enlargement processing 306d is the image data after the composition to which the pixel value composition processing 308c described later is applied. The method of the enlargement processing is not limited to the bilinear method, for example, the nearest neighbor method and the bye. A cubic method, a Lanczos method, or the like may be used.

≪Priority use judgment unit≫
The priority use determination unit 252 determines whether or not to preferentially use the image data in the lower layer when synthesizing the image data of interest and the image data in the layer below it (in FIG. 3). Priority use determination processing 307a to 307d) is performed. The priority use determination in this embodiment is made based on the color noise reduction degree derived by the color noise reduction degree calculation unit 240. Here, the "image data in the next lower layer" is image data whose resolution is one step lower than that of the image data of interest, and is enlarged by the enlargement processing unit 251 to the same resolution as the image data of interest. Refers to the image data. This will be described with reference to the sequence diagram of FIG.

In the priority use determination process 307a, the image data reduced to 1/8 the resolution of the input image data and the image data reduced to 1/16 the resolution of the input image data were reduced (the same resolution was obtained by the enlargement processing). Judgment is made when synthesizing with image data. Then, in the priority use determination process 307b, the image data reduced to 1/4 of the resolution of the input image data and the image data reduced to 1/8 of the input image data (the same resolution is obtained by the enlargement processing). T) Judgment is made when synthesizing with image data. Then, in the priority use determination process 307c, the image data reduced to 1/2 the resolution of the input image data and the image data reduced to 1/4 the resolution of the input image data (the same resolution is obtained by the enlargement processing). T) Judgment is made when synthesizing with image data. Then, in the priority use determination process 307d, the determination when synthesizing the input image data itself and the image data that has been reduced to half the resolution of the input image data (the same resolution is obtained by the enlargement process) is performed. Is done.

Then, this priority use determination process is performed by applying the result on the upper layer side of the color noise reduction degree calculated by the above formula (8) to the following formula (9) on a pixel-by-pixel basis.

In the above equation (9), th _deg is a threshold value set according to the amount of noise. Since the amount of noise changes depending on the layer and shooting sensitivity, set _{the threshold value th deg} to a smaller value as the layer is lower or the shooting sensitivity is smaller, and conversely, as the layer is higher or the shooting sensitivity is higher, the value is larger. To do. Then, when the condition of the above equation (9) is satisfied, that is, _{if the degree of color noise reduction on the upper layer side is smaller than the threshold value th deg,} it is determined that the image data of the lower layer is preferentially used. Then, when the determination is made in this way, the composition ratio is changed so as to increase the ratio on the lower layer side at the time of image composition (for example, 100% of the image data in the lower layer is adopted). The data of the changed composition ratio is stored in the RAM 102.

The composition ratio may be changed according to the above equation (9) only when the pixel of interest is achromatic. At that time, _{it is desirable to set th deg to} a value larger than usual and design so that the image data on the lower layer side is used more preferentially. There are two reasons for this. One is that residual color noise is particularly noticeable in the achromatic region, so it is desirable to use image data on the lower layer side as much as possible. The other is that the region determined to be achromatic on the upper layer side is less likely to cause the harmful effect of color loss even if the image data on the lower layer side is used. In this way, the residual color noise may be reduced only in the region that can be regarded as an achromatic color.

≪Pixel value synthesizer≫
The pixel value synthesizing unit 253 sets the pixel value of the image data of interest and the pixel value of the image data (same resolution as the image data of interest) after enlarging the image data in the layer below it at each stage. Performs synthesis based on the synthesis ratio derived by the synthesis ratio derivation process of. In FIG. 3, the pixel value synthesis processes 308a to 308d correspond to this. _{Specifically, the pixel value I post} after composition when the pixel value on the lower layer side is I _down , the pixel value on the upper layer side is I _up , and the composition ratio is u uses the following equation (10). Is required.

For example, when the composition ratio u = 0.1, the lower layer side pixel value I _down = 3405, and the upper layer side pixel value I _up = 3621, the pixel value I _post after composition is "3427" from the above equation (10). Become. The pixel value here may be in any format, may be a YcrCb value, or may be an RGB value.

As described above, the composition ratio u is a coefficient indicating how much image data on the upper layer side is used, and is basically derived by the composition ratio derivation unit 230 and appropriately changed depending on the result of the priority use determination process. It will be. In the case of this embodiment, the color image data after synthesis, which is the result of the pixel value synthesis processes 308a to 308c, is temporarily stored in the RAM 102 and provided to the enlargement processes 306b to 306d, respectively. On the other hand, the image data after composition, which is the result of the pixel value composition process 308d, is output as it is (see the sequence diagram of FIG. 3). Output modes at this time include, for example, display on a monitor 108, storage in HDD 103, storage in an external memory 107 or an external server (not shown) via a general-purpose I / F 104, and printing with a printer (not shown). There is something.

(Flow of color noise reduction processing)
Next, a rough flow of each process in the color noise reduction processing unit 202 will be described. FIG. 6 is a flowchart showing the flow of each process in the color noise reduction process. This series of processing is realized by the CPU 101 loading the program stored in the HDD 130 into the RAM 102 and executing it.

When the image data is input to the color noise reduction processing unit 202, in step 601 the reduction processing unit 210 performs the above-mentioned reduction processing on the acquired input image data to generate multi-resolution image data. The details of the reduction process will be described later.

In step 602, the correction unit 220 performs correction processing for correcting the color signal of the pixel of interest so as to reduce the color noise with respect to the image data of each layer in the generated multi-resolution image data.

In step 603, the composite ratio derivation unit 230 derives a composite ratio with the image data of the lower layer for each image data excluding the lowermost layer among the multi-resolution image data whose color noise is reduced by the correction process. Perform processing.

In step 604, the color noise reduction degree calculation unit 240 performs a process of calculating the color noise reduction degree for each image data excluding the lowest layer among the multiple resolution image data whose color noise is reduced by the correction process. ..

In step 605, the enlargement composition processing unit 250 uses the composition ratio derived in step 603 and the color noise reduction degree calculated in step 604 for the multi-resolution image data whose color noise has been reduced by the correction process. , Perform the above-mentioned expansion synthesis process. The details of the expansion synthesis process will be described later.

The above is a rough flow of each process in the color noise reduction processing unit 202.

(Reduction processing flow)
Subsequently, the details of the reduction process (step 601) in the flow of FIG. 6 described above will be described. FIG. 7 is a flowchart showing the flow of the reduction process.

In step 701, the reduction processing unit 210 acquires the reduction processing parameters. The reduction processing parameters include information on the depth of the multi-resolution hierarchy and how much the minimum resolution is a fraction (multiple of 2) of the input image data. Then, such reduction processing parameters are set in advance and stored in the HDD 103 or the like. In the case of this embodiment, the minimum resolution is 1/16 of the input image data in all 5 layers (1x, 1/2x, 1/4x, 1/8x, 1/16x). The content reduction processing parameters are set and saved in advance, and are acquired by reading from the HDD 103 or the like.

In step 702, the reduction processing unit 210 determines the magnification to be applied to the reduction processing based on the reduction processing parameters acquired in step 701. In the case of this embodiment in which the contents of the reduction processing parameters are all 5 layers and the minimum resolution is 1/16 of the input image data, there are 4 types of 1/2, 1/4, 1/8, and 1/16. Magnification will be determined sequentially.

In step 703, the reduction processing unit 210 performs a process of reducing the image data to be processed to the magnification determined in step 702.

In step 704, the reduction processing unit 210 determines whether or not all the reduction processing based on the reduction processing parameters acquired in step 701 has been completed. In the case of this embodiment, it is determined whether or not all the reduction processes for the four types of magnifications of 1/2, 1/4, 1/8, and 1/16 have been completed. As a result of the determination, if all the reduction processing corresponding to the acquired reduction processing parameter is completed, this processing is terminated. On the other hand, if there is an unprocessed magnification, the process returns to step 702 to determine the next magnification and continue the reduction process.

(Expanded synthesis processing flow)
Next, the details of the expansion synthesis process (step 605) in the flow of FIG. 6 described above will be described. FIG. 8 is a flowchart showing the flow of the expansion synthesis process according to the present embodiment.

In step 801 the enlargement composition processing unit 250 acquires all the image data (multi-resolution image data) whose color noise is reduced by the correction process of step 602 described above.

In step 802, the expansion / synthesis processing unit 250 acquires the above-mentioned reduction processing parameters and synthesis processing parameters. Here, the composition processing parameter means the composition ratio derived in the above-mentioned step 603 and the degree of color noise reduction calculated in the above-mentioned step 604.

In step 803, the enlargement processing unit 251 of the enlargement composition processing unit 250 performs the above-mentioned enlargement processing (this embodiment) with respect to the image data of the lowest layer (small resolution) among the plurality of image data acquired in step 801. Then double).

In step 804, the enlargement composition processing unit 250 uses the image data having the lowest resolution among the unprocessed image data among the plurality of image data acquired in step 801 as one of the image data to be subjected to the pixel value composition processing. select. In this case, the unprocessed image data does not include the image data of the lowest layer.

In step 805, the pixel value compositing unit 253 of the enlargement compositing processing unit 250 selects the pixel to be the target of the pixel value compositing process. Specifically, in the case of the first routine, the image data enlarged in step 803 and the image data selected in step 804 having the same resolution as the enlarged image data are subject to pixel value composition processing. The pixel to be is selected. Further, in the case of the second and subsequent routines, the pixel value composition processing is performed on the image data enlarged in step 811 described later and the image data selected in step 804 having the same resolution as the enlarged image data. The target pixel of is selected.

In the following steps 806 to 807, the priority use determination unit 252 of the expansion synthesis processing unit 250 performs the above-mentioned priority use determination process for the pixels selected in step 805. That is, a determination process is performed as to whether or not to preferentially use the enlarged image data (image data on the lower layer side) of the two image data to be pixel-value composited.

In step 806, it is determined by using the above-mentioned equation (9) whether or not to preferentially use the image data on the lower layer side. As a result of the determination, if the degree of color noise reduction on the upper layer side _{is smaller than the threshold value th deg} , the process proceeds to step 807 in order to preferentially use the image data on the lower layer side. On the other hand, if the degree of color noise reduction on the upper layer side _{is equal to or higher than the threshold value th deg} , the process proceeds to step 808.

In step 807, the composition ratio obtained in the composition ratio derivation process (step 603 described above) is changed to a composition ratio such that the pixel value of the image data on the lower layer side is adopted, for example, 100%.

In step 808, the pixel value compositing unit 253 of the enlargement compositing processing unit 250 performs the above-mentioned pixel value compositing process based on the currently set compositing ratio. _{As a result, the pixel value I post} obtained by synthesizing the selected pixel values in the two image data is obtained.

In step 809, the enlargement composition processing unit 250 determines whether or not the composition processing for all pixels is completed. If the processing for all pixels is completed, the process proceeds to step 810. On the other hand, if there are unprocessed pixels, the process returns to step 805, selects the next pixel as the pixel to be processed, and continues the processing.

In step 810, has the enlargement / composition processing unit 250 performed enlargement processing on all the image data (that is, all the reduced image data except the input image data) other than the image data of the highest layer among the multiple resolution image data? Judge whether or not. As a result of the determination, if there is reduced image data that has not been enlarged, the process proceeds to step 811. On the other hand, if all the reduced image data has been enlarged, this process is completed.

In step 811, the enlargement processing unit 251 of the enlargement composition processing unit 250 performs the above-mentioned enlargement processing (twice in this embodiment) on the image data after the composition processing obtained in step 808. After executing the enlargement process, the process proceeds to step 804.

The above is the content of the expansion synthesis process according to this embodiment.

As described above, according to the present embodiment, it is possible to effectively reduce the residual color noise in the vicinity of the edge while maintaining the sharpness of the color.

In the first embodiment, the residual color is maintained while maintaining the color resolution by changing the composition ratio so that the weight of the image data on the lower layer side becomes larger in the portion where the color noise reduction degree is lower than the predetermined threshold value. An image with sufficiently reduced noise can be obtained. However, if the weight of the image data on the lower layer side is increased in all the places where the degree of color noise reduction is low, adverse effects such as color change and color loss may occur. Therefore, a mode in which the above-mentioned adverse effects are not generated as much as possible by evaluating the color similarity between the upper layer and the lower layer will be described as Example 2.

By the way, also in the configuration of the first embodiment, it is possible to prevent the above-mentioned adverse effects by limiting the application target so that the composition ratio is changed only when the pixel of interest is achromatic. However, in this case, as a matter of course, the effect obtained by the color noise reduction processing is limited to the achromatic region. In this embodiment, there is an advantage that the effect of reducing color noise can be obtained even in a chromatic color region. In the following description, the characteristic configuration of the present embodiment will be described in detail, and the description of the common parts with the first embodiment will be omitted or simplified.

(Details of color noise reduction processing unit)
FIG. 9 is a block diagram showing an internal configuration of the color noise reduction processing unit 202'according to the present embodiment. The color noise reduction processing unit 202'composed of a reduction processing unit 210, a correction unit 220, a composition ratio derivation unit 230, a color noise reduction degree calculation unit 240, an enlargement composition processing unit 250', and a color similarity determination unit 900.

The enlargement composition processing unit 250'of this embodiment adds two types of image data having different resolutions among the multi-resolution image data processed by the correction unit 220 to the composition ratio and the degree of color noise reduction, and the color similarity described above. A process of synthesizing is performed in consideration of the determination result in the determination unit 900.

Similar to the first embodiment, the priority use determination unit 252'of this embodiment preferentially uses the image data of the lower layer when synthesizing the image data of interest and the image data of the next lower layer. The process of determining whether or not to perform (lower layer priority determination processes 307a to 307d in FIG. 3) is performed. The difference from the first embodiment is that the priority use determination process is performed based on both the color noise reduction degree derived by the color noise reduction degree calculation unit 240 and the determination result by the color similarity determination unit 900.

≪Color similarity judgment unit≫
First, the color similarity determination unit 900 generates a color difference signal from the RGB color signal of the image data in which the color noise is reduced by the correction unit 220. The color difference signal is generated for the image data of interest and the image data one layer below it, which is enlarged to the same resolution as the image data of interest by the enlargement processing. The color difference signals (Cr and Cb) can be generated by a known conversion formula as described above. Further, Cr may be obtained by RG and Cb may be obtained by BG. The generated color difference signal is stored in the RAM 102.

Next, the color similarity determination unit 900 is based on the color difference signal of the image data of interest and the color difference signal of the image data one layer below it and expanded to the same resolution as the image data of interest. Then, the color similarity between the upper layer side and the lower layer side of the pixel of interest is determined. This determination evaluates the rough color similarity. The reason is that if the evaluation is performed strictly, the portion where the residual color noise is desired to be reduced cannot be determined, but if the evaluation is not performed at all, adverse effects such as color loss occur. Specifically, for example, the following equation (11) is used.

In the above equation (11), Cr _down and Cb _down represent the color difference signal on the lower layer side, and Cb _up and Cr _up represent the color difference signal on the upper layer side. Then, this equation (11) evaluates the color similarity based on whether or not the result of polar coordinate transformation of the color space of the color difference signal is within the same specific range on the lower layer side and the upper layer side. .. That is, when Cr and Cb are regarded as vectors on the xy coordinate plane, it is evaluated whether or not the quadrant to which the conversion result on the upper layer side belongs and the quadrant to which the conversion result on the lower layer side belongs match (FIG. 10). See). Thereby, it is possible to determine the rough similarity of the hue of the color signal between the upper layer side and the lower layer side.

Alternatively, instead of the above formula (11), the following formulas (12) to (14) may be used.

The above equations (12) and (13) mean that when the value of each color difference signal on the lower layer side is equal to or less than a predetermined value close to 0, coring is _performed . It is represented by Cb down _CMP. As a result, when the signal value at which erroneous determination is likely to occur due to the influence of noise is near zero, the aim is to exceptionally exclude it from the target for determining the color similarity based on the above equation (14).

It should be noted that the determination method is not limited to these examples as long as it evaluates the similarity of the hues of the color signal on the lower layer side and the color signal on the upper layer side. For example, the similarity may be determined based on whether or not the distance between the conversion result on the lower layer side and the conversion result on the upper layer side is within a predetermined threshold value. In the case of this determination method, even if the result of polar coordinate conversion of the color space of the color difference signal is located near the intersection 1000 of the Cr axis and the Cb axis and the quadrants do not match between the lower layer side and the upper layer side. If the distance between the two is short, it can be determined that they are similar. The color similarity determination result thus obtained is stored in the RAM 102 for use in the priority use determination process in the priority use determination unit 252'.

≪Priority use judgment processing≫
The priority use determination unit 252'determines whether or not to preferentially use the image data of the lower layer with the color noise reduction degree derived by the color noise reduction degree calculation unit 240 and the color in the color similarity determination unit 900 described above. Judgment is made based on the result of similarity judgment. Specifically, first, as in the case of the first embodiment, a comparison process is performed between the color noise reduction degree on the upper layer side and the predetermined threshold value th _deg. As a result of the threshold comparison process, if the degree of color noise reduction on the upper layer side _{is smaller than the threshold th deg} , then, referring to the result of the above-mentioned color similarity determination, the colors on the upper layer side and the lower layer side are similar. If so, the compositing ratio is changed so as to increase the ratio on the lower layer side at the time of image compositing.

By adding the color similarity condition in this way, it is possible to increase the composition ratio on the lower layer side by limiting the locations where the colors are similar to some extent on the lower layer side and the upper layer side. The data of the changed composition ratio is stored in the RAM 102.

(Expanded synthesis processing flow)
Next, the details of the expansion synthesis process (step 605) of this embodiment will be described. FIG. 11 is a flowchart showing the flow of the expansion synthesis process according to the present embodiment.

Steps 1101 to 1106 correspond to steps 801 to 806 of the flow of FIG. 8 of the first embodiment, respectively. That is, the multi-resolution image data in which the color noise is reduced is acquired (step 1101), and then the reduction processing parameter and the composition processing parameter are acquired (step 1102). In this case, the composition processing parameters of this embodiment include the result of the color similarity determination in addition to the composition ratio and the degree of color noise reduction. Then, the enlargement processing is executed on the image data of the lowest layer in the acquired multi-resolution image data (step 1103). Subsequently, among the acquired multi-resolution image data, the image data having the lowest resolution among the unprocessed image data is selected as one of the image data to be subjected to the pixel value composition processing (step 1104). When the image data is selected, the pixel to be subjected to the pixel value composition processing is selected (step 1105).

In the following steps 1106 to 1108, the priority use determination unit 252'of the expansion synthesis processing unit 250'performs the above-mentioned priority use determination process for the pixels selected in step 1105.

First, in step 1106, it is determined whether or not the above equation (9) is satisfied. As a result of the determination, if the degree of color noise reduction on the upper layer side _{is smaller than the threshold value th deg} , the process proceeds to step 1107. On the other hand, if the degree of color noise reduction on the upper layer side _{is equal to or higher than the threshold value th deg} , the process proceeds to step 1109.

In step 1107, the priority use determination unit 252'refers to the result of the color similarity determination included in the synthesis processing parameter acquired in step 1102, and separates the processing. Specifically, if it is a determination result that there is color similarity between the upper layer side and the lower layer side, the process proceeds to step 1108 in order to preferentially use the image data on the lower layer side. On the other hand, if it is determined that there is no color similarity, the process proceeds to step 1109. Then, in step 1108, the composition ratio obtained in the composition ratio derivation process (step 603 described above) is changed to a composition ratio such that the pixel value of the image data on the lower layer side is adopted, for example, 100%.

Subsequent steps 1109 to 1112 correspond to steps 808 to 811 of the flow of FIG. 8 and are not different from each other, and thus description thereof will be omitted.

The above is the content of the expansion synthesis process according to this embodiment. Here, the color similarity determination unit 900 is used as an independent component, and the determination result is referred to by the priority use determination unit 252', but this embodiment is not limited to this. For example, the function of the color similarity determination unit 900 may be incorporated into the priority use determination unit 252'.

As described above, in the present embodiment, the composition ratio on the lower layer side can be increased by limiting the locations where the colors are similar to some extent on the lower layer side and the upper layer side. As a result, it is possible to effectively reduce residual color noise in the vicinity of the edge while maintaining the sharpness of the color while suppressing adverse effects such as color change and color loss. Further, by applying Example 1 when the pixel of interest is achromatic, and by applying Example 2 in other cases, the harmful effects of color change and color loss are suppressed, and residual color noise is particularly noticeable. It is possible to enhance the noise reduction effect in the achromatic region.

<Other Examples>
In Examples 1 and 2, examples of processing by the image processing application have been described, but these may be a method of processing the image data captured by the image pickup device on the image processing hardware in the image pickup device. Absent. Further, the image data may be transmitted from the client device to the image processing application on the server device, and the image data may be processed on the server device.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (11)

A reduction processing means that performs reduction processing on the input image data to generate reduced image data of a plurality of layers with reduced resolution, and
A correction means for correcting the pixel value of the pixel of interest in the input image data and the reduced image data to a value with reduced color noise.
Among the corrected image data, a composite ratio deriving means for deriving a composite ratio of the image data excluding the reduced image data having the lowest resolution with the image data of another layer,
A color noise reduction degree calculation means for calculating the color noise reduction degree by the correction for each pixel for the image data excluding the reduced image data having the lowest resolution among the corrected image data.
A compositing means for synthesizing the image data of interest and the image data of the next lower layer of the corrected image data based on the compositing ratio and the color noise reduction degree.
Equipped with a,
The synthetic means
Enlargement processing means for enlarging image data and
The color difference signal of the pixel of interest of the image data of interest and the color difference signal of the pixel of interest of the image data one layer below the image data of interest and enlarged to the same resolution as the image data of interest by the enlargement processing means. Color similarity determination means for determining color similarity with and
When synthesizing the image data of interest and the image data of the next lower layer, it was determined that the degree of color noise reduction was smaller than a predetermined threshold value and the colors were similar by the color similarity determining means. In some cases, it is determined that the image data of the next lower layer is preferentially used, and when it is determined that the image data of the next lower layer is preferentially used, the image data of the next lower layer is preferentially used. The priority use determination means for changing the composite ratio so as to increase the weight of
The composition ratio of the pixel value of the image data of interest and the pixel value of the image data one layer below the image data expanded to the same resolution as the image data of interest by the enlargement processing means. Pixel value synthesizing means for synthesizing based on
To have a,
An image processing device characterized by this. The image processing apparatus according to claim 1 , wherein the priority use determination means performs the determination only when the pixel of interest is achromatic. The image processing apparatus according to claim 1, wherein the predetermined threshold value is set to a smaller value as the hierarchy is lower and a larger value as the hierarchy is higher. The color similarity determining means determines that the colors are similar when the results of polar coordinate transformation of the color space of the color difference signals on the lower layer side and the upper layer side are within the same specific range on the lower layer side and the upper layer side. The image processing apparatus according to claim 1 , wherein the image processing apparatus is used for determination. The color similarity determining means determines that the colors are similar when the distance between the lower layer side and the upper layer side is within a predetermined threshold as a result of polar coordinate transformation of the color space of the color difference signals on the lower layer side and the upper layer side. The image processing apparatus according to claim 1 , wherein the image processing apparatus is used for determination. Any of claims 1 to 5 , wherein the color similarity determining means does not determine the color similarity when the value of the color difference signal in the next lower layer is equal to or less than a predetermined value. The image processing apparatus according to item 1. The pixel value synthesizing means obtains the pixel value after synthesizing using the following equation (5), and obtains the pixel value after synthesizing.

In the above equation (5), I _post represents the pixel value after synthesis, I _down represents the pixel value of the image data on the lower layer side, I _up represents the pixel value on the upper layer side, and u represents the pixel value on the upper layer side. Represents a coefficient as the composite ratio, which indicates how much image data is used.
The image processing apparatus according to any one of claims 1 to 6 , wherein the image processing apparatus is characterized by the above. The correction in the correction means is a smoothing process using a reference region composed of adjacent pixels surrounding the pixel of interest.
The color noise reduction degree calculated by the color noise reduction degree calculating means is the number of pixels actually referred to in the smoothing process among the pixels constituting the reference area, and all the pixels constituting the reference area. The image processing apparatus according to any one of claims 1 to 7 , wherein the ratio is obtained by dividing by a number. The input image data is image data having RGB color components, and is
The correction means corrects the value of the color signal of the pixel of interest for each color component using equations (1) to (3).

In the equations (1) to (3), R [i], G [i], and B [i] represent pixels satisfying the equation (4) in the reference region with respect to the pixel of interest, and M is the pixel of interest. Represents the number of pixels satisfying the equation (4) in the reference area with respect to.
_{R diff <= Th r ∧G diff} <= Th g ∧B diff <= Th b ··· formula (4)
In the formula (4), ∧ represents the logical product (AND), Th _r represents the threshold for the R signal, Th _g is a threshold for the G signal, Th _b is a threshold value for B signals, respectively, R _diff, G _diff, B _diff represents the difference between each pixel value in the reference region and the pixel value to be compared.
The image processing apparatus according to claim 8. A step of performing reduction processing on the input image data to generate reduced image data of a plurality of layers with reduced resolution, and
A step of correcting the pixel value of the pixel of interest in the input image data and the reduced image data to a value with reduced color noise.
Among the corrected image data, the step of deriving the composition ratio of the image data excluding the reduced image data having the lowest resolution with the image data of other layers, and
A step of calculating the degree of color noise reduction by the correction for each pixel for the image data excluding the reduced image data having the lowest resolution among the corrected image data.
A step of synthesizing the image data of interest and the image data of the next lower layer of the corrected image data based on the composition ratio and the color noise reduction degree, and a step of synthesizing the image data.
Have,
The step of synthesizing is
Steps to enlarge the image data and
The color difference signal of the pixel of interest of the image data of interest and the color difference of the pixel of interest of the image data one layer below it and expanded to the same resolution as the image data of interest in the expanding step. Steps to determine color similarity to the signal,
When the image data of interest and the image data of the next lower layer are combined, the degree of color noise reduction is smaller than a predetermined threshold value, and the colors are similar in the step of determining the similarity of the colors. If it is determined determines to use the image data of preferentially said one of a hierarchy, when determining to use the image data of preferentially said one of a hierarchy, the hierarchy under the one The step of changing the composition ratio so as to increase the weight of the image data of
The pixel value of the image data of interest and the pixel value of the image data one layer below the image data expanded to the same resolution as the image data of interest in the enlargement step are combined. Steps to synthesize based on ratio and
An image processing method comprising. A program for causing a computer to function as the image processing device according to any one of claims 1 to 9.

Images