JP2016110340A - Image processing device, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize noise reduction processing using block matching excellent even for a gradation region.SOLUTION: An image processing device comprises: setting means for setting a target pixel in input image data and a target area corresponding to the target pixel, and a reference pixel and a reference area corresponding to the reference pixel; analysis means for analyzing a magnitude relation between a pixel value of a pixel in the target area and a pixel value of a corresponding pixel in the reference area to calculate an index which shows deviation in the magnitude relation between the corresponding pixels included in the target area and the reference area; weight derivation means for, on the basis of the pixel value of the pixel in the target area, the pixel value of the pixel in the reference area and the index, deriving weight of the reference pixel corresponding to the reference area; and calculation means for, on the basis of the pixel value of the reference pixel and the weight corresponding to the reference pixel, calculating a weighted mean to calculate a pixel value of the target pixel.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a technique for reducing noise in image data.

  In recent years, in order to capture darker scenes and faster subjects, it has been desired to make the camera more sensitive than before. However, if the sensitivity of the camera is increased, noise in the captured image increases, so noise reduction processing is required to more strongly reduce noise. When noise reduction processing using a simple weighted average is performed, the resolution is lowered. Therefore, there is a need for a technique for reducing noise while preserving image features such as edges and textures. A general technique for noise reduction technology that preserves such edges is to obtain the similarity between the target pixel whose noise is to be reduced and the surrounding reference pixels, and to determine the weight for each reference pixel according to the similarity. There is a way to determine and calculate a weighted average. Non-Patent Document 1 discloses a noise reduction process called a non-local means method. In order to calculate the similarity between the target pixel and the reference pixel, it is disclosed to use block matching that compares each corresponding pixel in the target region near the target pixel and the reference region near the reference pixel. The sum of squared differences between each pixel included in the region of interest and each pixel included in the reference region is calculated as the similarity.

“A non-local algorithm for image denoising” IEEE Computer Vision and Pattern Recognition 2005, Vol. 2, 60-65, (2005)

  According to the method disclosed in Non-Patent Document 1, in the gradation part, it is easy to determine that the reference areas whose positions are different in the gradation direction are not similar to the attention area. Differences occur between all corresponding pixels between regions having different positions in the gradation direction. In the sum of squared differences, the difference between the pixels is accumulated, so that the value of the sum of squared differences that becomes the similarity of the reference region becomes large. As a result, there are fewer reference pixels with a large weight value, and there is a problem that a sufficient noise reduction effect cannot be obtained even if the pixel value after the noise reduction processing of the target pixel is calculated by the weighted average of the reference pixels. .

  Therefore, an object of the present invention is to realize a good noise reduction process even for a gradation area in a noise reduction process using block matching.

  In order to solve the above problems, the present invention is an image processing apparatus that performs noise reduction processing, and includes a target pixel and a target region corresponding to the target pixel in input image data, and a reference pixel and a reference region corresponding to the reference pixel. And a corresponding pixel included in the target region and the reference region by analyzing the magnitude relationship between the pixel value of the pixel in the target region and the pixel value of the corresponding pixel in the reference region. The reference pixel corresponding to the reference region based on the analysis means for calculating the index indicating the bias of the magnitude relationship between the pixel value, the pixel value of the pixel in the region of interest, the pixel value of the pixel in the reference region, and the index A weight deriving means for deriving a weight of the reference pixel, and calculating a weighted average based on a pixel value of the reference pixel and a weight corresponding to the reference pixel. It characterized by having a calculating means for calculating a pixel value of the target pixel.

  According to the present invention, a better noise-reduced image can be obtained in the block matching gradation portion.

It is a block diagram which shows the structure of an image processing apparatus. It is a figure explaining the relationship between an attention pixel, an attention area, a reference pixel, and a reference area. It is the figure which showed one example of the gradation part. Diagram explaining gradation index It is a block diagram which shows the logic structure of a noise reduction process part. It is a flowchart which shows the flow of a noise reduction process.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. In addition, the structure shown in the following Examples is only an example, and this invention is not limited to the structure shown in figure.

<First Embodiment>
A hardware configuration of the image processing apparatus according to the first embodiment will be described with reference to FIG. The image processing apparatus according to the present embodiment will be described as an example of an image processing application executed on a personal computer (PC). The image processing application performs noise reduction processing using non-local means method on the input image data. The image processing apparatus includes a CPU 101, a RAM 102, an HDD 103, a general-purpose interface (I / F) 104, a monitor 108, and a main bus 109. The general-purpose I / F connects an imaging device 105 such as a camera, an input device 106 such as a mouse and a keyboard, and an external memory 107 such as a memory card to a main bus 109.

  In the following, the CPU 101 controls the entire image processing apparatus by operating various software (computer programs) stored in the HDD 103. Although the case where the CPU 100 controls the entire apparatus will be described here, the entire apparatus may be controlled by a plurality of hardware sharing the processing.

  First, the CPU 101 activates an image processing application stored in the HDD 103, expands it in the RAM 102, and displays a user interface (UI) on the monitor 108. Subsequently, various data stored in the HDD 103 and the external memory 107, an image captured by the imaging device 105, an instruction from the input device 106, and the like are transferred to the RAM 102. The RAM 102 has a storage area for temporarily storing received data. In accordance with processing in the image processing application, data stored in the RAM 102 performs various calculations based on commands from the CPU 101. The calculation result is displayed on the monitor 108 or stored in the HDD 103 or the external memory 107.

  In the above-described configuration, details of processing for performing noise reduction processing by inputting an image to be subjected to noise reduction to an image processing application based on a command from the CPU 101 will be described. First, basic processing of the non-local means method will be described.

<Outline of Non-local means method>
It is known that an image obtained by photographing is obtained by adding a noise component to a signal component that is an original value without noise. An ideal noise reduction process is to calculate the average of the pixel values of a plurality of pixels having the same signal component as the signal component of the pixel of interest. As a result, the standard deviation of the noise component is reduced by the average while the signal component remains as it is, the noise component of the pixel of interest is reduced, and a noise-reduced image is obtained. However, since there is a noise component, the signal component of each pixel in the image cannot be accurately understood.

  Therefore, in the non-local means method, the weight of the reference pixel is determined based on the similarity of the distribution of the pixel values in the target region corresponding to the target pixel and the reference region corresponding to the reference pixel, and weighted averaging is performed. As a result, the noise component of the pixel of interest is reduced. FIG. 2 is a diagram for explaining the non-local means method. A region of 5 pixels × 5 pixels including the target pixel with respect to the target pixel (black pixel) is set as the target region. In addition, an area of 5 pixels × 5 pixels including the reference pixel with respect to the reference pixel (hatched pixel) is set as a reference area. Based on the difference in pixel value between each pixel in the region of interest and each pixel in the reference region, a similarity indicating how much the region of interest and the reference region are similar is calculated. The degree of similarity can be calculated by the sum of squared differences of pixel values between corresponding pixels as in equation (1).

  In the image data, the pixel value of the i-th pixel of interest is Ii, and the pixel value of the j-th reference pixel is Ij for the i-th pixel of interest. Further, k indicates a corresponding position in each region. That is, Iik indicates the pixel value of the pixel at the k-th position in the region of interest corresponding to the i-th pixel of interest, and Ijk is the pixel value of the pixel at the k-th position in the reference region corresponding to the j-th reference pixel. Indicates. Nb indicates the number of reference pixel groups corresponding to the i-th region of interest (81 in this embodiment). Similarity Sij is the total sum of the squares of the difference of pixel values for the number of pixels included in the target area (equivalent to the number of pixels included in the reference area), as in Expression (1). The similarity calculated by the difference between pixel values means that the region of interest and the reference region are more similar as the value indicated by the similarity is smaller (closer to 0), and the region of interest as the value indicated by the similarity is larger. Means that the reference area is not similar. The similarity calculated in this way is calculated for all the pixels of the reference pixel group (9 pixels including the target pixel in FIG. 2 and 81 pixels of 9 pixels), and a weight corresponding to the similarity is determined. Therefore, the similarity Sij is calculated for the reference pixel of the Nb pixel for the i-th pixel of interest.

  The weight of each reference pixel may be derived such that the smaller the value indicated by the similarity (the more similar), the larger the weight, and the smaller the value indicated by the similarity, the smaller the value indicated by the similarity. The noise reduction process as described above can be expressed by Expression (2). Assuming that the pixel value of the j-th reference pixel is Ij and the weight of the j-th reference pixel with respect to the i-th target pixel is wij, the pixel values Ii, new after the noise reduction processing of the i-th target pixel are expressed by the formula (1 ).

  The denominator is normalized so that the sum of the weights becomes 1. In the example shown in FIG. 2, the target area includes the target pixel, and the reference area includes the reference pixel. However, the target area does not necessarily include the target pixel, and each reference area does not necessarily include each reference pixel. There is no need.

  Here, the similarity when the image data is a gradation portion will be described. FIG. 3A is an example of an image of a gradation portion in which pixel values gradually decrease from bottom to top. Each square represents a pixel, and a number written in the square represents a pixel value stored in the pixel. In order to facilitate the following description, the pixel value indicates only the signal component. A pixel surrounded by a thick line at the center is defined as a pixel of interest. The region of interest has a size of 5 pixels × 5 pixels around the pixel of interest. When the region of interest is set as shown in FIG. 3A, the reference pixels corresponding to the reference region determined to have a low similarity (similarity) based on the sum of squares of the difference to the region of interest are shown in FIG. The pixel group is limited to an approximately one-dimensional pixel group in the horizontal direction surrounded by a thick line in b). This is because, in the calculation of the similarity between the reference region shifted by one pixel or more in the vertical direction and the region of interest, a difference in pixel value occurs between each corresponding pixel in the region, and the difference between all the pixels is accumulated. It is to do. Accordingly, it is determined that the reference region shifted by one pixel or more in the vertical direction is not similar to the target region, and the number of reference pixels to which a weight greater than 0 is given is reduced. As a result, the reference pixel used for the weighted average for calculating the pixel value after the noise reduction processing of the target pixel is reduced, and the noise reduction effect is reduced.

<Configuration of noise reduction processing section>
As described above, when the similarity between the target area and the reference area is calculated using block matching and the weight is derived, in the gradation part, many of the surrounding reference areas are not similar to the target area. And the noise reduction effect is reduced. Therefore, in the present embodiment, a method for determining in advance whether or not a predetermined area near the target pixel related to the target area and the reference area is a gradation portion will be described. FIG. 5 shows a detailed logical configuration of the noise reduction processing unit in the present embodiment.

  The area setting unit 601 sets a target area, a plurality of reference pixels, and a reference area for the i-th target pixel for the input image data, and acquires a pixel value. Assume that the image data input here is image data corresponding to each color of R (red), G (green), and B (blue). Each image data is, for example, 8-bit data composed of pixels in which any pixel value from 0 to 255 is stored. Subsequent processing is executed for image data corresponding to each color. The area setting unit 601 sets a target pixel, a target area, a reference pixel, and a reference area as shown in FIG. That is, when the target pixel i is set, the region setting unit 601 sets 5 pixels × 5 pixels around the target pixel i as the target region. For one target pixel, reference pixels are set in order from the reference pixel group, and a reference area of 5 pixels × 5 pixels centering on the reference pixel is set in the same manner as the target area. In the present embodiment, 91 pixel reference areas are set for one target pixel.

  The magnitude relation analysis unit 502 analyzes the magnitude relation between the pixel in the target area and the pixel in the reference area. By comparing the magnitude relationship between the kth pixel in the region of interest and the kth pixel in the reference region, it is determined whether the region of interest and the reference region are gradation portions. The similarity calculation unit 502 calculates the similarity of the reference region with respect to the region of interest based on the pixel value of each pixel in the region of interest, the pixel value of each pixel in the reference region, and the analysis result by the magnitude relationship analysis unit 502. Details of the analysis method and the similarity calculation method using the analysis result will be described later.

  The weight calculation unit 504 determines a weight for each reference pixel based on the similarity for each reference pixel obtained from the similarity calculation unit 503. The weighted average calculation unit 505 calculates a pixel value after noise reduction processing of the pixel of interest based on a weighted average of each pixel value of the reference pixel group and the corresponding weight. Hereinafter, the analysis method in the first relationship analysis unit 502 and the similarity calculation method according to the analysis result in the similarity calculation unit 502 will be specifically described. As described above, here, a region of 5 pixels × 5 pixels including the target pixel is set as the target region with respect to the target pixel. Further, an area of 5 pixels × 5 pixels when a certain pixel in the reference pixel group is used as a reference pixel is set as a reference area. Therefore, 25 pairs of pixels are formed in the target area and the reference area. The magnitude relation analysis unit 502 analyzes the magnitude relation between corresponding pixels in each region.

  The similarity calculation unit 503 calculates the difference square sum Dij as one of the indexes for calculating the similarity between the attention area and the reference area.

  In addition, the right side of Formula (3) is the same as Formula (1).

Here, Iik−Ijk in the contents of the sum symbol Σ is a difference value Δijk between the pixel value of the kth pixel in the region of interest and the pixel value of the kth pixel in the reference region.
Δ _ijk = I _ik −I _jk equation (4)
The magnitude relation analysis unit 502 calculates Gij defined by Expression (5) based on the difference value Δijk. Here, the index Gij is a rough index for determining whether or not the region of interest and the reference region are gradation portions, and is therefore referred to as a gradation index.

  Here, H is a function defined by equation (6).

  The function H (Δijk) derives −1 when the pixel value of the kth pixel in the region of interest is smaller than the pixel value of the kth pixel in the reference region (Δijk <0). If the pixel value of the kth pixel in the attention area is the same as the pixel value of the kth pixel in the reference area (Δijk = 0), 0 is derived. When the pixel value of the kth pixel in the region of interest is larger than the pixel value of the kth pixel in the reference region (Δijk> 0), 1 is derived. The gradation index Gij shown in Expression (5) is an absolute value obtained by accumulating values output in accordance with the magnitude relationship between the pixel value of the kth pixel in the region of interest and the pixel value of the kth pixel in the reference region. It is. Accordingly, the fact that the gradation index Gij indicates a large value indicates that the magnitude relationship is biased among the corresponding pixels in each region. Here, the value of the gradation index Gij when the pixel value of the pixel in the region of interest is either higher or lower than the pixel value of the pixel in the reference region at most positions in the 25 pairs of pixel groups in the region. Becomes larger.

  Here, FIG. 4 is a diagram for explaining the reason why the gradation index Gij is a rough index for determining whether or not both the target region and the reference region are gradation portions. Consider a one-dimensional image for simplicity. In FIG. 4A, the horizontal axis indicates the pixel position in the one-dimensional direction, and the vertical axis indicates the pixel value. A black circle indicates a pixel value at each pixel position. That is, FIG. 4A schematically shows pixel values of 12 pixels constituting the image. If this image is a flat part, the black circles should be arranged at the same pixel value. However, the pixel value of the flat portion including noise changes randomly.

  The pixels included in the region of interest are 5 pixels, the pixels included in the reference region are 5 pixels, and the corresponding pixels between the regions are indicated by dashed arrows. The arrow portion indicated by the solid line ahead of the broken line indicates the magnitude relationship of the pixel pair. In the case of a flat part, the pixel value is the same if there is no noise, but the magnitude relationship changes randomly depending on the noise. Therefore, when the image is a flat portion, the gradation index Gij does not take a very large value.

  FIG. 4B shows an example of an image of the edge portion. The target pixel and reference pixel are also shown in the same manner as in FIG. Here, in order to make the change in the magnitude relationship easy to understand, it is assumed that the focus area and the reference area each include 7 pixels. In the case of the edge portion shown in FIG. 4B, there is a sharp change in the pixel value between the fifth pixel and the sixth pixel from the left, and both ends of the edge at the boundary between the fifth pixel and the sixth pixel. Has a flat part. As a result, the contribution of the magnitude relationship of pixel values generated at the edge portion (the boundary between the fifth pixel and the sixth pixel) to the gradation index is small. In the very vicinity of the edge portion, when the pixel value difference is several times larger than the noise, the magnitude relation is biased, but the magnitude relation is biased only in a part where there is an edge, and therefore the gradation index Gij Is rarely a large value.

  On the other hand, FIG. 4C shows an example of the image of the gradation portion. In the gradation portion, the pixel value gradually changes, and when the target region and the reference region are both in the gradation region, the magnitude relationship (sign of Δijk) between the corresponding pixels is biased to any one. Even if noise is added to each pixel in the gradation portion, if the target pixel and the reference pixel are separated from each other to some extent, the magnitude relationship (sign of the difference value Δijk) between the corresponding pixels is biased in either direction. For this reason, in the gradation part, the gradation index Gij tends to be a large value. Accordingly, it can be said that a region having a large gradation index Gij is highly likely to be a gradation portion, and a region having a small gradation index Gij is unlikely to be a gradation portion.

In order to maintain the noise reduction effect even in the gradation portion, when the gradation index Gij is high, the value is made smaller than the similarity calculated only by the sum of squared differences with respect to the weight of the reference pixel (similarity Correct it in the direction of Therefore, the similarity calculation unit 503 according to the present embodiment calculates the similarity Sij based on the index Dij based on the sum of squared differences and the gradation index Gij, as shown in Expression (7).
S _ij = D _ij −ρG _ij equation (7)
Here, ρ is a positive number. Since the correction effect is stronger as ρ is larger, the noise reduction effect in the gradation portion is enhanced. Note that if ρ is set to an extremely large value, the region other than the gradation portion is blurred. The specific value of ρ may be determined empirically by checking the image data after the noise reduction processing while adjusting ρ, but as a guideline, it is preferably about twice the noise variance.

<Processing flow in noise reduction processing>
FIG. 6 is a flowchart of the processing of the noise reduction processing unit in the first embodiment. The CPU 101 may read a program for executing the flowchart shown in FIG.

  First, in step S601, the region setting unit 501 sets a target region, a plurality of reference pixels, and a reference region for the i-th target pixel from the input image data. In step S602, the similarity calculation unit 503 calculates the sum of squared differences using Expression (3) based on the pixel value of each pixel in the region of interest and the pixel value of each pixel in the reference region, and determines an index Dij. In step S603, the magnitude relation analysis unit 502 calculates the gradation index Gij using the equations (5) and (6). Note that the order of processing of step S602 and step S603 is not limited. Processing may be performed in parallel, or the gradation index Gij may be calculated first.

  In step S <b> 604, the similarity calculation unit S <b> 503 calculates the similarity of each reference region using the expression Dij and the gradation index Gij that are sums of squares of differences using Expression (7). In step S605, the weight calculation unit 506 determines the weight Wij for each reference pixel corresponding to the reference area as in Expression (8) based on the similarity of each reference area.

  In step S606, the weighted average calculation unit 505 performs weighted average using the pixel value and the weight of each reference pixel as in Expression (1), and outputs the result as the pixel value after the noise reduction processing of the target pixel. .

  As described above, in the present embodiment, the magnitude relationship between each corresponding pixel pair in the target area and the reference area is analyzed, and an index as to whether it is a gradation portion is obtained from the deviation in the magnitude relation. By calculating the similarity using this index, the noise reduction effect can be enhanced even in the gradation portion.

<Modification 1>
In the first embodiment, the magnitude relationship between the pixel value of the kth pixel in the target area and the kth pixel value in the reference area is classified into three using Expression (6), and the magnitude is represented by Expression (5). An index that reflects the bias of the relationship in the magnitude of the value was calculated. The gradation index Gij only needs to be able to calculate the bias of the magnitude relationship between Iik and Ijk, and is not necessarily defined by the equations (5) and (6). For example, the ratio Rijk = Iik / Ijk may be used instead of the difference Δijk. Then, instead of the function H, a function J that is 1 when the ratio Rijk is greater than 1, 0 when the ratio Rijk is 1, and -1 when the ratio Rijk is smaller than 1 is introduced. It is possible to get.

  Further, the magnitude relation analysis unit 502 may determine the magnitude relation bias by a counter that counts the sign of the difference value Δij. When the sign of Δij is plus (+), +1 is counted, and when the sign of Δij is minus (−), −1 is counted. As a result, when there is a bias in the magnitude relationship, the absolute value of the counter increases, so that the gradation portion can be easily determined as in the first embodiment.

<Modification 2>
In the first embodiment, the similarity calculation unit 503 outputs the similarity based on the gradation index that is the magnitude relationship analysis result and the sum of squared differences. However, the weight calculation unit 504 may refer to the result of the magnitude relation analysis unit 502. In this case, the similarity calculation unit 503 outputs the sum of squared differences as the similarity. When the result of the analysis by the magnitude relation analysis unit 502 indicates that the region of interest and the reference region are likely to be a gradation portion, the weight calculation unit 504 determines the similarity (sum of squared differences) obtained from the similarity calculation unit 503. ) Is corrected in the direction of decreasing, and the weight is determined. Alternatively, when the result of analysis by the magnitude relation analysis unit 502 indicates that there is a high possibility that the region of interest and the reference region are gradation portions, it is determined from the similarity (sum of squared differences) obtained from the similarity calculation unit 503. You may correct | amend to the direction which increases a weight with a hand in a weight.

<Modification 3>
The weighted average calculation unit 505 calculates the pixel value after the noise reduction processing of the pixel of interest by Expression (2). As another example, the average value Mi of the pixel values in the region of interest and the average value Mj of the pixel values in the reference region may be calculated, and the weighted average may be calculated using Equation (9).

  Expressions (2) and (7) have a difference between whether or not the difference between the average values is added to the pixel value of the reference pixel used for the weighted average. When the weighted average is calculated by the equation (1), the low frequency noise is easily reduced, and when the weighted average is calculated by the equation (7), the contrast tends to be high.

<Modification 4>
In the first embodiment, the similarity calculation unit 503 calculates the sum of squared differences between each pixel in the attention area and each pixel in the reference area as one index for determining the similarity. An index based on a method other than the sum of squared differences can be applied to calculate the similarity. For example, instead of the sum of squared differences, a sum of absolute differences or a sum of squared differences is possible. However, in that case, ρ in equation (7) is also changed accordingly.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

501 Area setting unit 502 Size relationship analysis unit 503 Similarity calculation unit 504 Weight calculation unit 505 Weighted average calculation unit

Claims (6)

Setting means for setting a target pixel in the input image data, a target region corresponding to the target pixel, and a reference region corresponding to the reference pixel and the reference pixel;
Analyzes the magnitude relationship between the pixel value of the pixel in the target area and the pixel value of the corresponding pixel in the reference area, and shows the bias of the magnitude relation between the corresponding pixels included in the target area and the reference area An analysis means for calculating an index;
Weight deriving means for deriving a weight of the reference pixel corresponding to the reference region based on a pixel value of the pixel in the region of interest, a pixel value of the pixel in the reference region, and the index;
An image processing apparatus comprising: a calculation unit that calculates a pixel value of the target pixel by calculating a weighted average based on a pixel value of the reference pixel and a weight corresponding to the reference pixel.   The image according to claim 1, wherein the analysis unit calculates the index such that the value increases as there is a bias in a magnitude relationship between corresponding pixels included in the region of interest and the reference region. Processing equipment.   The weight deriving means may be configured so that the weight calculated when the index is biased in the magnitude relationship is larger than the weight calculated when the index is biased in the magnitude relationship. The image processing apparatus according to claim 1, wherein a pixel weight is derived.   The analysis means calculates a difference between a pixel value of a pixel in the region of interest and a pixel value of a corresponding pixel in the reference region, and determines a value depending on whether the difference is greater than 0 or less than 0. The image processing apparatus according to any one of claims 1 to 3, wherein the index is calculated using an image.   A computer program that causes a computer to function as the image processing apparatus according to claim 1 by being read and executed by a computer. Set the target pixel and the target region corresponding to the target pixel in the input image data, the reference pixel and the reference region corresponding to the reference pixel,
Analyzes the magnitude relationship between the pixel value of the pixel in the target area and the pixel value of the corresponding pixel in the reference area, and shows the bias of the magnitude relation between the corresponding pixels included in the target area and the reference area Calculate the metrics,
Based on the pixel value of the pixel in the region of interest, the pixel value of the pixel in the reference region, and the index, the weight of the reference pixel corresponding to the reference region is derived,
An image processing method comprising: calculating a weighted average based on a pixel value of the reference pixel and a weight corresponding to the reference pixel to calculate a pixel value of the pixel of interest.

Images