JP2021077037A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To provide an image processing technology that can easily remove noise in a fixed pattern.SOLUTION: An image processing apparatus detects a plurality of second partial regions similar in the distribution of pixel values to a first partial region in an input image. The image processing apparatus generates, for each of the plurality of second partial regions, vectors having pixel values in the second partial regions as elements. The image processing apparatus converts the generated vectors such that the standard deviations of the elements at corresponding positions in the generated vectors become equal to each other. On the basis of a plurality of eigenvalues determined from a covariance matrix based on the converted vectors and dispersion between the converted vectors, the image processing apparatus calculates results of correcting the plurality of eigenvalues as correction values. The image processing apparatus corrects the converted vectors on the basis of the calculated correction values and the eigenvectors determined from the covariance matrix. The image processing apparatus performs inverse conversion for the corrected vectors. The image processing apparatus creates an image after noise reduction processing on the input image on the basis of the converted vectors.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing apparatus, an image processing method, and a program.

The image captured by the camera contains noise, and the amount of noise increases especially when imaging with high sensitivity. Conventionally, in order to suppress such an increase in the amount of noise, a technique has been known in which a patch composed of a plurality of pixels is focused on from an image and noise is reduced based on such a patch.

For example, in the technique disclosed in Non-Patent Document 1, a patch set is selected from an captured image, a patch probability model represented by the average of the pixel values of the patch set and a covariance set is generated, and the maximum a posteriori method is used. Noise reduction processing is performed for all patches based on. Next, a denoised image can be generated by performing a synthesis process called aggregation using the patch to which the noise reduction process has been performed. Such a method is called the NL Bayes method.

Further, for example, in Patent Document 1, principal component analysis is performed on the patch set, noise reduction processing is performed on all patches in the patch set based on the analysis, and then patch synthesis is performed as in Non-Patent Document 1. A process for generating a denoised image by performing the process is disclosed.

Japanese Unexamined Patent Publication No. 2013-0266669 Implementation of the “Non-Local Bayes” (NL-Bayes) Image Denoising Algorithm, Image Processing On Line, 3 (2013), pp. 1-42.

However, the techniques disclosed in Non-Patent Document 1 and Patent Document 1 have a problem that noise of a fixed pattern may occur in the output image.

The present invention provides a technique for easily removing noise of a fixed pattern.

In order to achieve the object of the present invention, for example, one embodiment has the following configuration. That is, the detection means for detecting a plurality of second partial regions having a similar distribution of pixel values to the first partial region in the input image, and the second partial region for each of the plurality of second partial regions. The first generation means for generating a vector having the pixel value in the element is equal to the standard deviation of the element at the corresponding position in each vector generated by the first generation means. A first conversion means for converting each vector generated by the first conversion means, a plurality of eigenvalues obtained from a covariance matrix based on each vector converted by the first conversion means, and the first conversion. A first correction means that calculates the result of correcting each of the plurality of eigenvalues as a correction value based on the variance between the vectors converted by the means, and each correction calculated by the first correction means. Based on the value and the eigenvector obtained from the covariance matrix, the second correction means for correcting each vector converted by the first conversion means and the second correction means for correction. After the noise reduction processing for the input image based on the second conversion means that performs the inverse conversion of the conversion by the first conversion means for each vector and the vector converted by the second conversion means. It is characterized by comprising a second generation means for generating the image of.

It is possible to provide a technique that makes it easy to remove noise of a fixed pattern.

The figure which shows an example of the hardware composition of the computer apparatus which concerns on each embodiment. The figure which shows an example of the functional structure in the image processing apparatus which concerns on Embodiment 1. FIG. The flowchart of the noise removal processing which concerns on Embodiment 1. The figure which shows an example of the table used for the correction of the eigenvalue according to Embodiment 1. The figure which shows an example of the functional structure in the image processing apparatus which concerns on Embodiment 2. FIG. The flowchart of the processing example in the image processing method which concerns on Embodiment 2.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

[Embodiment 1]
The image processing apparatus according to this embodiment executes patch-based noise reduction processing. FIG. 1 is a block diagram showing an example of the hardware configuration of the image processing device according to the present embodiment. In the example of FIG. 1, the image processing device 100 has a CPU 101, a RAM 102, a storage unit 103, a general-purpose interface 104, and a display unit 105, which are connected to each other via a main path 106. Further, an external memory 110 and an input device 120 are connected to the image processing device 100 via a general-purpose interface 104.

By executing software (computer program) stored in the RAM 102, the CPU 101 controls the operation of the entire image processing device 100, and also executes or controls each process described later as what the image processing device 100 performs. The RAM 102 has an area for holding computer programs and data loaded from the storage unit 103, and a work area used by the CPU 101 to execute various processes. The storage unit 103 is a storage device such as an HDD, a ROM, or an SSD that stores various computer programs and various data. The general-purpose interface is an interface used for data communication between the external memory 110 and the input device 120. The display unit 105 is a display device that displays the processing result of the CPU 101 with images, characters, and the like. The display unit 105 may be, for example, a liquid crystal display or an organic EL display. The external memory 110 is a storage device that stores the captured image. The input device 120 is an input device that acquires user input, for example, is composed of a mouse, a keyboard, a touch panel, buttons, and the like.

Hereinafter, an example of various processes realized by the CPU 101 realizing various software (computer programs) stored in the storage unit 103 will be described. First, the CPU 101 can start the image processing program stored in the storage unit 103. Next, the CPU 101 may expand the image processing program into the RAM 102 and display the user interface screen (hereinafter referred to as the UI screen) on the display unit 105. Further, the CPU 101 can read the input image stored in the external memory 110 into the RAM 102 based on the user input via the input device 120. Further, the CPU 101 performs noise reduction processing, which will be described later, on the input image stored in the RAM 102 based on the user input. The input image that has undergone the noise reduction processing is stored in the RAM 102 again after the processing, and after being subjected to a series of processing based on the user's instruction, is displayed on the display unit 105 or sent to the external memory 110. It may be used for a predetermined output such as storage.

The device for storing various software (computer programs) and various data is not limited to the storage unit 103. For example, various software (computer programs), various data, and the like may be stored in a server or the like connected to the image processing device 100 through a network (not shown), or may be stored in the external memory 110. Further, the various data stored in this way are not particularly limited, and for example, an input image may be stored.

In the present embodiment, the image processing device 100 has a CPU 101, a RAM 102, a storage unit 103, a general-purpose interface 104, and a display unit 105 inside, and has an external memory 110 and an input device 120 outside. .. However, the configuration of the image processing device 100 is not particularly limited to this. For example, the image processing device 100 may have a display unit 105 externally. Further, the input device 120 and the display unit 105 may be integrated like a touch panel. Further, the image processing device 100 and the input device 120 may be integrated like a smartphone or a tablet terminal.

[Patch-based noise reduction processing]
First, an outline of patch-based noise reduction processing will be described. The image processing apparatus according to the present embodiment sets a partial region (hereinafter, referred to as a patch of interest) having a predetermined range in which the position is determined according to the position of the pixel of interest with respect to the input image. Next, the image processing device uses each pixel in a predetermined partial region near the pixel of interest, which is determined according to the position of the pixel of interest, as a reference pixel, and has the same size as the patch of interest, the position of which is determined according to the position of each reference pixel. Each region is set as a reference patch corresponding to a reference pixel. The reference patch is referred to by the detection unit 203 described later when performing noise reduction processing on the patch of interest. Further, the image processing device detects one or more reference patches having a high degree of similarity with the patch of interest (described later) from the set set of reference patches as similar patches, and patches the set of similar patches for one patch of interest. Let it be a set. The image processing apparatus performs noise reduction processing on each patch set for each similar patch of the patch set. Finally, the image processing apparatus generates an image in which the noise reduction processing is performed for the input image by synthesizing (aggregating) each similar patch that has been subjected to the noise reduction processing. In the following, the definitions of the patch of interest, the reference patch, the similar patch, and the patch set shall follow this. A detailed description of each process performed by the image processing apparatus according to the present embodiment will be described later using the flowchart of FIG.

FIG. 2 is a block diagram showing an example of the functional configuration of the image processing device 100 according to the present embodiment. The image processing device 100 according to the present embodiment includes an input unit 201, a setting unit 202, a detection unit 203, an acquisition unit 204, a conversion unit 205, a matrix calculation unit 206, an eigenvalue calculation unit 207, a calculation unit 208, a correction unit 209, and a reverse. It has a conversion unit 210 and a synthesis unit 211. The image processing apparatus 100 according to the present embodiment may be configured by one device as shown in FIG. 2, or may have a configuration in which a plurality of devices are connected to each other.

The input unit 201 acquires the image data of the input image. In the present embodiment, the input unit 201 acquires a monochrome input image having a pixel value of 0 to 255 (8 bits), but the format of the input image acquired by the input unit 201 is not particularly limited. The input image acquired by the input unit 201 may be, for example, a 1-bit image, or an image having a large number of bits such as 10 bits, 12 bits, 14 bits, or 16 bits. Further, the input image acquired by the input unit 201 may be a color image composed of images for each of a plurality of color components represented by a color system such as RGB, XYZ, Lab, and the like. For example, the input image acquired by the input unit 201 may be a 14-bit Bayer array image.

The setting unit 202 sets the pixel of interest and the patch of interest in the input image acquired by the input unit 201. In the present embodiment, the setting unit 202 sets an arbitrary pixel as a pixel of interest, and then sets a region having a size of 5 × 5 pixels centered on the pixel of interest as a patch of interest. In the present embodiment, all the pixels on the input image are set as the pixels of interest, but the present invention is not particularly limited to that. For example, the setting unit 202 may set the pixels of interest at predetermined intervals so that the patches of interest are arranged in a tile shape on the input image. In the present embodiment, the pixels of interest are set in raster order from the upper left pixel of the input image. That is, the pixels in the upper left corner to the rightmost pixel in the same column are set as the pixels of interest in order, then the pixels from the left end to the right end in the next lower row are set in order as the pixels of interest, and such work is performed at the lower end. It shall be repeated in order up to the line. Further, each patch to be described later including the patch of interest is expressed as a column vector in which the pixel values of the pixels included in the patch are arranged in a row (that is, in the present embodiment, 25 elements are arranged in a row). ..

Further, in the present embodiment, the patch of interest is set as a region of 5 × 5 size centered on the pixel of interest, but is not particularly limited to that. For example, a region having a size of 5 × 5 with the pixel of interest at the end (for example, the upper left corner) may be the patch of interest, or the patch of interest may not include the pixel of interest. Further, the size of the patch of interest is not limited to the 5 × 5 size, and may be appropriately set to have a desired size. Further, when the input unit 201 acquires an input image including a plurality of color components, the setting unit 202 may set different patches of interest for each color component of the pixel, and the patch of interest may be set to 5 × 5 × colors. It may be a single column vector in which several elements are arranged in a row. For example, when different patches of interest are set for each color component of a pixel, the image processing apparatus 100 according to the present embodiment performs subsequent processing separately for each color component.

The detection unit 203 sets a reference patch having a size equivalent to that of the patch of interest, which corresponds to each reference pixel included in a predetermined region in the vicinity of the pixel of interest, which is determined according to the position of each pixel of interest. Next, the detection unit 203 detects a reference patch having a pixel value distribution similar to that of the patch of interest as a similar patch from the set of the set reference patches. For example, the detection unit 203 may calculate the similarity with the patch of interest for each reference patch, and determine whether or not to use the similar patch according to the calculated similarity. The calculation process of the degree of similarity will be described later. The range of a predetermined region in the vicinity of the pixel of interest in which the reference pixel is set is not particularly limited, but for the sake of explanation below, each pixel of the region having a size of 15 × 15 centered on the pixel of interest is referred to as a reference pixel. It shall be. Further, the setting position of the reference patch based on the position of the reference pixel is not particularly limited, but in the present embodiment, it is assumed that the setting position of the reference patch for the pixel of interest is set in the same manner. Therefore, the detection unit 203 sets 225 pixels centered on the pixel of interest as reference pixels, and detects a maximum of 225 similar patches from the reference patches corresponding to those reference pixels. Further, the reference pixel and the reference patch may include the pixel of interest and the patch of interest, respectively. The detection process of similar patches will be described later.

The acquisition unit 204 estimates the standard deviation of the noise for each corresponding pixel between similar patches, and has a value obtained by multiplying the estimated value by a constant (hereinafter referred to as a scale value) as the pixel value of the pixel position. , Get the standard deviation patch. Therefore, the acquisition unit 204 according to the present embodiment acquires an average patch obtained by averaging each corresponding pixel with respect to a patch set which is a set of similar patches. Next, the acquisition unit 204 acquires the standard deviation patch based on the average patch, for example, based on the equation (3) described later. In this embodiment, the mean patch and standard deviation patch have the same size as similar patches.

The conversion unit 205 converts each pixel of each similar patch so that the standard deviation of noise for each corresponding pixel between similar patches is equal at any position. The conversion unit 205 performs the above conversion based on the average patch and the standard deviation patch. Details of the conversion process by the conversion unit 205 will be described later in step S308. Further, in the following, the standard deviation of the noise of similar patches equalized by this conversion is defined as the converted standard deviation, and the square of the converted standard deviation is defined as the post-conversion variance. Further, each similar patch converted by the conversion unit 205 is referred to as a conversion patch.

The matrix calculation unit 206 calculates the covariance matrix based on each transmutation patch. The covariance matrix calculated by the matrix calculation unit 206 shows what kind of correlation there is between a pixel having a conversion patch and another pixel. Details of the calculation by the matrix calculation unit 206 will be described later in step S309.

The eigenvalue calculation unit 207 calculates the eigenvalues and eigenvectors of the covariance matrix from the calculated covariance matrix. The eigenvector calculated by the eigenvalue calculation unit 207 is a principal component calculated from the covariance matrix based on the conversion patch, and indicates the texture component extracted from the conversion patch. Therefore, each conversion patch can be shown using the extracted texture component and the average patch. Further, the eigenvalues calculated by the eigenvalue calculation unit 207 indicate the variance of the corresponding eigenvectors. That is, it shows the variation in the degree to which such texture components appear in each conversion patch. Therefore, the larger the eigenvalue corresponding to the eigenvector, the larger the variation of the texture component indicated by the eigenvector. Further, although the details will be described later in step S311, when the variation of the texture component is extremely small, the eigenvalue corresponding to the texture component is a value of the degree of noise dispersion in the present embodiment.

The calculation unit 208 calculates a correction matrix used for correction of the conversion patch based on the eigenvalues and eigenvectors calculated by the eigenvalue calculation unit 207. The correction matrix calculated by the calculation unit 208 is used to reduce noise in the conversion patch. Therefore, the calculation unit 208 first corrects the eigenvalues. In the present embodiment, when the eigenvalues happen to be small due to noise, the calculation unit 208 corrects the eigenvalues so that the eigenvectors corresponding to the eigenvalues are used as the texture of the subject in the image after noise reduction. It can be made difficult to remain. Next, the calculation unit 208 generates a correction matrix using the corrected eigenvalues (hereinafter referred to as correction values). The detailed steps of calculating the correction value and generating the correction matrix will be described later in steps S311 and S312, respectively.

The correction unit 209 corrects each conversion patch based on the generated correction matrix. That is, the noise of each conversion patch is reduced. Further, the inverse conversion unit 210 converts each conversion patch with noise reduction based on the average patch and the standard deviation patch. In the present embodiment, the inverse conversion unit 210 performs a conversion that cancels the conversion by the conversion unit 205. Hereinafter, the conversion patch converted by the inverse conversion unit 210 will be referred to as an inverse conversion patch. The compositing unit 211 generates image data with reduced noise by synthesizing (aggregating) each inverse conversion patch converted by the inverse conversion unit 210. Examples of the correction process, the inverse transformation process, and the synthesis process will be described later in steps S313, S314, and S315 to S317, respectively.

Each functional part shown in FIG. 2 may be implemented by hardware or may be implemented by a computer program. In this embodiment, each functional unit shown in FIG. 2 is implemented by a computer program. Further, in the following, the functional unit shown in FIG. 2 will be described as the main body of the processing, but in reality, the CPU 101 executes a computer program for realizing the function of the functional unit in the CPU 101. The function is realized.

Hereinafter, the flow of noise reduction processing by the image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 3 is a flowchart showing an example of a processing procedure for performing the noise reduction processing according to the present embodiment.

In S301, the input unit 201 reads the image data of the input image. In S302, the synthesis unit 211 initializes the pixel value of each pixel in each of the numerator image and the denominator image for performing aggregation to 0. The numerator image and denominator image have the same number of pixels as the input image in both the vertical direction and the horizontal direction. In the synthesis process described later, the pixel values of each inverse conversion patch are added to the corresponding positions in the molecular image in S315. Next, in S317, the synthesis unit 211 averages the values added on the molecular image according to the number of additions at each position. Therefore, the denominator image plays a role of storing the number of times the pixel values of each inverse transformation patch are added on the numerator image at each position.

In S303, the setting unit 202 sets the pixel of interest in the input image. In the present embodiment, the setting unit 202 sets the pixels of interest in raster order from the upper left as described above. Next, the setting unit 202 selects the pixel of interest to be processed from the set pixels of interest, and sets the patch T of interest, which is a partial region corresponding to the pixel of interest. In the present embodiment, the setting unit 202 sets a region of 5 × 5 pixels centered on the pixel of interest as a patch of interest. In the following, the patch of interest is represented by a column vector T in which the pixel values of the M pixels (M = 25 in this embodiment) of the patch of interest are arranged in raster order.

In S304, the detection unit 203 sets a plurality of reference pixels for the patch T of interest from the input image, and sets a reference patch corresponding to the reference pixel. The shape of each reference patch matches the patch T of interest. Therefore, each reference patch is composed of M pixels. Hereinafter, the i-th reference patch of the plurality of reference patches, and represents a column vector R _i formed by arranging M pieces of pixel values in which the reference patch has in raster order. In the present embodiment, it is assumed that there are a total of 225 reference patches for one patch of interest (i = 1, 2, ..., 225), but the present invention is not limited to this as described above.

In S305, the detection unit 203 detects a patch having a high degree of similarity to the patch of interest as a similar patch from among the reference patches. Therefore, first, the detection unit 203 acquires the similarity with the patch of interest for each of the reference patches set in S304. In the present embodiment, the similarity is calculated by the sum of squared differences (SSD) of the pixel values at the corresponding positions of the patch of interest and the reference patch, but the method of acquiring the similarity is not particularly limited to this. .. For example, the detection unit 203 may calculate the similarity between the patch of interest and the reference patch by using the sum of the difference absolute values (SAD) of the pixel values at the corresponding positions of those patches. The detection unit 203 calculates the difference squared sum SSD _i of the i-th reference patch by the following equation (1).

Here, _Ri (j) is the pixel value of the j-th pixel in the i-th reference patch. T (j) is the pixel value of the jth pixel in the patch T of interest. In the present embodiment, the detection unit 203 calculates the sum of squared differences by cumulatively adding the squared difference of the pixel values at the corresponding positions of the patch T of interest and the reference patch. When the similarity is acquired using the difference squared sum SSD _i in this way, the smaller the value of the difference squared sum, the more _{similar the reference patch Ri} is to the patch T of interest. That is, the smaller the value of the sum of the difference squares, the higher the degree of similarity, and conversely, the larger the value of the sum of the difference squares, the lower the degree of similarity.

The detection unit 203 detects the reference patch having a high degree of similarity calculated in this way as a similar patch. For example, the detection unit 203 compares a predetermined threshold value with the above-mentioned difference square sum for each reference patch, and detects a reference patch such that the difference square sum is less than the threshold value as a similar patch. You may. The threshold value to be compared with the sum of squared differences is not particularly limited, and may be appropriately set according to desired conditions. Hereinafter, it is assumed that the detection unit 203 detects N (N ≧ 1) similar patches by such processing. That is, it is assumed that the patch set includes N similar patches. Further, those of the i-th among the detected N number of similar patches, denote the M pixel value in which the similar patches have as a column vector P _i which are arranged in raster order.

In S306, the acquisition unit 204 acquires the average patch based on the set of similar patches detected in S305. That is, the acquisition unit 204 calculates the average of the pixel values at the corresponding positions of the plurality of similar patches, and generates an average patch in which the average value is stored at each position. Therefore, the average patch has the same shape as the patch of interest and is composed of M pixels. In the following, it is assumed that the average patch is represented by a column vector Q in which the M pixel values of the average patch are arranged in raster order. In such a case, the acquisition unit 204 calculates the average patch Q by the following equation (2).

In the present embodiment, the acquisition unit 204 calculates the average patch Q based on all similar patches, but the acquisition unit 204 is not particularly limited to that. For example, the acquisition unit 204 may calculate the average patch Q using a part of similar patches (for example, selected according to the degree of similarity). In such a case, the set of similar patches used in S306 shall be used as the set of similar patches in the subsequent processing.

In S307, the acquisition unit 204 calculates the standard deviation patch S representing the scale value of the standard deviation of the noise at each of the pixel positions of the average patch. That is, the standard deviation patch S has the same shape as the patch of interest, and is composed of M pixels. In the following, it is assumed that the standard deviation patch S is represented by a column vector in which M pixel values of the standard deviation patch are arranged in raster order. The acquisition unit 204 estimates the j-th pixel value S (j) of the pixels of the standard deviation patch by the following equation (3) using the pixel value Q (j) at that position in the average patch. ..
S (j) = (α (kQ (j) + v)) ^1/2 (3)

In the equation (3), k and v are parameters for expressing the pixel value dependence of the noise of the input image (that is, the standard deviation of the noise is estimated based on these parameters). These parameters are set in advance as predetermined values according to the model of the imaging device and the sensitivity setting. Further, these parameters are measured in advance for each sensitivity and stored in the storage unit 103 as a table (not shown). Further, α is a predetermined value determined in advance, and α = 1 in this embodiment, but the standard deviation of noise at the corresponding position of the conversion patch is constant due to the conversion in Equation 4 described later. Considering that it should be, the value of α is not particularly limited.

In S308, the conversion unit 205, based on the average patch Q and the standard deviation patches S, calculates the conversion patch U _i of converting each pixel value included in each similar patch P _i. In the present embodiment, the conversion unit 205, a j-th pixel value U _i having the U _{i (j),} and calculates the converted pixel value by the following equation (4).
U _i (j) ≡ (P _i (j) -Q (j)) / S (j) (4)

Since S (j) represents the scale value of the noise standard deviation by the conversion according to the equation (4), the noise standard deviation at the corresponding position of the conversion patch becomes constant. Further, the conversion unit 205 calculates the post-conversion variance V, which is the square of the standard deviation of the noise of the conversion patch that has become constant in this way. In this embodiment, since the standard deviations are equal at each position of the conversion patch, the post-conversion variance V is the variance between the vectors representing each conversion patch. In the present embodiment, V = 1 because α = 1, that is, S (j) represents an estimated value of the standard deviation of the noise of the input image. Even when both the parameters k and v are multiplied by a constant α at the same time, the standard deviation of noise at the corresponding position of the conversion patch is constant. When the parameters are set in this way, V = α- ² .

In S309, the matrix calculation unit 206 calculates the covariance matrix C of the conversion patch based on the conversion patch converted in S308. The covariance matrix C is a square matrix in which the size of one side is equal to the number of pixels M constituting the conversion patch. In this embodiment, matrix computation unit 206 uses the conversion patch U _i, to calculate the covariance matrix C by the following equation (5). In the covariance matrix C calculated by the equation (5), the diagonal component is the dispersion of the corresponding pixel values of the conversion patch, and the off-diagonal component is a matrix showing the correlation between the pixel with the conversion patch and another pixel. Is.

In S310, the eigenvalue calculation unit 207 calculates the eigenvalue and the eigenvector of the covariance matrix C. Generally, for a square matrix, there are a plurality of eigenvalues and eigenvectors, and the number of each eigenvalue and eigenvector is equal to the size of one side of the square matrix. That is, in this embodiment, there are M eigenvalues and M eigenvectors of the covariance matrix C, respectively. Further, the number of elements of the eigenvector is equal to the number of pixels constituting the conversion patch (that is, M). Here, it is assumed that the j-th eigenvalue of the covariance matrix C is λ _j and the j-th eigenvector is E _j . The eigenvalues λ _j and E _j of the covariance matrix C are calculated by the known properties of the covariance matrix. Further, the eigenvalues λ _j and E _j of the covariance matrix C satisfy the following equation (6). Therefore, λ _j and E _j correspond to each other.
CE _j = λ _j E _j (6)

Here, the eigenvector E _j corresponds to the texture component extracted in the set of conversion patches. The eigenvalue λ _j represents the variance of the texture component represented by each eigenvector. That is, the eigenvalue λ _j corresponds to the variation between the conversion patches of the texture components represented by each eigenvector. When the texture component represented by the eigenvector is a texture due to noise, the eigenvalue corresponding to the eigenvector is a value of the degree of dispersion in the noise conversion patch on the image. Therefore, the larger the value of the eigenvalue is, the higher the possibility that the eigenvector corresponding to the eigenvalue is the texture component of the subject to be left in the image in the noise reduction processing. In S313, which will be described later, the correction unit 209 performs noise reduction processing for each conversion patch so that the texture component to be left can be left and the texture component due to noise can be removed according to the relationship between the eigenvalue and the noise dispersion value. I do.

In S311 the calculation unit 208 corrects each eigenvalue calculated in S310 based on the converted variance V calculated in S307. In the noise reduction processing in S313 described later, the correction unit 209 saves the texture component, which is an eigenvector corresponding to the eigenvalue (correction value) after the correction, in the output image after the noise reduction processing. Decide whether or not it will be done. When the correction value is about the variance V after conversion, the correction unit 209 considers the texture component represented by the corresponding eigenvector as noise and removes it from the conversion patch. On the other hand, when the correction value is separated from the variance V after conversion, the correction unit 209 considers the texture component represented by the corresponding eigenvector as the texture to be saved and saves it in the conversion patch. However, in reality, the eigenvalues of the texture components caused by noise vary in the vicinity of the converted variance V. As a result of the variation in the eigenvalues, the texture component due to noise that should be removed may be regarded as the texture to be preserved. From such a viewpoint, the calculation unit 208 corrects an eigenvalue smaller than the value calculated when the texture component should be preserved to a value of about the variance V after conversion. For example, the calculation unit 208 may correct an eigenvalue smaller than the value calculated when the texture component is to be preserved to a value randomly determined within the range of the converted variance V ± ε. It is assumed that ε in this embodiment is a sufficiently small predetermined number. According to such a correction process, it is possible to suppress the phenomenon of being stored in the image even though it is a texture component due to noise.

Hereinafter, the specific correction process in S311 will be described with reference to FIG. In the present embodiment, the calculation unit 208 a value obtained by correcting the j-th eigenvalue lambda _j, and represent the j-th correction value eta _j. The calculation unit 208 refers to the correction table based on the value of the eigenvalue and corrects the eigenvalue. FIG. 4A shows an example of such a correction table referred to by the calculation unit 208. In the example of this figure, the calculation unit 208 calculates the threshold value βV based on the converted variance V. Next, the calculation unit 208 _{corrects the eigenvalue λ j} to be about the same as the converted variance V _{when the eigenvalue λ j} is equal to or less than the threshold value. Further, when the eigenvalue λ _j is larger than the threshold value, the eigenvalue λ _j is corrected so that the correction value η _j is larger than the converted variance V. Further, when the eigenvalue λ _j is sufficiently large, the correction value η _j so corrected is approximately the same as the original eigenvalue λ _j (that is, the eigenvalue is hardly corrected).

The value of β is a predetermined parameter determined in advance, and in the present embodiment, β = 2.0, but the value of β is not particularly limited. From the viewpoint that the larger β is, the more easily the noise of the fixed pattern disappears, β may be a real number of 1 or more. Further, if β is too large, the texture that should be left is easily removed as noise, so β may be 10 or less. Further, the sections may be smoothly connected as shown in FIG. 4A until _{the eigenvalue λ j} is larger than the threshold value βV and the corrected correction value η _j roughly coincides with the eigenvalue λ _j. It may be discontinuous as shown in FIG. 4 (b). In the case of the correction method as shown in FIG. 4B, the calculation unit 208 does not refer to the correction table _{, and if λ j} > βV, the correction value is λ _j, and _{if λ j} ≦ βV, the correction value is set to λ j. Let V be the correction value. The calculation unit 208 can perform the above-mentioned processing on all the eigenvalues calculated in S310.

In S312, the calculation unit 208 calculates a correction matrix used for correction of the conversion patch based on the correction value calculated in S311 and the eigenvector calculated in S310. In the present embodiment, the calculation unit 208 uses the correction value η _i (i = 1, ... M) obtained _{by correcting a plurality of eigenvalues λ i} and the eigenvector E _{i corresponding to the eigenvalue λ i} , and uses the following equation. The correction matrix H shown in (7) is calculated. Correction matrix H shown in equation (7), by calculating the product HU _i the conversion patch U _i, a matrix capable of calculating a value that is regarded as noise included in the conversion patch U _i. HU _i is each element of the transformation patch U _i, performs a mapping to the M-dimensional space by _{(E 1 E 2 ... E M} ), performs correction using the correction value eta, back into the space before mapping The resulting vector.

Correction unit 209 in S313 calculates the correction patch _{O i} with reduced noise from the conversion patch _{U i.} Correction patch O _i has the same shape as noted patches are constituted by M pixels. In the following, correction patch O _i is assumed to be represented by a column vector obtained by arranging M pieces of pixel values included in the correction patch in raster order. Correction unit 209, for example, by using a conversion patch _{U i} and correction matrix H, the equation (8), calculates the correction patch _{O i} obtained by correcting the conversion patch _{U i.}
O _i ≡ U _i −HU _i (8)

_{As described above, HU i} , which is the second item on the right side in the equation (8), is a value regarded as noise included in the conversion patch U _i. That is, HU _i is a correction value for the conversion patch U _i. Correction unit 209 in the present embodiment, by excluding this correction value HU _i from the conversion patch U _i, to calculate the correction patch O _i with reduced noise from the conversion patch U _i.

Inverse transform unit 210 in S314, to the conversion patch _{U i} correction patch _{O i} obtained by correcting the performs conversion so as to offset the conversion performed in S308. Below in denote the patch obtained by such a conversion and inverse conversion patch J _i. Inverse transform patch J _i has the same shape as noted patches composed of M pixels. In the following, the inverse transform patch J _i is assumed to be represented by a column vector obtained by arranging M pieces of pixel values included in the inverse transform patch J _i in raster order. The inverse transform unit 210, for example, by conversion using the following equation (9), the j-th pixel of the i-th correction patch O _i, the standard deviation patch j-th pixel value S (j), and the average patch from Q (j), to calculate the inverse transform patch j-th pixel value _J i (j).
J _i (j) = S (j) · O _i (j) + Q (j) (9)

Combining unit 211 at S315, based on the inverse transform patch _{J i} calculated in S314, to calculate the molecular images and denominator image. Both the molecular image and the denominator image have the same shape as the input image, and in S317 described later, the pixel value of each pixel of the molecular image is divided by the pixel value at the same position of the denominator image. This is image data for which an output image is generated. Combining unit 211, the pixel value of each pixel position included in the inverse transform patch J _i, on the molecule images, similar patch P _i in the input image is respectively added to the detected position. Further, the synthesis unit 211 adds 1 to each position on the denominator image where a similar patch _{Pi is detected in the input image.} The synthesis unit 211 performs such an operation on all M inverse conversion patches.

In S316, the synthesis unit 211 determines whether or not there is an unprocessed patch of interest among the patches of interest set in S303. If there is an unprocessed pixel of interest, the process returns to S303, otherwise the process proceeds to S317.

In S317, the synthesis unit 211 divides the pixel value at each pixel position in the numerator image by the pixel value at the same position in the denominator image. According to such processing, even when a plurality of pixel values are added to the same pixel positions in the molecular image by the repeated processing of S303 to S315, the average value of the pixel values is generated as an output image. Can be done.

According to such a configuration, the texture accidentally generated by the noise is regarded as noise and removed by performing the noise reduction processing using the eigenvalues appropriately corrected in consideration of the brightness dependence of the noise standard deviation. Can be done. Therefore, it is possible to suppress the generation of fixed pattern noise in the output image.

The output image may be stored in the storage unit 103, displayed in the display unit 105, or stored in the external memory 110 via the general-purpose interface 104. In this way, the output destination of the output image is not limited to a specific output destination.

[Modification 1]
The acquisition unit 204 according to the first embodiment calculated each pixel value of the standard deviation patch S based on the average patch Q and the pixel value dependence of the noise represented by the equation (3). However, the method for calculating the standard deviation patch S is not particularly limited to this. In this example, the acquisition unit 204 calculates the standard deviation patch S _{i for a similar patch P i} , as shown in Eqs. (10) and (11) below, and such S _i (i = The standard deviation patch S is calculated as the average of 1 to M).

In this way, by calculating the scale value for each similar patch and using the average as the standard deviation patch S, the output image can be obtained even when there is a similar patch with strong noise such as outliers in a certain pixel. It is less susceptible to the outliers. That is, for example, when there is a similar patch having pixels that are very bright due to noise, the 0.5th power processing is performed based on the equation (10) before averaging between the similar patches. The influence of pixels having such outliers on the output image is likely to be reduced.

[Modification 2]
In S306, the acquisition unit 204 may calculate the median patch MP in which the median value of each pixel value is taken instead of the average patch Q for the patch set which is a set of similar patches. The median patch MP has the same shape as the patch of interest and is composed of M pixels. In the following, the median patch MP is represented by a column vector in which the M pixel values of the median patch MP are arranged in raster order. Then, in the first embodiment, the image processing apparatus 100 uses the median patch MP instead of the average patch Q.

Medians are generally known to be less susceptible to outliers. Therefore, for example, by performing the processing according to the first embodiment using the median patch MP instead of the average patch Q, it is possible to perform the noise reduction processing that is not easily affected by the outliers due to noise.

[Embodiment 2]
The image processing apparatus according to the second embodiment calculates the covariance matrix C in the same manner as in the first embodiment, and then uses the eigenvector group obtained by performing the principal component analysis on the covariance matrix C to reduce noise. Perform processing. That is, the eigenvectors used for the noise reduction processing are selected based on the eigenvalues of the covariance matrix C. The image processing apparatus according to the second embodiment calculates a matrix B in which eigenvectors F satisfying a predetermined condition are arranged as shown in the following equation (13) based on the eigenvalues of the covariance matrix C. Then, by correcting the transformation patch U _i by using the projection matrix W represented by the formula (14) described later, calculates a correction patch O _i, similar to the embodiment 1 on the basis of such a correction patch O _i Perform processing. The image processing apparatus 100 according to the present embodiment has the same configuration as that of the first embodiment except that it has a calculation unit 508 for such processing, and redundant description will be omitted. FIG. 5 is a block diagram for showing an example of the functional configuration of the image processing device 100 according to the present embodiment. The calculation unit 508 calculates the projection matrix W based on the eigenvalues and eigenvectors calculated by the eigenvalue calculation unit 207 and the threshold value θ. The projection matrix W will be described later in S611. Further, since the principal component analysis is a known technique, detailed description thereof will be omitted.

Hereinafter, the flow of noise reduction processing by the image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 6 is a flowchart showing an example of a processing procedure for performing the noise reduction processing according to the present embodiment. Since the image processing apparatus 100 according to the present embodiment performs the same processing step as the processing of FIG. 3 according to the first embodiment except for S611 to S613, duplicate description will be omitted.

In S611, the calculation unit 508 calculates the threshold value θ based on the converted variance calculated in S308. Based on the threshold value θ and each eigenvalue calculated in S610, the calculation unit 508 determines whether the eigenvector corresponding to each eigenvalue is treated as a vector representing the texture component to be left in the covariance matrix C or as a vector corresponding to noise. judge. That is, the calculation unit 508 according to the present embodiment treats the eigenvector whose corresponding eigenvalue is the threshold value θ or more as the texture component to be left in the covariance matrix C.

The calculation unit 508 assumes that the eigenvector whose corresponding eigenvalue is less than the threshold value θ is a vector corresponding to noise. The image processing apparatus 100 according to the present embodiment can make it difficult for fixed pattern noise to appear in the output image by performing noise reduction processing that removes a vector corresponding to such noise. From the viewpoint that the eigenvalue corresponding to the eigenvector corresponding to noise is a value of about the dispersion of noise, in the present embodiment, the calculation unit 508 uses the following equation (12) with the coefficient γ as a real number of 1 or more. The threshold value θ is calculated by.
θ = γV (12)

The coefficient γ is a parameter given in advance, and in the present embodiment, γ = 2.0, but the value of γ is not particularly limited to this. In the present embodiment, the larger the γ, the easier it is for the noise of the fixed pattern on the output image to disappear. On the other hand, the larger the γ, the easier it is for the texture that should be left on the output image to be removed as noise. Therefore, the γ may be 10 or less.

Calculator 508 in S612, based on the eigenvalues and the threshold theta, calculates the projection matrix W used for correction of the converted vector U _i. In the present embodiment, the calculation unit 508 calculates the projection matrix W based on the following equations (13) and (14).
B = (F ₁ F ₂ ... F _K ) Equation (13)
W = BB ^t (14)

_{Each of F 1 to} F _K in the equation (13) is an eigenvector whose corresponding eigenvalue is determined to be equal to or higher than the threshold value θ. In the present embodiment, it is assumed that there are K (K ≦ M) eigenvectors for which the corresponding eigenvalues are determined to be equal to or greater than the threshold value. In the present embodiment, the calculation unit 508 sets the _{eigenvectors E j whose corresponding eigenvalues λ j} are θ or more as _{F 1 to} F _K , respectively. Therefore, the matrix B is a matrix of K rows and M columns, and the projection matrix W is a square matrix of M rows and M columns. Projection matrix W by taking the product of the projection matrix W and the conversion patch U _i in equation (15) below is a matrix that leaves only the component of F ₁ to F _K from the conversion patch U _i.

Correction unit 209 in S613, based on the projection matrix W and the conversion patch _{U i,} to calculate the correction patch _{O i.} In the present embodiment, the correction unit 209 by using equation (15) below, based on a conversion patch U _i and projection matrix W, and calculates the correction patch O _i.
O _i = WU _i (15)

According to such a configuration, based on the likely component that corresponds to the noise in the correction patch O _i corrected by projection matrix W to remove from the texture component synthesized similarly inverse transform patch as in Embodiment 1 Is done. Therefore, the texture generated by chance due to noise can be regarded as noise and removed. Therefore, fixed pattern noise can be removed from the output image.

(Other embodiments)
In the above-described embodiment, software realized by operating a computer program has been described as an example, but the present invention is not particularly limited to this. For example, a part of each configuration of the block diagram shown in FIG. 2 or FIG. 5, or all of them may be realized by a dedicated image processing circuit.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

201: Input unit, 202: Setting unit, 203: Detection unit, 204: Acquisition unit, 205: Conversion unit, 206: Matrix calculation unit, 207: Eigenvalue calculation unit, 208: Calculation unit, 209: Correction unit, 210: Reverse Conversion unit, 211: Synthesis unit

Claims (20)

A detection means for detecting a plurality of second subregions having a pixel value distribution similar to that of the first subregion in the input image, and
For each of the plurality of second subregions, a first generation means for generating a vector having a pixel value in the second subregion as an element, and
With the first conversion means that transforms each vector generated by the first generation means so that the standard deviations of the elements at the corresponding positions in each vector generated by the first generation means are equal. ,
Based on the plurality of eigenvalues obtained from the covariance matrix based on each vector converted by the first conversion means and the variance between the vectors converted by the first conversion means, the plurality of eigenvalues The first correction means for calculating the result of each correction as a correction value, and
A second correction means for correcting each vector converted by the first conversion means based on each correction value calculated by the first correction means and an eigenvector obtained from the covariance matrix. When,
A second conversion means that performs the inverse transformation of the conversion by the first conversion means for each vector corrected by the second correction means, and
A second generation means for generating an image after noise reduction processing on the input image based on the vector converted by the second conversion means, and
An image processing apparatus comprising. The second generation means uses the vector converted by the second conversion means to perform noise reduction processing on the input image, which corresponds to the position of a plurality of the second partial regions in the input image. The image processing apparatus according to claim 1, further comprising calculating a pixel value of a position in an image. The second generation means performs noise reduction processing on the input image in which pixel values, which are elements of the vector converted by the second conversion means, correspond to a plurality of the second partial regions in the input image. Add to each position in the later image,
Each of the added pixel values is calculated as a value to be output as a result of dividing by the number of additions for each position.
The image processing apparatus according to claim 1 or 2, wherein each of the output values is a pixel value at each position in the image after noise reduction processing for the input image. The variance between the vectors converted by the first conversion means is a predetermined value set based on a predetermined parameter given in advance for the image pickup apparatus for capturing the input image. , The image processing apparatus according to any one of claims 1 to 3. The first conversion means estimates the standard deviation based on the respective vectors generated by the first generation means and the predetermined parameters.
The image processing apparatus according to claim 4, wherein a vector having a value obtained by multiplying the estimated standard deviation by a constant as an element of each position is calculated. The first conversion means is based on a vector having the average value of the elements at the corresponding positions of each vector generated by the first generation means as the elements at the corresponding positions, and the predetermined parameters. The image processing apparatus according to claim 4 or 5, wherein a vector having a value obtained by multiplying the estimated standard deviation by a constant value as an element of each position is calculated. The first conversion means has a standard deviation of elements at each position of the vector generated by the first generation means from each vector generated by the first generation means based on the predetermined parameter. Guess each
For each vector having a value obtained by multiplying the standard deviation estimated from each vector generated by the first generation means by a constant as an element of the corresponding position, the average value of the elements of the corresponding position is the element of each position. The image processing apparatus according to claim 4 or 5, wherein a vector having a value obtained by multiplying the estimated standard deviation by a constant is calculated as an element of each position. The first conversion means estimates from a vector having the average value of the elements at the corresponding positions of each vector generated by the first generation means as the elements at the corresponding positions and the elements at the corresponding positions. The claim is characterized in that each vector generated by the first generation means is converted based on a vector having a value obtained by multiplying the standard deviation by a constant as an element of the corresponding position. The image processing apparatus according to 6 or 7. The first conversion means is based on a vector having the median value of the element at the corresponding position of each vector generated by the first generation means as the element at the corresponding position, and the predetermined parameter. The image processing apparatus according to claim 4 or 5, further comprising calculating a vector having a value obtained by multiplying the estimated standard deviation by a constant as an element at each position. The first conversion means has a standard deviation of elements at each position of the vector generated by the first generation means from each vector generated by the first generation means based on the predetermined parameter. Guess each
For each vector having a value obtained by multiplying the standard deviation estimated from each vector generated by the first generation means by a constant, as an element of the corresponding position, the median value of the element of the corresponding position is the element of each position. The image processing apparatus according to claim 4 or 5, wherein a vector having a value obtained by multiplying the estimated standard deviation by a constant is calculated as an element of each position. The first conversion means estimates from a vector having the median value of the element at the corresponding position as the element at the corresponding position of each vector generated by the first generation means and the element at the corresponding position. The claim is characterized in that each vector generated by the first generation means is converted based on a vector having a value obtained by multiplying the standard deviation by a constant as an element of the corresponding position. The image processing apparatus according to 9 or 10. The first correction means converts the eigenvalue into the first conversion when the eigenvalue is equal to or less than a predetermined threshold value calculated based on the variance between the vectors converted by the first conversion means. The image processing apparatus according to any one of claims 1 to 11, wherein the image processing apparatus is corrected to a variance between vectors converted by means. When the eigenvalue is larger than a predetermined threshold value calculated based on the variance between the vectors converted by the first conversion means, the first correction means converts the eigenvalue into the first conversion means. The image processing apparatus according to any one of claims 1 to 12, wherein the image processing apparatus is corrected to a value larger than the variance between the vectors converted by the above and smaller than the eigenvalue. The first correction means does not correct the eigenvalue when the eigenvalue is larger than a predetermined threshold value calculated based on the variance between the vectors converted by the first conversion means. The image processing apparatus according to any one of claims 1 to 12, wherein the image processing apparatus is characterized by the above. The second correction means calculates a matrix used for correction of each vector converted by the first conversion means based on each eigenvalue corrected by the first correction means and the eigenvector. And
The image processing apparatus according to any one of claims 1 to 14, wherein each vector converted by the first conversion means is corrected based on the matrix. The second correction means is based on the eigenvalues corrected by the first correction means and the variance between the vectors converted by the first conversion means, and from the eigenvectors, the first correction means. The image processing apparatus according to any one of claims 1 to 15, wherein an eigenvector used for correction of each vector converted by the conversion means is selected. The image processing apparatus according to claim 16, wherein each vector converted by the first conversion means is corrected based on the elements of the selected eigenvector. Further, a setting means for setting a plurality of third subregions having the same size as the first subregion in the input image is provided.
A plurality of detection means are used based on the similarity of the corresponding elements of the vector having the pixel value in the first partial region as an element and the vector having the pixel value in the third partial region as an element. The image processing apparatus according to any one of claims 1 to 17, wherein the second partial region is selected from the third partial region of the above. This is an image processing method performed by an image processing device.
A step in which the detection means of the image processing device detects a plurality of second subregions having a pixel value distribution similar to that of the first subregion in the input image.
A step in which the first generation means of the image processing apparatus generates a vector having a pixel value in the second partial region as an element for each of the plurality of second partial regions.
A step of converting each of the generated vectors so that the first conversion means of the image processing apparatus makes the standard deviations of the elements at the corresponding positions in each of the generated vectors equal.
The first correction means of the image processing apparatus determines the plurality of eigenvalues obtained from the covariance matrix based on the transformed vectors, and the variances between the transformed vectors, based on the plurality of eigenvalues. The process of calculating the result of each correction as a correction value, and
The second correction means of the image processing apparatus corrects each of the converted vectors based on the respective correction values calculated by the first correction means and the eigenvectors obtained from the covariance matrix. And the process to do
A step in which the second conversion means of the image processing apparatus performs the inverse transformation of the conversion on each of the corrected vectors.
A step in which the second generation means of the image processing apparatus generates an image after noise reduction processing on the input image based on the converted vector.
An image processing method comprising. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 18.

Images