JP5482007B2 - Image processing method - Google Patents

Description

  The present invention relates to image noise removal processing and edge enhancement processing.

  There is Patent Document 1 as a conventional technique for noise removal and edge enhancement using multi-resolution. In Patent Document 1, in the case of multi-resolution conversion, for example, wavelet conversion, a noise component and an edge component are extracted from a high-frequency subband image composed of LH, HL, and HH, and these are integrated into a multi-resolution to integrate an integrated noise component and an integrated edge component. Is generated, noise is removed by subtracting the noise component from the original image, and edge enhancement is performed by adding the edge component. In this appearance, noise components projected in the frequency space consisting only of the high-frequency subbands LH, HL, and HH, which form a complete system, can generate a frequency gap band in combination with a noise removal filter. The extraction omission is easy to occur.

  Therefore, in Patent Document 2 by the inventor of the present application, in order to prevent them, noise extraction is performed in a redundant frequency space combined with the projection of the sequentially generated low frequency subband LL onto the frequency space. A technique for preventing noise extraction leakage by integrating both noise components of the frequency subband and the high frequency subband is disclosed. At that time, in order to minimize the harmful effects of image structure destruction due to noise removal, paying attention to the difference in the properties of the frequency distribution of each surface of the luminance component and color difference component, the luminance component strongly denoises the high frequency subband, The low frequency sub-band is weakly noise-removed, and the color difference component should be reversed.

  In Patent Document 3 by the inventor of the present application, a weighting coefficient k between L / H (0 ≦ k) at the time of integration of noise components giving the strength of noise removal between the low frequency (L) and high frequency (H) subbands. ≦ 1) is disclosed to set k: 1: 1: 1 among LL, LH, HL, and HH, for example, in the case of a luminance component. The value of k greatly affects the appearance of the noise removal result because the frequency characteristics of noise can be changed freely. Since the degree of the noise removal effect varies depending on the preference of the person, an example of a strategy for opening this parameter through the user interface is shown. In the case of edge enhancement, the edge components extracted from both the low-frequency subband and the high-frequency subband can be integrated in the same way as noise reduction, and the weighting coefficient between the L / H bands at the time of integration can be changed freely. I am doing so.

U.S. Patent No. 6,754,398 International Publication No. 2007/116543 Pamphlet International Publication No. 2007/114363 Pamphlet

  However, in the methods of Patent Literature 2 and Patent Literature 3, the noise removal parameter is applied when a conventional noise removal filter is applied in addition to the parameter that gives the frequency mixing degree between L / H newly added as described above. There are a noise removal strength parameter that gives a fluctuation width threshold for discriminating between the noise and the edge structure, and a noise removal rate parameter that determines how many percent the extracted noise component is actually subtracted from the original image. Therefore, there are three adjustment parameters for the user, and there is a risk that it may be difficult to understand what is to be moved and how to obtain the best target level image. The same applies to edge enhancement.

The invention according to claim 1 is applied to an image processing method, inputs an original image composed of a plurality of pixels, decomposes the input original image, and is a low-frequency image with the lowest resolution capable of completely reconstructing the original image In order to express the frequency space more redundant than the equivalent representation of the original image using both a set consisting of a series of high-frequency images with multiple resolutions and a set consisting of a series of low-frequency images with multiple resolutions. Multiple low-frequency images with low resolution and multiple high-frequency images with low resolution sequentially are generated, and edge components are extracted by applying an edge extraction filter to each of the low-frequency image and high-frequency image, and corresponding to each The low frequency edge component and the high frequency edge component are generated, and the weight between the frequency bands of the edge component is modulated by multiplying at least one of the generated low frequency edge component and high frequency edge component by a weighting coefficient. Sequentially into one edge components by combining the low-frequency edge component and the high-frequency edge component having undergone modulation with successively higher resolution, multiplied by the edge enhancement rate set by the user arbitrarily to integrated edge component By adjusting the intensity and adding the adjusted edge component to the original image, the edge enhancement of the original image is performed, and the value of the weighting coefficient that modulates the weight between the frequency bands of the edge components is used as the edge enhancement ratio. It is characterized by changing according to the above.
The invention of claim 5 is applied to an image processing method, inputs an original image composed of a plurality of pixels, decomposes the input original image, and is a low-frequency image with the lowest resolution capable of completely reconstructing the original image In order to express the frequency space more redundant than the equivalent representation of the original image using both a set consisting of a series of high-frequency images with multiple resolutions and a set consisting of a series of low-frequency images with multiple resolutions. a plurality of low-frequency image having a lower resolution to produce a plurality of high-frequency images with successively lower resolution, by extracting the noise components included in each of the low-frequency image and the high-frequency image, the low frequency corresponding to each A noise component and a high frequency noise component are generated, and a weighting factor is applied to at least one of the generated low frequency noise component and high frequency noise component to modulate the weight between the frequency bands of the noise component. Sequentially into one noise component having sequential high resolution by combining frequency noise component and the high frequency noise components, the intensity attenuated by multiplying the noise removal rate set by the user arbitrarily to integrated noise component The noise component is subtracted from the original image by subtracting the attenuated noise component, and the weighting coefficient value for modulating the weight between the frequency bands of the noise component is changed according to the noise removal rate. It is what.

  Since the present invention is configured as described above, processing with consideration of the difference in image quality effect between the low-frequency image and the high-frequency image is performed in noise removal or edge enhancement with a simple configuration, and an image with optimum image quality is provided. be able to.

It is a figure which shows the image processing apparatus which is embodiment of this invention. It is a figure which shows the structure of a personal computer. It is a figure which shows the flowchart of the image processing of 1st Embodiment which the personal computer 1 processes. It is a figure which shows the flowchart of the noise removal process and edge emphasis process based on the multi-resolution of 1st Embodiment. It is an image figure explaining edge refining of a 1st embodiment. It is a figure which shows the nonlinear function which has a strong damping characteristic. (A) is a figure which shows the gamma characteristic in output color space and work color space, (b) shows the differential ratio of the gamma characteristic of output color space shown in (a), and the gamma characteristic in work color space. It is a graph. It is a figure which shows the flowchart of the noise removal process of 2nd Embodiment, and an edge emphasis process. 1 is a diagram illustrating a configuration of a digital camera 100. FIG. It is a figure which shows the relationship between a signal and noise. It is a figure which shows the mode of the frequency space in multi-resolution. The enhancement effect is easy to see in the table.

-Basic idea-
First, the background and reason for the necessity of adopting the algorithm described in the embodiment and the basic idea of a method for dealing with it will be described.

<Problems of noise removal processing>
A noise removal filter called an edge-preserving smoothing filter is generally used for the noise removal processing, which distinguishes between an edge structure having a large signal level difference and noise having a small amplitude and performs smoothing adaptively. A typical example of this noise removal filter is a so-called bilateral filter. These noise reduction filters basically compare the level difference of the pixel value between adjacent pixels with the noise fluctuation index value, and determine the weighting coefficient for determining whether or not to make the pixel to be smoothed based on the ratio. go.

  However, no matter how high the noise removal filter is, a weak edge comparable to noise cannot be distinguished from noise (see FIG. 10D). As a result, the texture disappears due to noise removal (Problem 1). In addition, when there is an edge of a steep step in the process of determining the smoothing weighting coefficient (see FIG. 10C), in a normal flat part or a monotonous inclined part (see FIGS. 10A and 10B). Unlike the weighting coefficient that separates the edge and noise, the contour component is mistaken for a relatively small edge due to the effect of the step, and falls into the same situation as in Problem 1 and is easily included in the smoothing target, and the contour is distorted. An inevitable problem arises. Therefore, so-called outline blurring occurs due to noise removal (Problem 2). FIG. 10 is a diagram showing the above situation.

  These problems mainly describe the phenomenon from the viewpoint of the noise removal processing of the luminance component. However, in the case of the noise removal processing of the color difference component, these are the color loss phenomenon (problem 3) or the color boundary of the weak color change portion. This causes the problem of color bleeding phenomenon (problem 4). Furthermore, with the smoothing process by noise removal, all the local gradations are gathered at the average level, so for example, it becomes impossible to reach a completely dark state where the average value becomes zero, and the black level floats. This also causes a problem of so-called black floating phenomenon (Problem 5). This can also be said for the white side. As a whole, the range in which gradation can be expressed by the smoothing process is narrowed, and gradation is lowered.

  That is, the problem of noise removal is summarized as follows in terms of subjective relation and physical relation. (1) Loss of sharpness and stereoscopic effect due to a decrease in contour contrast due to contour blur (Problem 2), (2) Loss of weak edge structure buried in noise reduces resolution (Problem 1), (3) Black It can be said that the gradation is lowered by the floating phenomenon (problem 5), (4) the color reproducibility is lowered by the loss of color (problem 3), and (5) the color resolving power is lowered by the bleeding of the color (problem 4). ).

  In addition, these problems are common to high-sensitivity and noisy images even without noise removal processing. This is because, when noise with large amplitude is added to a clear image with low sensitivity, the contrast of the contour of the actual subject is relatively lowered due to the influence of the amplitude of the noise, and is less sharp than with low sensitivity. Visible, texture is indistinguishable from noise, noise amplitude raises the black level itself, so it floats black, and uncorrelated fluctuation between RGB rides as color spot noise, lowering the color discrimination resolution of that area As a result, the color identification accuracy (color reproducibility) is lowered, and the color noise appears to cause color blur at the color boundary portion at random. These can also be referred to generically as a reduction in contrast due to noise coverage.

<Consideration of simultaneous noise removal and edge enhancement>
To deal with such problems, when noise removal is involved, edge enhancement that restores the contrast of the edge lost by noise removal, tone correction processing to restore tone reproducibility, or contrast correction Processing is required. Further, with respect to a high-sensitivity photographed image, it is possible to provide a sharp image that is much closer to a low-sensitivity photographed image by simply performing processing corresponding to these even if noise removal is not performed. Further, by correcting the influence of contrast reduction caused by noise covering on a low-sensitivity photographed image, the original sharpness, resolution, gradation, and color reproducibility can be realized.

  Therefore, edge enhancement processing using multi-resolution as shown in International Publication No. 2007/114363 pamphlet and International Publication No. 2007/116543 pamphlet is performed by the present inventors. This edge enhancement process is essentially different from the conventional edge enhancement process using multi-resolution (for example, US Pat. No. 6,754,398). The edge extracted by projecting to the frequency space of the multi-resolution high-frequency subband group forming the complete system. In addition to the components, the edge components extracted by projecting onto the frequency space of the low-frequency Cebu band group that is sequentially generated with another redundancy are also used to freely mix them. FIG. 11 is a diagram showing a state of the frequency space in the multi-resolution. The complete system is used in the sense that the original image can be completely reconstructed from the image group in the frequency projection space. Actually, it is necessary to add only one low-frequency subband with the lowest resolution, but this is ignored here.

  However, since these practical physical effects have not been elucidated in detail, they have not yet come to present a methodology for further enhancing the image enhancement processing effect. Therefore, in the present invention, after clarifying these physical effects, the best measures for giving the most natural impression as the edge emphasis on the problems and countermeasures that emerge from them are examined.

<Elucidation of physical effects of multi-resolution edge enhancement processing>
First, the physical effect of edge enhancement by redundant multi-resolution conversion is elucidated. Experimentally, the edge enhancement effect in the frequency projection space expressed by two redundant multi-resolutions of the low-frequency subband group and the high-frequency subband group was confirmed, and extracted in each of the low-frequency subband group and the high-frequency subband group. It was found that the obtained edge component has the following excellent individual image enhancement effects.

  When the input image is a luminance component, edge enhancement projected on the frequency space of the low-frequency subband group has a contrast enhancement effect that enhances blackness and gradation. In other words, the same effect as the tone correction processing can be obtained from the multi-resolution low-frequency subband group of the luminance component even in the edge enhancement processing. The edge enhancement process projected onto the frequency space of one high-frequency subband group of luminance components has a sharpness restoration effect that restores the texture buried in the noise by raising the contrast of the contour.

  Next, when the input image is a color difference component, edge enhancement projected onto the frequency space of the low frequency subband group has a colorfulness recovery effect that extremely increases the saturation of a flat area of a large area. In other words, even in the case of the color difference component, the same effect as the saturation enhancement process can be obtained even though the edge enhancement process is performed. The edge enhancement process projected on the frequency space of the high-frequency subband group of one color difference component has a color contrast restoration effect that reduces the color blur at the color boundary and sharpens the color boundary or restores the color structure of the color texture. There is.

  FIG. 12 is a table showing these enhancement effects in an easy-to-see manner. At first glance, the term “contrast” on the luminance plane and the color difference plane is shifted to the low frequency side in the case of the luminance component, and shifted to the high frequency side in the case of the color difference component. This is considered to be because the frequency characteristics of the image structures of the original luminance plane and the color difference plane are different. That is, since the luminance plane has many high-frequency structures and the color difference plane has many low-frequency structures, the edge enhancement effect of the luminance component projected on the low-frequency subband group is the same as that of the color difference component projected on the high-frequency subband group. This is probably because the edge enhancement effect is closer.

  Experimentally, the image obtained by adding only the noise removal processing described in the first embodiment to the original image and the edge component extracted on each frequency projection surface of the first embodiment are simply subjected to multi-resolution inverse transformation. The integrated edge component was generated to obtain an image showing the edge enhancement effect obtained by adding the integrated edge component to the noise-removed image. As a result, in the luminance component, when the edge enhancement by the low-frequency subband group is actually strong, the black tightening effect has been achieved so as to be considered abnormal compared to the noise-removed image. Also, the edge enhancement by the high frequency sub-band group successfully recovered the texture structure such as the black and white spots of the background and the fur of animal dolls that disappeared in the noise-removed image. The effect shown in FIG. 12 was achieved in the same manner for the color difference component.

<Optimization of multi-resolution edge enhancement processing>
It can be understood that such individual physical effects are clarified. In order to prepare the frequency of the edge component extracted in the low frequency subband group and the edge component extracted in the high frequency subband group, some kind of law is provided. If the constraint is not applied, the original image may produce an unnatural edge enhancement effect that cannot be seen far away from the original image. Further, when noise removal is involved, the degree of subjective damage due to the noise removal effect varies depending on the strength of noise removal. Therefore, it is necessary to attempt recovery according to the degree of damage. As a solution to this problem, in the embodiment, a policy linked to the edge enhancement rate and the noise removal rate is shown.

<Problems of multi-resolution edge enhancement processing>
Next, as shown in the image examples acquired in the above-described experiment, edge enhancement by multi-resolution involves a risk of causing a large area halo or howling. This is the same as the ringing problem near the edge inherent in normal unsharp mask processing, but the cause of the problem is increased due to the multi-resolution processing.

  To deal with this ringing problem in the past, the relationship between the input edge component and the output edge component of the extracted edge component is directly proportional to the vicinity of the origin, and the others gradually increase monotonously. Ingenuity has been devised to suppress through non-linear transformation of characteristics. However, this is still inadequate for pursuing higher quality multi-resolution edge enhancement processing than ever before. In other words, in order to bring edge enhancement by multi-resolution to a practical level, it is necessary to introduce a measure for completely closing the halo and howling problem. Therefore, in the present invention, in order to reinforce these measures, a hypothesis is introduced about the ideal appearance of an edge component that has no unnaturalness in the result of the edge enhancement processing, and a solution is made so as to approach it. The method is shown.

<Edge enhancement processing issues>
By the way, when performing edge emphasis, the point to be noted other than the ringing problem as described above is that the risk of amplifying noise components due to edge emphasis must be avoided as much as possible. In the noise component extraction process in the noise removal process and the edge component extraction process in the edge enhancement process, the problem of mutual mixing of the edge component into the noise component and the edge component into the noise component must be avoided. There is no such problem. This can be schematically expressed as follows.
N _extracted = N _true + e _{undistinguished}
E _extracted = E _true + n _{undistinguished}

  In this embodiment, as shown in the flowchart of the second embodiment, an effort is made to prevent the noise component from being mixed as much as possible by extracting the edge component from the image after noise removal, but it is still included. It is a reality. As described in the problem of the noise removal filter, the edge component that cannot be distinguished by the noise removal filter is always mixed in the noise component. The edge component mixed in the noise component causes the problem described in the problem of noise removal, and the noise component mixed in the edge component causes the problem of noise amplification in edge enhancement. As a result, noise removal with low texture reproducibility and edge enhancement with low texture reproducibility are performed.

<Improved edge enhancement processing>
Therefore, in the present embodiment, based on the above-described hypothesis of the ideal shape of the edge component, impurities are extracted from the edge component by approaching it and the purity is increased, and the edge itself is subjected to a self-refining process. Similarly, it is possible to make a hypothesis of how the noise component should be for the noise component, and based on the hypothesis, impurities are extracted from the noise component to increase the purity, and the noise itself is subjected to a self-refining process. In the present embodiment, the process for improving the purity by removing impurities of each component is called refining.

  However, the reality is that it is still difficult to separate impurities even after such operations. In other words, even if most of the edge components are actually noise components, they cannot be distinguished if they behave as they should be. The same applies to the edge component in the noise component. For such problems, in addition to estimating the amount of large impurity components contained in them by introducing the following hypothesis and referring to the mutual size of the noise component and the edge component: There is no way to deal with it.

  That is, in a region where the local edge component has a large value, it is highly likely that the edge component is erroneously extracted as a noise component in the noise extraction process, and most of the noise component is an edge component. Conversely, in a region where the local noise component has a large value, it is highly likely that the noise component is erroneously extracted as an edge component in the edge extraction process, and most of the edge component is a noise component. Therefore, it is possible to increase mutual purity by performing mutual refining between the edge component and the noise component with reference to the mutual size. For that purpose, the purity of the noise component and the edge component needs to be improved to a considerable degree of accuracy.

  The state of self-refining and mutual refining of edge components and noise components is typically shown in the flowchart of the second embodiment. In the process of extracting and integrating edge components and noise components using multi-resolution according to the first embodiment, this self and mutual refining is not limited to once at each resolution level, but for the integrated edge components and noise components. Even so, by repetitively verifying the self-smelting and the mutual refining, a contrivance is made to ensure the purity of each of them and at the same time to eliminate the howling component from the edge component.

<High performance edge enhancement processing>
However, there are always intermixed components that cannot be completely excluded even if the purity of the edge component and the noise component is increased in this way, and the influence cannot be ignored. That is, these still cause the above-mentioned problems such as the edge amplification noise amplification problem and the noise removal edge rounding problem.

  Therefore, in the present embodiment, the output color space is predicted for the final edge enhancement rate and the noise removal rate with respect to the final edge enhancement rate and the noise removal rate by predicting those influences in advance, minimizing adverse effects, and maximizing the effects of edge enhancement and noise removal. This is realized by introducing a method of expressing a functional (a function having a function as a variable) with a contrast ratio function with respect to the brightness expressed by the mutual differential ratio of the gradation characteristics between the color space and the working color space. .

This is shown in the flowchart of the second embodiment. That is, when expressed by an expression, this corresponds to changing the first expression to the second expression. Here, λ represents the noise removal rate, ζ represents the edge enhancement rate, γ represents the tone curve γ (Y) of the output color space for the linear tone Y, and Γ represents the working color space for the linear tone Y. It is assumed that the gradation curve Γ (Y) is represented.

  This method of functional representation of the contrast ratio includes a method of increasing or decreasing the edge enhancement rate and noise removal rate with respect to a certain reference point between the brightness levels of the entire image, and a local brightness level near the edge. There are two ways to increase or decrease them only between the two. In the present embodiment, both are used. These two concepts are based on a general tone correction technique in which only the brightness of the entire image is adjusted by changing the tone curve curve on average (also referred to as a histogram equalization method) and a local edge structure. It corresponds to each of the methods called Retinex processing that emphasizes the contrast around and allows the magnitude relationship of the brightness level to be exchanged with other regions. If the former is called gamma adjustment processing and the latter is called Retinex processing in tone correction processing, it can be said that a gamma adjustment version and Retinex version for edge enhancement or noise removal are constructed.

  That is, in the functional expression of the contrast ratio of the edge enhancement rate introduced in this embodiment, the increase / decrease width characteristic of the edge enhancement rate with respect to the brightness level is uniform in the entire image in the case of the gamma adjustment version. On the other hand, in the case of the Retinex version, they are non-uniform in the whole image, but are increased or decreased according to one rule uniformly in the vicinity of a local edge. The functional representation of the edge enhancement rate is the same functional representation as the noise removal rate in the sense that when noise removal and edge enhancement are performed at the same time, the edge components lost in noise removal are reproduced in exactly the same way. I came to the conclusion that it should be done.

-First embodiment-(Multi-resolution version)
An embodiment in which noise removal and edge enhancement are simultaneously performed using multi-resolution will be described. FIG. 1 is a diagram showing an image processing apparatus according to an embodiment of the present invention. The image processing apparatus is realized by the personal computer 1. The personal computer 1 is connected to a digital camera 2, a recording medium 3 such as a CD-ROM, another computer 4, etc., and receives various image data. The personal computer 1 performs image processing described below on the provided image data. The computer 4 is connected via the Internet and other telecommunication lines 5.

  A program executed by the personal computer 1 for image processing is provided from a recording medium such as a CD-ROM or another computer via the Internet or other electric communication line, as in the configuration of FIG. 1 is installed. FIG. 2 is a diagram illustrating a configuration of the personal computer 1. The personal computer 1 includes a CPU 11, a memory 12, a peripheral circuit 13 and the like, and executes a program in which the CPU 11 is installed.

  When the program is provided via the Internet or another telecommunication line, the program is transmitted after being converted into a signal on a carrier wave carrying a telecommunication line, that is, a transmission medium. As described above, the program is supplied as a computer-readable computer program product in various forms such as a recording medium and a carrier wave.

  Hereinafter, image processing executed by the personal computer 1 will be described. FIG. 3 is a diagram illustrating a flowchart of image processing according to the first embodiment processed by the personal computer 1. In step S1, image data is input. In step S2, conversion to a uniform color / uniform noise space is performed. In step S3, noise removal processing and edge enhancement processing are performed. In step S4, the color space is inversely converted to the output color space. In step S5, the processed image data is output. Hereinafter, the details of the processing of each step will be described.

1. Color Space Conversion When image data (hereinafter simply referred to as an image) is input in step S1, in step S2, the input image is first subjected to color space conversion and re-projected to an image processing space suitable for noise removal processing. As this image processing space, a uniform color / uniform noise space described in International Publication No. 2006/064913 pamphlet (the same inventor as the present applicant) is used. Usually, an input image is often expressed in a standard color space such as sRGB. Here, an sRGB image subjected to color correction processing and gamma correction processing will be described as a main example.

1-1. Inverse gamma correction
Remove the gamma characteristics specified by sRGB or the gamma characteristics used by each camera manufacturer to create a unique picture, and restore the original linear gradation.

  In addition, linear tone R, G, B signals are input directly after demosaic processing is performed on an image signal such as a Bayer array with a color filter having spectral sensitivity distribution characteristics in the image sensor. May be.

1-2. Conversion from RGB color space to XYZ color space When converting from an sRGB image returned to linear gradation to XYZ space, the following conversion is performed.

  In the case of RGB signals having sensor spectral sensitivity distribution characteristics immediately after demosaic processing, a matrix is formed in accordance with each spectral sensitivity distribution characteristic and converted into a device-independent XYZ space.

1-3. Conversion from XYZ color space to uniform color / noisy color space (L ^ a ^ b ^) Non-linear gradation L ^ a representing the perceptual attribute of pseudo uniform color distribution from XYZ space by the following formula ^ b ^ Convert to space. The L ^ a ^ b ^ space defined here is a modified version of the conventional so-called uniform color space L * a * b * in consideration of uniform noise. It is named b ^.

を用いる。 Here, as a gradation characteristic that usually makes uniform color and uniform noise

Is used.

  X0, Y0, and Z0 are values determined by the illumination light. For example, in the case of a two-degree field under the standard light D65, X0 = 95.045, Y0 = 100.00, and Z0 = 108.892. The value of ε varies depending on the sensor, but takes a value close to 0 when the sensitivity setting is low and takes a value of about 0.25 when the sensitivity setting is high.

  Hereinafter, the noise removal processing and edge enhancement processing in step S3 will be described.

ただし、S(x,y)はL^, a^, b^の各面に対してサブバンド画像Vij(x,y)を生成する。 2. Multiresolution Representation of Image FIG. 4 shows a flowchart of noise removal processing and edge enhancement processing based on multiresolution. The original image of each of the luminance component L ^ and the color difference components a ^, b ^ is expressed in multiple resolutions to remove noise. The original image is sequentially decomposed into low-resolution images by an analysis process of wavelet transform (processing (1-0) (2-0) (3-0) (4-0) (5-0)). The analysis process in which the subband image is generated is summarized as follows:

However, S (x, y) generates a subband image Vij (x, y) for each surface of L ^, a ^, b ^.

  The wavelet transform is to convert an image (image data) into frequency components, and divides the frequency components of the image into a high pass component and a low pass component. Data consisting of a high-pass component is called a high-frequency subband, data consisting of a low-pass component is called a low-frequency subband, LL is called a low-frequency subband, and LH, HL, and HH are called high-frequency subbands. The low frequency subband may be referred to as a low frequency image, and the high frequency subband may be referred to as a high frequency image. Furthermore, each subband may be referred to as a frequency band limited image. The low frequency subband is an image in which the frequency band of the original image is band limited to the low frequency side, and the high frequency subband is an image in which the frequency band of the original image is band limited to the high frequency side.

  Normal multi-resolution conversion only leaves the high-frequency subbands obtained by sequentially decomposing the low-frequency subband LL components, but here the low-frequency side so that there is no extraction leakage between the subband frequency bands of the noise components. Both subband LL and high frequency subbands LH, HL and HH are used.

  As the wavelet transform, for example, the following 5/3 filter is used.

<Wavelet transform: Analysis / Decomposition process>
High-pass component: d [n] = x [2n + 1]-(x [2n + 2] + x [2n]) / 2
Low-pass component: s [n] = x [2n] + (d [n] + d [n-1]) / 4
The one-dimensional wavelet transform defined above is subjected to wavelet decomposition by performing two-dimensional separation filter processing independently in the horizontal and vertical directions. The coefficient s is collected on the L plane, and the coefficient d is collected on the H plane.

<Inverse wavelet transform: Synthesis / Reconstruction process>
x [2n] = s [n]-(d [n] + d [n-1]) / 4
X [2n + 1] = d [n] + (x [2n + 2] + x [2n]) / 2
However, as shown in FIG. 4, a signal representing an image is input to the value of x at the time of wavelet transform, noise components included in the generated wavelet transform coefficients s and d are extracted, and the extracted noise components are Substitute for s and d at the time of inverse wavelet to generate and use noise image x. The same method is used for the edge component.

  The multi-resolution expression uses a 5-stage wavelet transform, but may be increased or decreased according to the size of the input original image. Further, the multi-resolution expression method is not limited to the orthogonal wavelet transform as described above, and a Laplacian pyramid expression, a steerable pyramid expression, or the like may be used.

ここで、i,jはサブバンド特定記号を表す。iは解像度の違いを、jはLL,LH,HL,HHの違いを表す。 3. 3. Noise extraction processing by virtual noise removal 3-1. Noise extraction processing by noise removal filter 3-1-1. Noise removal processing Arbitrary noise removal filters may be used. Here, the following formula described in pamphlet of International Publication No. 2006/068025 (the same inventor as the present application), which is an improvement of a generally known high-performance bilateral filter, is used. (2) (2-2) (2-2) (3-2) (4-2) (5-2)).

Here, i and j represent subband identification symbols. i represents the difference in resolution, and j represents the difference between LL, LH, HL, and HH.

  The threshold σth ij is set according to the expected noise fluctuation width for each subband, and noise components are extracted while distinguishing edges and noise. Once the noise fluctuation index value σth in the real space is determined, if the propagation amount of fluctuation to each subband signal is evaluated based on the error propagation law from the wavelet transform equation, the optimum σth ij for each subband is automatically set. The value is determined. The value of σth is set larger as the ISO sensitivity becomes higher. For example, the fluctuation width in real space is set to a value of about 10 for 256 gradations in ISO6400.

  The threshold value rth should be about 0.5 to 3.0 pixels so that the range of the noise removal filter overlaps between resolution layers, and if the integration range is about 2 to 3 times rth, the coefficient value will be sufficiently small. falling. In general, even with other noise removal filters, a sufficient noise removal effect can be obtained with a filter that refers to pixel signals in the range of about 3 × 3 to 9 × 9 in a subband image expressed in multiple resolutions.

3-1-2. Noise extraction processing In each subband, noise extraction processing as shown in the following equation is performed (processing (1-3) (2-3) (3-3) (4-3) (5-3)).

3-2. Sequential noise extraction It is difficult to extract noise components without omission only by noise removal filtering of each subband surface, while referring to noise extracted at other resolutions so as not to generate gaps between frequency bands due to multiresolution decomposition. Extract noise sequentially. There are two sequential noise removal methods that are performed at the time of analysis and at the time of synthesis. In the present embodiment, only the case at the time of synthesis is shown.

  Further, in order to accurately extract noise components, the concept of noise removal is divided into two, virtual noise removal and actual noise removal. Specific processing for virtual noise removal is as follows. Performs virtual noise integration that is used only for noise extraction (Process (2-4) (3-4) (4-4) (5-4)) After subtracting from the band surface (processing (1-1) (2-1) (3-1) (4-1) (5-1)) to make it easy to extract noise components from the LL surface, the above Perform noise removal filtering.

ただし、下層のノイズ統合により生成された１つ上の階層のLLバンドのノイズ成分と、「ノイズ抽出処理」により抽出された同じサブバンド面の冗長なノイズ成分とを合成するときには、同じLLサブバンド面上で加算を行って統合する。その様子は図４では「＋」記号で表されている。 That is, the virtual noise integration is expressed by the following equation.

However, when synthesizing the noise component of the upper LL band generated by the noise integration of the lower layer and the redundant noise component of the same subband surface extracted by the “noise extraction process”, the same LL sub Perform integration on the band surface. This is represented by the “+” symbol in FIG.

を行なった後に、３−１−１及び３−１−２の処理を行なう処理を繰り返すことを意味している。M=5のときは何もノイズ抽出行われていないノイズ成分を統合してくるので、N5(x,y)=0となる。 (Supplementary explanation)
As is clear from FIG. 4, if the processing performed in FIG.

Means that the process of 3-1-1 and 3-1-2 is repeated after the process is performed. When M = 5, N5 (x, y) = 0 because noise components from which no noise is extracted are integrated.

4). Edge extraction Edge components are extracted from each subband surface from which virtual noise has been removed by the following equation (processing (1-5) (2-5) (3-5) (4-5) (5-5)).

Here, a Laplacian filter is used as the edge detection filter. As a Laplacian filter, the simplest filter having a center defined by 3 × 3 and having a coefficient of 8 and a periphery of −1 may be used. However, in order to accurately extract edges that remain even after virtual noise removal, it is preferable to link with the filtering range of the noise removal filter. For example, when the smoothing target range of the noise removal filter is about 9x9, the Laplacian filter is also set to about 9x9. That is, the Laplacian may be defined by {(original image) − (Gaussian blur image)}, and examples of filters include the following. However, t represents a transposed matrix and is composed of products of one-dimensional separation filters.

  The edge component extracted here has a distribution as shown in FIG. 5A when an image diagram of the local frequency distribution of the edge strength is drawn. The reason why the weak edge with the maximum peak near zero is extracted even though the edge structure that fluctuates as much as the noise fluctuation width is all extracted from the surface after disappearing by virtual noise removal From the trace of the weak edge structure that remains slightly after the removal of virtual noise and the state of the image structure of the large edge component that exists around it, the weak edge component presumed to have been in the vicinity is inferred.

5. Self-refining of noise components There is no guarantee what kind of noise components are extracted from among the noise components extracted by the noise removal filter. Depending on the situation of the image structure, a singular component is included. In addition, the manner in which it is included varies depending on the performance of the noise removal filter.

Therefore, if we recall that the noise component originally extracted or smoothed by the noise removal filter is random noise, the behavior of the extracted noise component also has a Gaussian distribution characteristic due to the Poisson distribution in the gradation direction. If not, it is reasonable to think that something is wrong on the noise removal filter side. In other words, by statistically verifying whether the noise extraction result behaves like noise, it is possible to exclude erroneously mixed singular edge components and to approximate the appearance of random noise (process (1- 6) (2-6) (3-6) (4-6) (5-6)).

  This is an assumption that can be placed because noise extraction is performed in a uniform noise space. In an image processing space where the amplification factor of noise amplitude differs between bright and dark areas, all brightness levels are asymmetrical without such a symmetric Gaussian distribution, and what distribution is distributed for each brightness level. It is impossible to predict whether such a smart process will be performed.

  Here, the value of σn th ij is preferably set to about 6 times the value of the noise fluctuation index value σth ij used in the noise removal filter. That is, if it exceeds Six Sigma, it will be recognized as statistically abnormal.

6). Self-refining of edge component The edge component extracted by the edge extraction process includes a special disturbance component as seen from the frequency distribution that causes ringing and halo as shown in FIG. Conventionally, in order to attenuate such components, a function in which the output edge strength monotonously increases with respect to the input edge strength is passed. However, since there is no clear model and guideline regarding the authenticity of the extracted edge component, a monotonically increasing function has been set based on the idea that the extracted edge component must be handled effectively. However, leaving a finite value no matter how large the edge strength is, there is a risk of causing ringing.

  In this embodiment, a model of an edge component capable of natural edge enhancement without causing ringing is established, and a measure is taken to completely erase other components based on the determination guideline. This is because the edge component targeted by the present embodiment can be accurately estimated and restored by using the edge component that is buried by the noise component as described in the section “Basic idea”. It is because of that. In that sense, it is not allowed to extract ringing or halo components that should not exist. This must be met for each subband of the multi-resolution.

Therefore, the hypothesis is that natural edge components extracted in an ideal uniform color / uniform noise space are Gaussian distributed. Accordingly, when a non-linear function having a strong attenuation characteristic such that an edge component having a certain amplitude level or more as shown in FIG. 6 completely disappears is passed, it becomes possible to approximate the shape of the edge component using the edge component as a model. . The self-refining of the edge component can be expressed as follows (processing (1-7) (2-7) (3-7) (4-7) (5-7)).

  Here, the value of σe th ij needs to be set to exactly the same value as the value of the noise fluctuation index value σth ij used in the noise removal filter. By doing so, the ringing component is eliminated, and the weak edge component that is predicted to disappear due to noise removal because it cannot be distinguished by the noise removal filter can be recovered again by the same amount here and extracted. This is because the weak edge component can be restored when added as edge enhancement. Even without the noise removal process, it is possible to accurately extract the same amount of the amount of the contour component corresponding to the contrast of the edge that decreases due to the noise covering and the amount of the texture component that becomes invisible due to the noise covering.

  In this way, when handled in a uniform color / uniform noise space, both the edge component and noise component are ideally both noise components based on the guideline that the frequency distribution in the local region is ideally Gaussian. It is possible to prevent contour blurring due to noise removal due to included edge components, and to prevent occurrence of halos and howling during edge enhancement due to ringing components included in edge components. Further, by creating a model having the same Gaussian distribution width as that of the noise fluctuation index value, it is possible to extract a weak edge and a contrast component of the contour exhibiting properties similar to the noise amplitude width.

  The edge component self-refining is performed by correcting each edge component so that the frequency distribution related to the strength of the edge component approaches a Gaussian distribution having a predetermined width based on the noise fluctuation index value σth ij. Further, the noise fluctuation index value σth ij is set in accordance with the noise fluctuation width expected for each subband of each resolution as described above. Therefore, the extracted edge component of each subband (band-limited image) of each resolution has a frequency distribution related to the intensity of the edge component for each subband to a Gaussian distribution with a predetermined width unique to each resolution (each band). It is corrected to approach.

  In other words, the self-refining of the edge component estimates the amount of the false edge component contained in the edge component based on the size of the edge component itself. The real edge component is extracted by excluding the false edge component from the extracted edge component. The edge component thus refined has a frequency distribution as shown in FIG.

7). Refinement of noise components by edge (mutual refining 1)
As described above, the purity of the noise component and the edge component that have been subjected to the processing of item number 5 and item number 6 is increased with considerable accuracy. However, the self-refined noise component in item number 5 still contains an edge component that behaves like a noise component. This edge component can only be excluded by estimating the extent to which a noise component can be included with reference to a highly reliable edge component. Since the edge component mixed here is an edge component such as a texture that behaves at the same intensity as noise, such a texture region often has a large fractal edge structure around it. Therefore, the amount of contamination can be estimated by referring to the edge strength of the region.

That is, based on the following hypothesis. In the region where the edge component strength (absolute value) is large, the majority of the noise component is a mixed edge component, and in the region where the edge component strength (absolute value) is small, most of the noise component is a true noise component. Is likely. Using this Gaussian distribution for this probabilistic model, mutual refining is performed as follows (processing (1-8) (2-8) (3-8) (4-8) (5-8)). Since a model with the same Gaussian distribution is used, the shape of the ideal noise model is not destroyed by this mutual refining, and it approaches the ideal form more.

  Here, the value of σne th ij is preferably set to 6 times the noise fluctuation index value σth ij as in the case of σn th ij used in item number 5. That is, the mixed edge component is eliminated in such a region that is considered to be statistically perfectly an edge.

8). Refinement of edge components by noise (mutual refining 2)
Similar to the description of item number 7, the edge component self-refined in item number 6 also includes a noise component that behaves like an edge component. This noise component can only be excluded by estimating the extent to which such an edge component can be included with reference to a highly reliable noise component. The noise component mixed here is a noise component that behaves with the same intensity as the edge component to be extracted. The noise component mixed in such an edge component should be extracted as a noise component having the same level as the noise component extracted in the item number 7.

Therefore, it is possible to estimate the ratio of the noise component mixed in the edge component based on the following hypothesis. That is, in the area where the intensity (absolute value) of the noise component is large, the majority of the edge component is a mixed noise component, and in the area where the intensity (absolute value) of the noise component is small, most of the edge component is true. There is a high possibility of being an edge component. Using this Gaussian distribution for this probabilistic model, mutual refining is performed as follows (processing (1-9) (2-9) (3-9) (4-9) (5-9)). Since a model with the same Gaussian distribution is used, the shape of the ideal edge model is not destroyed by this mutual refining, and it approaches the ideal form more.

  Here, the value of σen th ij needs to be set to the same value as the noise fluctuation index value σth ij, similarly to σe th ij used in item number 6. By doing so, it is possible to exclude a noise component that is also observed in the extracted noise component included in the edge component. Therefore, it is possible to prevent the noise feeling from increasing due to edge enhancement.

  The above processing means that the larger the absolute value of the extracted noise component is, the larger the ratio of the residual noise component included in the edge component is, and the residual noise component estimated from the edge component is excluded. . In other words, the above processing is performed by estimating the existence ratio of the actual edge component in the extracted edge component using a Gaussian distribution function with the absolute value of the extracted noise component as an argument, and obtaining the estimated actual edge component. Means that In other words, the noise fluctuation index value used when the noise component is extracted from the original image is included in the edge component by comparing it with the absolute value of the noise component at each extracted pixel position. The ratio of residual noise components is estimated.

9. Integration of real noise components After the noise components are extracted without omission in each sub-band, the finely refined noise components are now integrated with frequency band weights that have the least damage to the real image and high noise reduction effect. To do.

となる。図４の重みkn ijは(重み(1-14)(2-14)(3-14)(4-14)(5-14))、低周波サブバンドと高周波サブバンドの間の重みkn iと解像度レベル間の重みkn jの２つが入っている。 As weights between frequency bands, there are weights kni between low-frequency subbands and high-frequency subbands, and weights kjj between resolution levels. The integrated weight of the noise component of the band. In other words,

It becomes. The weight kn ij in FIG. 4 is (weight (1-14) (2-14) (3-14) (4-14) (5-14)), and the weight kn i between the low frequency subband and the high frequency subband. And a weight kn j between resolution levels.

9-1. Setting weights between L / H bands According to WO 2007/114363 pamphlet and WO 2007/116543 pamphlet by the inventor of the present application, integrated weights of noise components that minimize damage to actual images If the original image is a luminance component, it is recommended to increase the weight of the high-frequency subband noise and decrease the weight of the low-frequency subband noise. Also, if the original image is a color difference component, the weight of the low frequency subband noise should be increased, the weight of the high frequency subband noise should be decreased, or it should be the same as the weight of the low frequency subband noise. Yes. In addition, the weight of the low frequency subband and the high frequency subband greatly change the frequency characteristics of the integrated noise component and greatly change the appearance of noise removal. The band-to-band weight is open to the user as a granularity setting parameter.

  However, in addition to the parameters that control the graininess, setting of the degree of blur (unsharpness) by controlling the noise removal rate λ and setting of the noise removal intensity (intensity) by controlling the noise fluctuation index value σth of the noise removal filter It is necessary to do. This is a difficult situation for general users who are not familiar with technology to understand the three-axis control and obtain the best results. Therefore, in the present embodiment, based on the physical effect on the image quality, parameters that do not need to be completely independently controlled can be easily linked with other parameters so that a high-quality noise removal result can be easily obtained. To do.

  Noise removal by the low-frequency sub-band of the luminance component produces a flat image that has lost its tonal characteristics and has a great detrimental effect on noise removal. On the other hand, noise removal by the high-frequency subband of the luminance component produces a smooth image and has less adverse effects of noise removal. Therefore, the image quality destruction to the luminance component is the least when the weight of the low-frequency subband noise is eliminated. At this time, noise extraction omission due to not using the low-frequency subband occurs. Similarly, for the color difference component, the image quality destruction is least when the weight of the high frequency subband noise is reduced to some extent, but at this time, the protruding color noise remaining due to the reduction of the weight of the high frequency subband noise. Is likely to occur.

  Considering these factors, when the noise removal rate is small, the most important point is to eliminate image quality destruction even if there is a slight noise extraction omission, and when the noise removal rate is large, there is planar image quality destruction. It is considered that the best results are obtained by designing the most important thing to eliminate noise extraction leakage. Here, we introduce an overall noise removal rate λjoint to control various parameters collectively and control the integrated weight of noise components between L / H bands as follows (Process (1-16) ( 2-16) (3-16) (4-16) (5-16)). However, it is normal to set the value of λjoint to 0 ≦ λjoint ≦ 1, but in consideration of the functionalization of the subsequent noise removal rate and edge enhancement rate, approximately 0 ≦ λjoint ≦ 8 is a guideline. The above upper limit is taken, and there is no limit to a clear upper limit.

  The setting of λjoint is performed by displaying, for example, a setting screen having a slide bar on a monitor (not shown) of the personal computer 1 (processing (0-5)). The user sets the value of λjoint by operating the cursor in the slide bar on the setting screen to an arbitrary position using a keyboard (not shown) or a mouse (not shown). As a result, the user can easily set the parameter of the λjoint. When processing with the camera, the camera manufacturer should determine and set the corresponding value in advance according to the noise removal strength level “weak”, “medium”, “strong”, etc. provided to the user by the camera. Also good.

In case of luminance component

ただし、0≦k_ni≦1 (i=LL,LH,HL,LL)を満たすようにする。 For color difference components

However, 0 ≦ k _ni ≦ 1 (i = LL, LH, HL, LL) is satisfied.

  The above formula is an example, and it does not stop in this way. Normally, LL is treated as a low-frequency subband and LH, HL, and HH are treated as high-frequency subbands. Considering that LH and HL also have some low-frequency subband characteristics, for example, the luminance component LH Therefore, the HL band may be slightly reduced in weight within a range close to 1 in conjunction with the overall noise removal rate λjoint.

9-2. Setting weights between resolution levels Normally, noise components caused by shot noise can be considered as white noise that is uniformly distributed from low frequency components to high frequency components, so the noise integration weights between resolution levels are all the same. Set the value to 1. That is, the following formula is obtained.

9-3. Integration processing of actual noise components The noise components weighted optimally between the frequency bands that are optimal for both the above-mentioned noise removal effect and its countermeasures are integrated as shown in the following equation (Process (1-10) ( 2-10) (3-10) (4-10) (5-10)).

  However, as in the case of the virtual noise integration of item number 3, the integration is performed by adding the LL band noise component integrated from the lower layer and the two noise components existing from the LL band noise extraction. (Processing (1-11) (2-11) (3-11) (4-11) (5-11)). Others are integrated by performing inverse wavelet transform processing.

  Since the noise components extracted in this way are frequency-adjusted according to the frequency characteristics of the original image of the luminance plane and the color difference plane, the optimal frequency projection space for noise extraction is matched to the characteristics of the original image. It may be said that it is changing. Further, the frequency projection space corresponds to changing the noise removal rate.

10. Integration of real edge components We extract weak edge components that are buried in noise in each subband, and use the edge components that have been refined with the noise components and ringing components included in the subbands. The effects of contrast restoration and noise removal are combined to give the most natural impression, with weights between frequency bands that have a high enhancement effect by edge enhancement.

となる。図４の重みke ijは(重み(1-15)(2-15)(3-15)(4-15)(5-15))、低周波サブバンドと高周波サブバンドの間の重みke iと解像度レベル間の重みke jの２つが入っている。 As in the case of the noise component, there are weights ke i between the low-frequency subband and the high-frequency subband and a weight ke j between the resolution levels as weights between the frequency bands. Let ij be the integrated weight of the final edge component of that subband. In other words,

It becomes. The weight ke ij in FIG. 4 is (weight (1-15) (2-15) (3-15) (4-15) (5-15)), the weight ke i between the low frequency subband and the high frequency subband. And the weight ke j between the resolution levels.

10-1. Setting of weights between L / H bands In the pamphlet of International Publication No. 2007/114363 and the pamphlet of International Publication No. 2007/116543 by the inventor of the present application, there is no description about how to mix frequencies. However, according to the elucidation of the subjective amount and the physical effect at the beginning of the present embodiment, the frequency blending method that gives the most natural impression and has the high enhancement effect can be assembled from the following discussion.

  First, let us consider a case in which an image that has become blurred due to noise covering due to amplification of noise fluctuation amplitude when the sensitivity of simple high-sensitivity shooting is increased without noise removal is considered. First of all, this edge of the original image, when no edge emphasis is performed, restores gradation contrast that has become white due to the amplitude width of the noise of the luminance component, and also restores texture that has been buried in noise. First, everything needs to be restored, such as restoration of saturation that has been made dull due to the color noise of the color difference component, and restoration of blurring of the color boundary caused by the superimposition of the color noise. Therefore, all of the edge enhancement effects by the edge components extracted from each of the two resolution bands of the low-frequency subband and the high-frequency subband in the multi-resolution space are required.

  However, it has been experimentally found that, when the edge enhancement rate is increased to some extent, some of these redundant components cause significant image destruction. In other words, when the edge enhancement rate is raised to the maximum of 100%, the contrast enhancement effect by the integrated edge component on the low frequency side is excessively abnormal on the luminance surface, and a halo that floats white at the main image boundary occurs. On the color difference plane, the effect of sharpening the color boundary due to the integrated edge component on the high frequency side acts too much, and the main image boundary portion is similarly colored. These frequency bands are extremely sensitive on both the luminance side and the color difference side, and have a side effect of causing a catastrophic image collapse if care is not taken in handling.

  On the other hand, the image restoration effect by the integrated edge component in the other conjugate frequency band does not bring about unnatural image destruction even if the edge enhancement rate is increased. In other words, the texture restoration effect by the integrated edge component on the high frequency side in the luminance component and the saturation enhancement effect by the integrated edge component on the low frequency side in the color difference component are natural image restoration effects even if the edge enhancement rate is increased linearly. Can be obtained. However, in order to obtain these natural edge enhancement effects, the size of the edge component in each multi-resolution subband surface must not exceed the size of the noise component that the original subband surface has. If the edge component is not superimposed within the range buried in the noise, the edge component is visually recognized as a ringing, howling, and halo phenomenon. Since the effect at that time is multi-resolution edge enhancement, it is extremely enormous. Therefore, it is necessary to pay careful attention to the frequency blending method between the low frequency and high frequency bands of the edge component.

  The same is true for image restoration when noise removal is involved. There is another point to be aware of when noise removal is involved. That is, in order to obtain a natural edge enhancement effect, the edge components must be within an amplitude that is buried in noise. However, if noise reduction is added, the range of noise fluctuations will be reduced, so the range of visual tolerances for components that have an unnatural effect in edge enhancement will be narrower than without noise removal. That is. Therefore, a sensitive frequency band that has an adverse effect as the edge enhancement rate increases needs to be a monotonically decreasing function depending on the edge enhancement rate, and also needs to be a monotonically decreasing function depending on the noise removal rate. There is.

  Based on these facts, an image with a high enhancement effect that gives the most natural impression can be restored by controlling as follows. That is, while the edge enhancement rate is small, it is necessary to place emphasis on both contrast enhancement and texture recovery in the luminance component, and emphasis on both colorfulness restoration and color boundary contrast sharpening in the color difference component. On the other hand, when the edge enhancement rate is increased, contrast enhancement is moderately performed in the luminance component, and the emphasis is shifted to the restoration of the texture. With the color difference component, the sharpening of the color boundary is moderated, and the emphasis is on recovery of colorfulness and saturation. That is, the edge enhancement rate dependency is controlled to control the coefficient applied to the edge of a certain frequency band as the edge enhancement rate is increased.

  As the noise removal rate increases, the frequency components on the side where the specific gravity needs to be reduced further according to the amount of noise components that cause the edge components to be buried in a visually natural manner. It is necessary to reduce it until it disappears completely. That is, the noise removal rate dependency was introduced in the calculation of the coefficient applied to the edge.

  If these relationships are expressed by equations, the integrated weight of the edge components between the L / H bands is controlled as follows. Here, an overall edge enhancement rate ζjoint is introduced. Normally, the value of ζjoint is normally set to 0 ≦ ζjoint ≦ 1, but in consideration of the functionalization of the edge enhancement rate later, an upper limit value about 0 ≦ ζjoint to <8 is set as a guideline. However, there is no clear upper limit. Speaking of the processing of FIG. 4, the processing (1-17), (2-17), (3-17), (4-17), and (5-17) correspond.

  The setting of ζjoint is performed by displaying, for example, a setting screen having a slide bar on a monitor (not shown) of the personal computer 1 (processing (0-6)). The user sets the value of ζjoint by operating the cursor in the slide bar on the setting screen to an arbitrary position using a keyboard (not shown) or a mouse (not shown). As a result, the user can easily set the parameter of the ζ joint. When processing with the camera, the camera manufacturer will set the corresponding value in advance according to the strength setting level “weak”, “medium”, “strong”, etc. for edge enhancement performed in the new multi-resolution space provided to the user by the camera. May be determined and set.

色差成分の場合

なお、MIN(ζjoint,1)は、ζjointと1のどちらか小さい方の値を取るという意味である。 In case of luminance component

For color difference components

Note that MIN (ζjoint, 1) means that the smaller value of ζjoint and 1 is taken.

  These satisfy the condition of 0 ≦ kei ≦ 1 (i = LL, LH, HL, HH). The above formula is an example, and it does not stop in this way. Normally, LL is treated as a low-frequency subband and LH, HL, and HH are treated as high-frequency subbands. Considering that LH and HL also have some low-frequency subband characteristics, for example, the luminance component LH Therefore, the HL band may be slightly reduced in weight in the range close to 1 in conjunction with the overall edge enhancement rate ζjoint.

Next, a frequency blending method for obtaining a multi-resolution edge enhancement effect that gives a more natural impression in a region where the overall edge enhancement rate ζjoint takes a value of about 1 to 8 will be considered. The above formula gives an image with no problem at all in the range of ζjoint = 0 to 1, but when the edge enhancement rate exceeds 100%, the edge component exceeds the allowable range of noise fluctuation that should be allowed originally Begins to be emphasized. That is, since the intensity of the edge component is given in the form of the product of ζjoint * ke _ij , the sensitive frequency band that causes the collapse of the image structure is a monotonically increasing type that is slowed down in the region where ζjoint = 0-1. In contrast to the function, the linear increase increases without decelerating again in the region of ζjoint = 1-8. For example, if the LL component of luminance is expressed in a form that omits λjoint for simplicity, the interval of ζjoint = 0 to 1 is
ζjoint * (1-ζjoint / 2) =-(1/2) (ζjoint-1) ² + (1/2)
Thus, it is a deceleration type monotonically increasing function of a quadratic function that is convex upward and has a maximum when ζjoint = 1. On the other hand, when ζjoint ≧ 1, ζjoint / 2 turns into a proportional type. The sensitive frequency band is a band having ζ joint dependency, and can be said to be an LL component in terms of luminance components, an LH component, an HL component, and an HH component in terms of color difference components.

To prevent this, a mechanism that completely stops at the value at the edge enhancement rate of 100% that gives the maximum tolerance that the contribution after multiplying the edge enhancement rate of the sensitive frequency band falls within the range of noise fluctuation Include. On the other hand, the frequency band that can realize natural edge enhancement even when exceeding the range of noise fluctuation allows edge enhancement by allowing the product value of ζjoint * ke _ij to monotonically increase even when the edge enhancement rate exceeds 100%. Provide an effect. The optimal formula is presented below. In this case as well, it is also considered that the range of edge enhancement permitted when noise removal is added is limited.

色差成分の場合

In case of luminance component

For color difference components

  Regarding the luminance component, the LL component is positioned on the low frequency side of the subband and has a contrast enhancement effect (sensitive frequency band), so it is completely enclosed within the allowable range so that halo does not occur ( It is locked to the value when the edge enhancement rate is 1. One LH component is usually positioned on the high-frequency side of the main band, but it also has some characteristics of the sub-band on the low-frequency side, so it is suppressed to some extent so that it does not increase linearly with respect to the edge enhancement rate. Yes. As a result, it is possible to prevent vertical and horizontal stripe structures from occurring in the region where the edge enhancement rate is 800%. In addition, the HH component corresponds to the high frequency side of the main band, but a non-linear increase to the edge enhancement rate considering the degree of LH component deterrence so as to create a natural texture recovery by smoothly connecting with the LH component deterrent mechanism It has characteristics.

  On the other hand, for the color difference component, the LL component as the main band, the HH component as the subband, and the LH and HL components in the middle of them are all the maximum edge enhancement effect at the point of 100% edge enhancement rate, that is, maximum saturation enhancement. Color contrast restoration completely stops. This is because if the color difference plane is further enhanced beyond the range of noise fluctuation, the hue of the image starts to change, leading to image collapse (sensitive frequency band).

  The optimal method derived experimentally in this way is the transition of the optimum frequency projection space according to the noise removal rate in noise removal and the optimum frequency projection space according to the edge enhancement rate in edge enhancement. It is better to change the state of the transition. However, it differs only in terms of whether to monotonously increase or monotonically decrease the side of the high-frequency subband and the low-frequency subband that serve as subbands with respect to the noise removal rate and edge enhancement rate. In both noise removal and edge enhancement, the high frequency subband of the luminance component and the low frequency subband of the color difference component, which are main bands, must always be utilized to the maximum. The sub-band side (the sensitive frequency band described above) needs to be used with a good adjustment so as not to destroy the image quality.

  In FIG. 4, the weight ke ij (weight (1-15) (2-15) (3-15) (4-15) (5-15)) is only dependent on the edge enhancement rate ζjoint. As described above, since it also depends on the noise removal rate λjoint as described above, it can be considered that there is an arrow from the noise removal rate λjoint.

10-2. Setting the weight between resolution levels Unlike the case of noise components, the weight between resolution levels considering the effect on human vision is set without setting the edge integration weight between resolution levels to the same value of 1. Set.

  If you look at an image that has been denoised as white noise evenly at each resolution, the effects of noise removal due to a decrease in the contour contrast, etc., do not appear to have been crushed at the same resolution. The medium frequency component that seems to cause the most damage to human visual characteristics is a medium frequency component or a medium frequency component rather than a high frequency component. When these outlines are lost, the sense of loss of the appearance of the information that the image has most is large. Therefore, in noise removal, it is necessary to put the maximum effort to protect the edge structure of components from the medium frequency to slightly higher frequency.

  On the other hand, if edge enhancement is set to the same value of 1 between resolution levels, the range of influence of edge components picked up from the low resolution side is vast, and the risk of causing howling and halo increases accordingly. Also, the edge component of the highest resolution level at the Nyquist frequency level is somewhat apt to be mistaken for a noise component. Therefore, it is safe to use those components lower than in the middle frequency band. Therefore, the frequency band that requires the maximum protection in noise removal and the frequency band that can be restored the most by edge enhancement, that is, the frequency band that requires restoration are exactly the same.

Therefore, it is desirable to set the weight between the resolution levels focusing on the integrated edge component also from the viewpoint of recovery of the visibility characteristic. Convenient distribution with characteristics that the intensity is slightly lower at high frequencies on the highest resolution level, the intensity is higher at the resolution level slightly higher than the intermediate frequency band, and the intensity decreases at lower frequencies on the lowest resolution level side. There is a Poisson distribution. Here, a setting example in which the average value μ of the Poisson distribution gives a resolution level of 2.0 on the low frequency side slightly lower than the middle with respect to the number of stages of multi-resolution conversion of five stages is shown. The average value μ of the Poisson distribution is usually set at 40% of the overall resolution level. That is, the following expression is obtained.

ピーク強度が一段目と２段目の間にあり、平均値の2.0段よりやや高周波解像度に寄っていることが分かる。 Specific numerical values are as follows. However, although not explicitly written in the formula, the normalization processing is performed with the maximum value of the set distribution intensity.

It can be seen that the peak intensity is between the first and second stages, which is slightly closer to the high frequency resolution than the average of 2.0 stages.

  Next, a method for dealing with the case where the edge emphasis rate exceeds 100% is similarly considered for the weight between resolutions. When the edge enhancement rate exceeds 100%, the risk of howling and halo due to the edge component picked up on the low resolution side increases more and more. The same applies to the edge component of intermediate resolution. Therefore, only the edge component on the high resolution side can perform edge enhancement without causing image collapse in the most natural form. This prevents image collapse if the edge enhancement effect by multi-resolution, that is, contrast enhancement, texture restoration, saturation enhancement, and color contrast restoration is not connected asymptotically to edge enhancement processing of only the luminance plane in real space. That means you can't.

As a control method for this purpose, the Poisson distribution has very convenient characteristics.As the edge enhancement rate increases, the Poisson distribution average value can be adjusted by bringing the average value of the Poisson distribution closer to 1 or 0 at the resolution level. realizable. The realization formula is described below. That is, the normal setting of the average value of the Poisson distribution is first performed, and an operation is performed so that the average value approaches 1 in a region where the edge enhancement rate exceeds 100%. If the number of stages of multi-resolution currently being handled is M, the average value μ is obtained from the following equation. Note that the following equation is also applicable when the edge enhancement rate is 100% or less.

In the limit state when the edge enhancement rate exceeds 100% and becomes extremely large, the distribution of weights between resolutions changes to the high resolution side as follows. This is an example in the case of M = 5 stages. Here, it is considered not only when the edge enhancement rate becomes large, but also when the noise removal rate becomes large, the edge component appears relatively larger than the noise fluctuation width.

  As described above, when the edge enhancement rate exceeds 100%, the distribution state of weights between resolutions, that is, the gravity center position of the weights between resolutions changes to the high resolution side according to the size of the edge enhancement rate. I am doing so. In this way, natural edge enhancement can be performed even when the edge enhancement rate reaches 800%, and image collapse is not caused.

  Here, the difference between the edge enhancement processing using an unsharp mask in a normal real space and the multi-resolution edge enhancement processing described in the present embodiment will be pointed out. Edge enhancement by normal unsharp mass processing takes the high-frequency component due to the difference between the original image and the smoothed image of about 7x7 at the resolution of real space LL0 located in the low-frequency subband, and the coring component is After removing it as a noise component, howling and ringing components are mainly extracted, emphasized and scaled, and then added to the original image or smoothed image. In other words, the unsharp mask removes edge components that are buried in noise from the extracted edge components by base clip processing, and only extracts strong edge components based on ringing and howling to some extent. It is.

  In contrast, multi-resolution edge enhancement basically extracts coring components corresponding to the degree of noise fluctuation, excludes howling and ringing components, and handles the frequency band in terms of luminance. When the emphasis rate is increased, the edge emphasis, function, and role by normal unsharp mask processing are considerably different in that the high frequency subband is mainly used. That is, edge enhancement by multi-resolution is based on the premise that only edge components that are buried in noise are extracted, and edge components having the same strength as noise are well selected and extracted from noise. Thereby, a large image improvement effect can be obtained by handling with multiple resolutions without noise amplification. Here, a slight fluctuation edge component buried in noise plays an important role.

  Therefore, since the contrast, texture, saturation, and color contrast recovery functions by multi-resolution edge enhancement and the edge enhancement by unsharp mask are independent functions, there is no problem even if both are used simultaneously in parallel.

10-3. Integration processing of actual edge components Edge components weighted optimally between frequency bands to obtain the edge enhancement effect of natural impression as described above are integrated as shown in the following formula (Process (1-12) ( 2-12) (3-12) (4-12) (5-12)).

  However, as in the case of the actual noise integration of item number 9, integration is performed by adding the edge component of the LL band integrated from the lower layer and the two edge components of the edge existing from the original LL band extraction. (Processing (1-13) (2-13) (3-13) (4-13) (5-13)). Others are integrated by performing inverse wavelet transform processing.

  The edge components extracted in this way are frequency-mixed according to the frequency characteristics of the original image on the luminance and chrominance planes. It may be said that it is changed according to. Further, this also corresponds to changing the frequency projection space depending on the edge enhancement rate and the noise removal rate.

11. Self-refining of integrated noise component Next, based on the same idea as item number 5 for the integrated noise component, self-refining of the noise component is performed by the following equation (processing (0-1)). That is, whether or not the noise component that has been finally integrated to the same level as the actual resolution has a Gaussian distribution that is an ideal noise distribution in the uniform color / uniform noise space based on the hypothesis again. It is important to verify.

  In addition to the meaning of double verification, there is another important meaning in the self-refining of noise for the integrated noise component. Because noise components are extracted in a frequency projection space consisting of two low-frequency subband groups and high-frequency subband groups that are redundant with respect to the complete system, the noise components extracted in each subband are Gaussian distributed. This is because there is no guarantee that noise components having frequency redundancy after integration will be Gaussian again. Therefore, it is very meaningful to approach the Gaussian distribution shape once more based on the ideal noise distribution hypothesis.

  Here, the value of σ N th is preferably set to 6 times the value of the noise fluctuation index value σ th in the real space based on the same idea as the item number 5. That is, noise components that are not considered to be statistically 100% noise exceeding Six Sigma with respect to the noise fluctuation index value are excluded.

12 Self-refining of the integrated edge component For the integrated edge component again, the edge component is self-refined according to the following formula based on the same concept as item number 6 (processing (0-2)). That is, whether or not the edge component finally integrated to the same level as the actual resolution has a Gaussian distribution, which is an ideal edge distribution in the uniform color / uniform noise space, again based on the hypothesis. It is important to verify.

  For the self-refining of the edge with respect to the integrated edge component, another important meaning is the same as the item number 11 in addition to the meaning of the double verification. Because the edge components are also extracted in the frequency projection space consisting of two low-frequency subband groups and high-frequency subband groups that are redundant with respect to the complete system, the edge components extracted in each subband are Gaussian distributed. This is because there is no guarantee that edge components having frequency redundancy after integration will be Gaussian again. Therefore, it is very meaningful to approach the Gaussian distribution shape once more based on the ideal edge distribution hypothesis.

  Here, the value of σE th is preferably set to the same value as the value of the noise fluctuation index value σth in the real space based on the same idea as the item number 6. By doing so, the ringing component is eliminated, and the weak edge component that is predicted to disappear due to noise removal because it cannot be distinguished by the noise removal filter can be recovered again by the same amount here and extracted. Become.

13. Refining integrated noise components using integrated edges (mutual refining 3)
Based on the same idea as item number 7 for the integrated noise component once again, mutual refining by the integrated edge component of the noise component is performed by the following equation (processing (0-3)). This is a model that has a high probability that an edge component is mixed in the noise component in a region where the edge strength (absolute value) is strong in the neighborhood, and an integrated edge component that takes into account multiple correlations of multiple resolution levels once again. This is to verify the neighboring edge situation, which could not be found only by the edge component observed at a single resolution, by changing the frequency viewpoint. That is,

  Here, the value of σNE th is preferably set to 6 times the value of the fluctuation index value σth of the real space based on the same concept as the item number 7. That is, an edge component that is likely to be erroneously included in a noise component is excluded in a statistically 100% edge region that exceeds six sigma with respect to the noise fluctuation index value. Moreover, since the mutual mixing model by the same Gaussian distribution is used, the shape of an ideal noise model is not destroyed by this mutual refining, and it approaches the ideal form more.

14 Refining integrated edge components using integrated noise (mutual refining 4)
Also for the integrated edge component, based on the same idea as the item number 8 again, mutual refining by the integrated noise component of the edge component is performed by the following equation (processing (0-4)). This is an integrated noise component that takes into account a high probability that noise components are mixed in the edge component in a region where the noise intensity (absolute value) is high in the neighborhood, and once again considers multiple resolution level correlations. Is to verify the distribution status of neighboring noise components, which could not be found only with noise components observed at a single resolution, by changing the viewpoint of frequency.

  Here, the value of σEN th is preferably set to the same value as the value of the fluctuation index value σth of the real space based on the same concept as the item number 8. By doing so, it is possible to exclude a noise component that should have been observed even in the extracted noise component included in the edge component. Moreover, since the mutual mixing model by the same Gaussian distribution is used, the shape of an ideal edge model is not destroyed by this mutual refining, and it approaches the ideal form more.

  The paraphrased expression described in item number 8 can be similarly applied to the refining of the integrated edge component by the integrated noise.

15. Generation of a virtual noise-removed luminance plane In order to obtain a noise-free reference luminance plane used in the next item number 16, a temporary virtual noise-removed luminance plane is generated using 100% of the noise component integrated with actual noise.

  However, in some cases, this process may be skipped and replaced with the luminance plane L ^ (x) (the first term on the right side of the above equation) of the original image from which noise has not been removed.

16. Actual noise removal processing and actual edge enhancement processing As described in the explanation of the basic idea at the beginning of the embodiment, self and mutual refining of noise components and edge components have been repeated in this way. Regardless, there are intermixing components that cannot be excluded. In order to minimize the negative effects on noise removal processing and edge enhancement processing and maximize the effects of noise removal and edge enhancement, introduce a device that predicts the negative effects and effects in advance to the noise removal ratio and edge enhancement ratio. There is room for it.

  The noise component and edge component have been extracted in uniform color and uniform noise space by the processing so far, and noise removal and edge enhancement are performed with a uniform noise removal rate λ and a uniform edge enhancement rate ζ over the entire image. In this case, the noise removal effect and the edge enhancement effect should be obtained uniformly over the gradations of all brightness levels in the uniform color / uniform noise space. However, once converted from the working color space to the output color space, the noise removal effect and edge emphasis effect of a part of the image viewed in the output space are substantially weakened due to the difference in gradation characteristics, and the adverse effects thereof. There is a possibility that it will appear highlighted in some parts.

These effects and the appearance of the increase / decrease width of the harmful effects are considered to be mainly affected by the differential ratio between the gradation characteristics of the output color space and the gradation characteristics of the work color space. Therefore, a contrast ratio function with respect to the brightness expressed by the differential ratio between the gradation characteristic γ of the output color space and the gradation characteristic Γ of the working color space is defined as follows. Here, Y represents a linear gradation characteristic. It is the same as Y in the XYZ space defined by item number 1.
Reference contrast ratio function

  In an actual example, the gamma curve γ (Y) of the numerator output color space is represented by a curve such as A or B in FIG. 7A, and the gamma curve Γ (Y) of the denominator working color space is a graph. 7 (a) is represented by a curve as drawn in C, and Γ (Y) is the same as the definition of the function f (t) of item number 1. Since only the brightness of the luminance plane is referred to as an argument of the contrast ratio function, the brightness S (x, y) of the original image is referred to L ^ (x, y) converted by item number 1. The same.

  The curve group in FIG. 7B is a diagram showing a state in which the second expression of the contrast ratio function is selected as the curves A and B as γ, and one straight line indicates the contrast ratio. It is the figure which showed the mode when the formula written in the 2nd of the function was chosen for the curve of C as (gamma). Therefore, the curve of the expression written in the first and third of the actual contrast ratio function may be drawn by scaling the horizontal axis from the linear gradation Y to Γ (Y). That is, when the horizontal axis is Γ (Y), the dark part of the horizontal axis is stretched and the bright part is contracted.

16-1. Functional representation of denoising rate by contrast ratio function (Process (0-7))
1) Functional expression 1 (gamma version)
The functional representation of the noise removal rate maintains the same level of appearance as the uniform noise removal in the working color space, and the noise removal effect in the output color space is homogenized at all brightness levels. Is the first goal. By doing so, it is possible to suppress the noise amplification feeling in a region where the gradation contrast of a certain brightness level stands out in the output color space, and only in a specific region where the gradation contrast is sleeping in the output color space. It is possible to suppress the non-uniformity due to the noise reduction effect.

  At this time, the problem of where to select the setting of the increase / decrease width reference point when increasing / decreasing the noise removal rate according to the output gamma characteristic becomes important. In the case of a luminance component, it is preferable to take an exposure reference point designed with the goal of achieving an average luminance level (around 128 for 256 gradations) in the output color space. This usually corresponds to the brightness level of an ISO standard gray chart reference subject called 18% gray in linear gradation. Then, the appearance of noise component noise removal in the output color space appears to be kept constant on average throughout the brightness. Therefore, it is possible to prevent a decrease in contrast due to noise coverage in a dark area where noise may be amplified by gradation conversion to the output color space, and to obtain a clear noise removal result.

  On the other hand, since the color difference component generally requires stronger noise removal than the luminance component, the noise removal rate cannot be manipulated dynamically like the luminance component, and contrast can be maintained without degrading the noise removal effect. It takes the form of loosening the noise removal of the part. Specifically, normalization is performed at a saturation reference point in a region where the output gamma characteristic becomes the knee characteristic. As the saturation reference point, for example, a point where the output gradation characteristic is about 180 to 220 with respect to 256 gradations is selected. This will weaken the color difference surface noise removal rate in the highlight area, and the highlight area generally corresponds to the high saturation area in the color difference area, thus preventing color loss in the high saturation area and eliminating color spot noise with high color reproducibility. Give birth. Further, coupled with the effect of removing the dark portion noise of the luminance component, the noise coverage and the color fogging in the dark portion are completely eliminated, and a transparent feeling that is more cloudy than the original image is generated.

  The functional representation of the noise removal rate that realizes these by the contrast ratio function is expressed by the following equation. In the case of the luminance component, normalization is performed by substituting the contrast ratio based on the exposure reference point into the denominator of the following equation. In the case of color difference components, normalization is performed by substituting the contrast ratio based on the saturation reference point into the denominator of the following equation.

色差成分の場合

ここで色差成分については、色相が変化してしまうことを防止するために、以下の式で示すようにクリッピング処理を入れる。
ζ（→ｘ） ＝ ＭＩＮ（ζ（→ｘ），１．０） In case of luminance component

For color difference components

Here, for the color difference component, in order to prevent the hue from changing, clipping processing is performed as shown by the following equation.
ζ (→ x) = MIN (ζ (→ x), 1.0)

2) Functional expression 2 (Retinex version)
A functional representation of the denoising rate can be made from another point of view. In other words, while the functional expression 1 aims to homogenize the noise removal effect between the brightness levels of the entire image, here the brightness level between the brightness levels within the local edge structure vicinity. The goal is to make the noise removal effect uniform. This is because important parts that determine the image structure are concentrated in the vicinity of the edges, and the noise removal effect of the entire image is non-uniform as long as the noise removal effect is uniform with respect to brightness. However, it is based on the idea that another aspect of the homogenizing action can be exerted to the maximum extent in the local region.

  In other words, the homogenization of the noise removal effect is another point where the adverse effect of edge structure destruction due to the edge component that could not be separated from the noise component is homogenized between brightness levels, so that the adverse effect is strongly visible. It means to reduce the visual defects of non-uniformity that worsen the overall impression with the existence and to minimize the appearance of harmful effects on average. Therefore, if the detrimental measures for noise removal are implemented mainly in the vicinity of the edge, the important structure information of the image can be effectively saved.

  In this case as well, for output color spaces with general gradation characteristics, the contrast of edge components mixed in the area becomes stronger as the gradation contrast of the work color space is higher in bright places. It is easy to cause harmful effects in that area. In the case of the luminance component, the structural information in the white image area surface is likely to disappear, and in the case of the color difference component, the color information of the high saturation portion is likely to be lost.

  Therefore, the noise removal rate may be expressed as a functional function of the contrast ratio function by the following equation. As edge information to be referred to at this time, it is optimal to use a multi-resolution integrated edge component for observing the image structure at any scale both locally and globally. Further, it is best to use the integrated edge component Ew ″ (x, y) in which a noise component and a ringing component are firmly excluded and a visually important frequency band is added.

色差成分の場合

ここで色差成分については、色相が変化してしまうことを防止するために、以下の式で示すようにクリッピング処理を入れる。
ζ（→ｘ） ＝ ＭＩＮ（ζ（→ｘ），１．０） In case of luminance component

For color difference components

Here, for the color difference component, in order to prevent the hue from changing, clipping processing is performed as shown by the following equation.
ζ (→ x) = MIN (ζ (→ x), 1.0)

  Here, the value of σgE th may be given as a value linked to the noise fluctuation index value σth in real space, or an absolute edge strength level. As a result, it is possible to set an edge strength level at which it is desired to prevent edge / contrast reduction due to outline blurring.

3) Functional expression 3 (composite version)
The functional representations 1 and 2 of the denoising rate are locally different in effect. Therefore, the usage which combined two methods is also considered. In that case, a functional expression may be basically obtained by taking the product of two functional expression parts.

16-2. Functional representation of edge enhancement rate by contrast ratio function (Process (0-8))
1) Functional expression 1 (gamma version)
Similarly, the functional expression of the edge enhancement rate also maintains the same level of appearance as the uniform edge enhancement in the working color space, while the edge enhancement effect in the output color space is uniformed at all brightness levels. The first goal is to be done. By doing so, the lack of edge enhancement due to edge components extracted with low contrast in the working color space in areas where the gradation contrast of a certain brightness level stands out in the output color space is suppressed, and It is possible to suppress a feeling of non-uniformity due to an excessive edge emphasis effect due to edge components extracted with a high contrast in the working color space of only the specific region where the tonal contrast is lower in the output color space. That is, in the working color space, the contrast of the edge component needs to be increased in advance in the gradation region where the contrast is lower than that in the output color space, and on the contrary, the contrast of the edge component needs to be weakened in advance in the region where the contrast is high.

  At this time, when increasing or decreasing the edge enhancement rate according to the output gamma characteristic, the issue of where to select the reference point for the increase / decrease width is important. In the case of a component, it should be taken as an exposure reference point designed to achieve an average luminance level (around 128 for 256 gray levels) in the output color space. Then, the appearance of edge enhancement of the luminance component in the output color space appears to be maintained constant on average throughout the brightness. Therefore, it is possible to prevent the black floating phenomenon due to insufficient gradation correction in the dark area where the edge / contrast may be insufficient due to the gradation conversion to the output color space. A clear edge emphasis effect can be obtained.

  On the other hand, the color difference component requires edge enhancement that is generally stronger than the luminance component in order to obtain an edge enhancement effect similar to that in the work color space in the output color space. This is because the color difference component does not include a high-frequency edge structure so much, and there are many gently changing edge structures, so that it is difficult to extract the edge component. Therefore, the edge emphasis rate cannot be dynamically manipulated like the luminance component, and the edge emphasis of only the portion that can maintain the contrast without degrading the edge emphasis effect is loosened.

  Specifically, normalization is performed at the saturation reference point in the region where the output gamma characteristic becomes the knee characteristic, as in the case of the noise removal rate. As a result, the color difference surface edge enhancement ratio in the highlight portion is weakened. The highlight part generally corresponds to the high-saturation part in the color difference plane, so it is easier to extract edge components, and the work color space has a higher contrast than the output color space. Occurs. Therefore, by reducing the edge enhancement rate in the highlight area, extreme saturation enhancement in the high saturation area is prevented, and the overall color reproducibility is maintained while maintaining the balance between color contrast and colorfulness restoration in other dark areas. Produces a high edge enhancement effect. Also, combined with the dark edge enhancement effect of the luminance component, the black floating due to the noise in the dark area and the color blur from other areas to the dark area completely disappear, and the black reproducibility is higher than the original image, and the clearness is good And create transparency.

色差成分の場合

The functional expression by the contrast ratio function of the edge enhancement rate that realizes these is expressed by the following equation. Similar to the noise removal rate, in the case of the luminance component, normalization is performed by substituting the contrast ratio based on the exposure reference point into the denominator of the following equation. In the case of color difference components, normalization is performed by substituting the contrast ratio based on the saturation reference point into the denominator of the following equation.
In case of luminance component

For color difference components

2) Functional expression 2 (Retinex version)
The functional representation of the edge enhancement rate can also be made from another viewpoint. In other words, while the functional expression 1 aims to homogenize the edge enhancement effect between the brightness levels of the entire image, here, the brightness level between the brightness levels within the vicinity of the local edge structure. The goal is to homogenize the edge enhancement effect. This is because important parts that determine the image structure are concentrated in the vicinity of the edges, and even if there is an edge enhancement effect that is uniform with respect to brightness, the edge enhancement effect as a whole image is non-uniform. However, it is based on the idea that another aspect of the homogenizing action can be exerted to the maximum extent in the local region.

  In other words, the homogenization of the edge enhancement effect is another aspect where the negative effects of noise amplification due to noise components that could not be separated from the edge components are homogenized between brightness levels, so that the negative effects are strongly visible. This means that the visual defect of non-uniformity that deteriorates the overall impression is reduced, and the appearance of harmful effects is minimized on average. Therefore, if the detrimental measures for edge enhancement are implemented mainly in the vicinity of the edge, the contrast of the important structural portion of the image can be effectively protected from the influence of noise.

  In this case as well, for the output color space with general gradation characteristics, the contrast of the noise component mixed in the area becomes stronger as the gradation contrast of the work color space is higher in a bright place. It is easy to cause adverse effects of noise covering in that area. In the case of the luminance component, the structure information in the white image region plane is easily buried in noise, and in the case of the color difference component, the color information in the high saturation portion is easily buried in noise.

  Therefore, the edge enhancement rate may be expressed as a functional function of the contrast ratio function by the following equation. As edge information to be referred to at this time, it is optimal to use a multi-resolution integrated edge component for observing the image structure at any scale both locally and globally. Further, it is best to use the integrated edge component Ew ″ (x, y) in which a noise component and a ringing component are firmly excluded and a visually important frequency band is added.

色差成分の場合

In case of luminance component

For color difference components

  Here, the value of σgE th may be given as a value linked to the noise fluctuation index value σth in real space, or an absolute edge strength level. As a result, it is possible to give an edge strength level at which contrast reduction due to the influence of noise coverage is desired to be prevented.

3) Functional expression 3 (composite version)
The functional expressions 1 and 2 of the edge enhancement rate are also locally different in effect. Therefore, the usage which combined two methods is also considered. In that case, a functional expression may be basically obtained by taking the product of two functional expression parts.

16-3. Execution of noise removal processing and edge enhancement processing In this way, using the functional representation of the noise removal rate and edge enhancement rate that maximizes the effects of noise removal and edge enhancement and minimizes the effects, noise is actually reduced. Removal processing and edge enhancement processing are performed.

２）エッジ強調処理だけの場合
エッジ強調処理だけの場合、図４の処理(0-9)を行わず、次式によりエッジ強調のみを行い(不図示)、エッジ強調後の画像を出力する(不図示)。

３）ノイズ除去処理とエッジ強調処理を同時に行う場合
ノイズ除去処理とエッジ強調処理を同時に行う場合、上記ノイズ除去処理だけの場合の処理(処理(0-9))を行った後、エッジ強調処理を行い(処理(0-11))、ノイズ除去およびエッジ強調後の画像を出力する(処理(0-12))。ノイズ除去処理とエッジ強調処理の両方を行う場合の式をまとめると次式のようになる。

1) In the case of only noise removal processing In the case of only noise removal processing, noise removal is performed according to the following equation (processing (0-9)), and the image after noise removal is output (processing (0-10)).

2) In the case of only the edge enhancement processing In the case of only the edge enhancement processing, the processing (0-9) in FIG. 4 is not performed, only the edge enhancement is performed by the following formula (not shown), and the image after edge enhancement is output ( Not shown).

3) When noise removal processing and edge enhancement processing are performed simultaneously When noise removal processing and edge enhancement processing are performed simultaneously, the edge enhancement processing is performed after the processing (processing (0-9)) only for the above noise removal processing is performed. (Processing (0-11)), and output the image after noise removal and edge enhancement (Processing (0-12)). The formulas for performing both noise removal processing and edge enhancement processing are summarized as follows.

のように設定するのがよい。そうするとλjointのノイズ除去率を上げるに伴ってノイズ除去効果が強くなっていく画質設計を保ったまま、ノイズ除去で失ってしまうノイズに埋もれたテキスチャをも復元しうる高性能なノイズ除去が可能となる。 Here, when performing edge enhancement for the purpose of recovering the outline blurring that has suffered a loss in noise removal,

It is better to set as follows. As a result, it is possible to perform high-performance noise removal that can restore the texture buried in the noise lost by noise removal while maintaining the image quality design that the noise removal effect becomes stronger as the noise removal rate of λjoint increases Become.

  On the other hand, if it is desired to remove only the unclearness due to noise in the original image of each ISO sensitivity without removing noise, λjoin = 0 and ζjoin ≠ 0 may be set.

  Here are some points to note. The optimum solution of the functional representation of the noise removal rate and the optimum solution of the functional representation of the edge enhancement rate, which have been elucidated by following the physical phenomenon as described above, coincide by chance. From the viewpoint of inseparable noise components contained in the extracted edge components, the functional representation of the edge enhancement rate is a function of the functional representation of the denoising rate because of fear of amplification of the noise components mixed in the edge component. A logic setting to the reciprocal can also be considered. However, the experimentally matched expression gives better results because the edge component is treated as a pure edge component by eliminating the noise component from the extracted edge component and bringing it closer to the true edge component. It can be said that it is proof that there is no need to worry about amplification of noise components. Therefore, it is possible to save the trouble of creating an individual functional expression for each processing. In addition, the physical meaning of this is not related to the fact that an edge structure lost by noise removal becomes the same functional form when trying to return it to the original with exactly the same behavior. However, although noise removal and edge enhancement are completely independent events, they are consistent with respect to the gradation direction. The frequency projection space of the other step S10 is inconsistent.

17. Conversion to Output Color Space Next, in step S4 of FIG. 2, the image from which noise has been removed and edge-enhanced is converted from the image processing space to the output color space. If the output color space may be the same as the input color space, the inverse conversion process for item number 1 is performed. If they are different, they may be converted in accordance with the standard color space of each input and output. For example, a case where the input is AdobeRGB and the output is sRGB can be considered. In addition to that, gradation correction may be further applied to the output image. For example, the characteristic of the gamma curve applied to the image is changed. At that time, the characteristic information is transmitted to the item numbers 16-1 and 16-2 in advance so that the contrast ratio function can be calculated in advance.

18. Image Output In step S5 of FIG. 2, the image data from which noise has been removed and edge-enhanced as described above is output.

  The following image quality effect is produced on the image subjected to the edge enhancement processing as described above. First, the contrast of the edge is reduced due to high-sensitivity photography and noise coverage, the sharpness of the image is restored for the image that is black, and the texture structure with the same amplitude as the noise is firmly contrasted. In addition to improving the sharpness and three-dimensionality of the image, it improves the blackness and eliminates the cloudiness caused by the noise, and enhances the gradation expression to enhance transparency and eliminate color boundary bleeding and increase the color contrast. , The effect of raising the colorfulness by eliminating the desaturation due to the overall noise coverage.

  Similarly, the effect of restoring the edge structure lost by noise removal can be obtained by simultaneously performing the edge enhancement processing on the image from which noise has been removed. In other words, the reduction in sharpness due to noise removal is recovered, the graininess noise is reduced while restoring the texture that changes to the same extent as the noise lost by noise removal, and the noise in the dark part is suppressed, and the bright part is suppressed. In addition, it is possible to improve gradation and prevent color spot noise with less blurring and disappearance of a gradual color structure on a flat portion.

  In addition, image enhancement by edge enhancement and noise suppression by noise removal can be continuously realized while maintaining natural image quality according to their strength, that is, the strength of the edge enhancement rate and the noise removal rate. . In other words, the enhancement effect with good image quality that surpasses the original image for high-sensitivity shot images ensures that it is done in a natural way close to that of low-sensitivity shot images and does not step out of it. .

  Therefore, an edge with high reproducibility of physical quantities as expressed by resolving power, color resolution, and effective gradation bandwidth as well as subjective reproducibility such as sharpness, sharpness, stereoscopic effect, and transparency. Emphasis and noise removal effects can be obtained.

  Note that noise is always present in low-sensitivity shot images, and it has been experimentally confirmed that these enhancement effects can be obtained in the same manner.

-Second embodiment-(real space version)
In the second embodiment, an embodiment is shown in which noise removal and edge enhancement are simultaneously performed in the real space plane.

  Since the configuration of the image processing apparatus of the second embodiment is the same as that of the first embodiment, a description thereof will be omitted with reference to FIG. The flowchart of the image processing of the second embodiment processed by the personal computer 1 is also the same as that shown in FIG. Hereinafter, a description will be given focusing on differences from the processing of the first embodiment. FIG. 8 is a flowchart illustrating noise removal processing and edge enhancement processing according to the second embodiment.

1. Color space conversion Same as item number 1 in the first embodiment.

2. 2. Noise extraction by virtual noise removal 2-1. Noise removal processing Since the real space plane is represented by S (x, y), the processing for the subband plane V (x, y) of the first embodiment need only be replaced from V to S (processing ( x-1)). However, it is necessary to increase the integration range.

As a noise removal filter, any method of noise removal processing for creating the smoothed surface S ′, such as a σ filter or an ε filter, may be used, but an example of an improved bilateral filter as in the first embodiment. Let me show you.

  To get a really clean noise removal effect, it is better to set the rth value to about 50 and the filtering range to about 101x101 pixels, but here for the sake of simplification of explanation, the integration is done in the range of 25x25 for rth = 12 And However, in the case of the σ filter and the ε filter, there is no parameter corresponding to rth because it is indifferent to spatial factors, and the integration range is simply set.

2-2. Noise extraction processing Noise extraction processing as shown in the following equation is performed (processing (x-3)).

3. Edge extraction Edge components are extracted from the real space surface from which virtual noise is removed (processing (x-3)).

Here, a Laplacian filter is used as the edge detection filter. The same 9 × 9 Laplacian filter as in the second embodiment may be used, but since the smoothing filter is set to 25 × 25, it is desirable to set this also to about 25 × 25. A 25x25 Laplacian can be created by taking the difference between the original image and a smoothed image that has been multiplied by a 9x9 Gaussian filter three times. That is,

4). Self-refining of noise components Self-refining of noise components is performed using the same equation as item number 11 in the first embodiment (processing (0-1)). However, replace Nw with N.

5. Self-refining of edge components Self-refining of edge components is performed using the same equation as item number 12 in the first embodiment (processing (0-2)). However, replace the Ew symbol with E.

6). Refinement of noise components by edge (mutual refining 1)
Using the same equation as item number 13 in the first embodiment, noise components are refined by edges (processing (0-3)). However, replace the Nw symbol with N and the Ew symbol with E.

7). Refinement of edge components by noise (mutual refining 2)
Using the same equation as item number 14 in the first embodiment, the edge component due to noise is refined (processing (0-4)). However, replace the Nw symbol with N and the Ew symbol with E.

8). Actual noise removal processing and actual edge enhancement processing The same formula as item number 16 in the first embodiment is used (processing (0-7) (0-8) (0-9) (0-11)). However, replace the Nw symbol with N and the Ew symbol with E. For convenience, processing corresponding to item number 15 in the first embodiment is omitted.

9. Conversion to output color space Same as item number 17 in the first embodiment.

  The image processing result obtained in this way has high noise component and edge component separation performance as in the first embodiment, except for the effect of the optimal selection of the frequency projection space of the first embodiment. In addition, it is possible to obtain high-quality noise removal and edge enhancement effects that minimize the adverse effects caused by the impurity components contained therein.

-Modification-
(1) In the above-described embodiment, it is assumed that both noise removal processing and edge enhancement processing exist, and whether or not to use them is described in a form of free selection at the end, but only the edge enhancement processing is performed at a higher speed. In the first embodiment, the noise removal filter is omitted, and in the first embodiment, the second processing is directly performed from the subband image expressed in multi-resolution. In the embodiment, extracting the edge component directly from the original image by unsharp mask processing is also considered as a simple usage.

(2) It should be noted that the self-reciprocal refining of the noise components and edge components of item numbers 5 to 8 and 11 to 15 of the first embodiment and the functional representation of the noise removal rate and edge enhancement rate of item number 16 It should be pointed out that the computation takes almost negligible processing time because it only passes through the lookup table when actually processed by software.

(3) Although the uniform color / uniform noise space is used as the best color space for the image processing space described above, the contrast ratio function is similarly defined in the case of a general uniform color space, and the present invention is used. Can be used. For example, the CIE-defined L * a * b * space, L * u * v *, or CIECAM02 may be used, and the contrast ratio function can be calculated from the gradation characteristics of the work color space and the gradation characteristics of the output color space defined by each. You can ask for.

(4) In the first embodiment described above, an example of using a high-performance improved bilateral filter as the edge-preserving smoothing filter is shown. However, although the degree of separation between edges and noise is inferior, it is fast and simple. The following Laplacian noise removal filter shown in WO2006 / 106919 of the applicant's invention is an example of a noise removal filter that operates in the following manner. In this case, the self-reciprocal and mutual refining functions of the noise component and the edge component defined in the embodiment are more effective than the improved bilateral filter because the purity of each of the original extracted components is poor. Demonstrate.

色差面のノイズ除去処理

Luminance surface noise removal processing

Color difference surface noise removal processing

  However, the Laplacian defined here can be the simplest 3x3 when using multiple resolutions.

(5) In the first embodiment described above, in the case of the 5-stage wavelet transform, the resolution levels of the low-frequency subband image group and the high-frequency subband image group are set as j = 1, 2,. Although an example of using it is shown, the real space can be handled as the resolution of j = 0 of the low-frequency subband image group. In that case, noise extraction processing and edge extraction processing as performed in j = 1 may be performed and added to each of the integrated noise component and the integrated edge component at the final stage. The weighting between the resolution levels at that time is knj = 1 because the noise component is white noise with respect to j = 0, and the value of kej is item number 10-2. It is good to use the numerical value already described.

(6) It should be noted that the image processing space described in the above embodiment is merely an example of the best color space, and even if noise removal and edge enhancement processing are performed in the conventional color space, the significance of the present invention is not found. It will not fade. For example, CIECAM02, which is the latest uniform color space, may be used. This space may be either a uniform noise space or a uniform color space.

(7) In the above embodiment, in the self-refining of the edge component, calculation is performed by an exponential function so that the frequency distribution related to the strength of the edge component approaches a Gaussian distribution with a predetermined width based on the noise fluctuation index value σth ij. However, this processing may be threshold determination based on the following equation instead of Gaussian distribution. Similarly, threshold determination may be performed in edge component refining by noise, noise self-refining, and noise component refining by edge.

(8) In the above embodiment, the example of the image processing apparatus realized by the personal computer 1 has been described. However, the noise removal processing by the personal computer 1 described above may be performed in a digital camera (electronic camera). FIG. 9 is a diagram showing the configuration of the digital camera 100. The digital camera 100 includes a photographic lens 102, an image sensor 103 including a CCD, a control device 104 including a CPU and peripheral circuits, a memory 105 and the like.

  The image sensor 103 captures (captures) the subject 101 via the photographing lens 102 and outputs the captured image data to the control device 104. This process corresponds to the image data input in step S1 of FIG. 3 described in the first embodiment. The control device 104 performs the noise removal processing of each of the embodiments and modifications described above on the image data captured by the image sensor 103, and appropriately stores image data that has been appropriately noise-removed and edge-enhanced. It stores in 105. The control device 104 performs the above-described noise removal processing and edge enhancement processing by executing a predetermined program stored in a ROM (not shown) or the like.

  In this way, even in the digital camera 100, it is possible to prevent ringing in edge enhancement and to produce a natural edge enhancement effect. Image data with appropriate edge enhancement is stored in the memory 105, and can be detached. It can be recorded on a recording medium such as a memory card.

The actions and effects of the present embodiment and the modifications described above are summarized as follows.
(1) An image processing method that performs edge enhancement of an original image generates a set of at least one low frequency band limited image and a high frequency band limited image from the original image, and applies an edge extraction filter to each band limited image Extract each low frequency edge component and high frequency edge component, generate one edge component by combining the low frequency edge component and the high frequency edge component, and perform edge enhancement of the original image based on the generated edge component The composition ratio of the low frequency edge component and the high frequency edge component is changed according to the strength of edge enhancement. Changing the composition ratio means that the above-described weight kei (i = LL, LH, HL, HH) is made different, and is represented by Expressions 23 and 24 in the item number 10-1.

(2) At this time, when the original image is a luminance component, the synthesis ratio of the low-frequency edge component to the high-frequency edge component is decreased as the edge enhancement strength is increased. In Expression 23, the weight keLL of the low frequency edge component changes in conjunction with ζjoint, and the fixed values 1 are set for the weights keLH, keHL, and keHH of the other high frequency edge components. Accordingly, as the edge enhancement strength (ζjoint) is increased, the synthesis ratio of the low frequency edge component to the high frequency edge component is decreased.

(3) Further, when the original image is a color difference component, the synthesis ratio of the high frequency edge component to the low frequency edge component is lowered as the edge enhancement strength is increased. In Formula 24, the weights keLH, keHL, and keHH of the high-frequency edge components change in conjunction with ζjoint, and the fixed value 1 is set for the weight keLL of the low-frequency edge component. Accordingly, as the edge enhancement strength (ζjoint) is increased, the synthesis ratio of the low frequency edge component to the high frequency edge component is decreased.

(4) Further, the low frequency edge component and the high frequency edge component are set to the same composite ratio as the edge enhancement strength is reduced. When the edge emphasis strength ζjoint varies between 0 ≦ ζjoint ≦ 1, and the noise removal strength λjoint is also sufficiently small, when the edge emphasis strength ζjoint decreases, the value approaches 1 and In both 23 and 24, the weights of the low-frequency edge component and the high-frequency edge component are all close to 1 and have the same synthesis ratio.

(5) When a noise component is extracted from the original image and noise removal of the original image is performed together with edge enhancement based on the extracted noise component, the synthesis ratio of the low frequency edge component and the high frequency edge component is further reduced. Changed according to strength. This means that the noise removal rate λjoint is also taken into consideration in Expressions 23 to 26 of the item number 10-1.

(6) At this time, when the original image is a luminance component, the synthesis ratio of the low-frequency edge component to the high-frequency edge component is decreased as the noise removal strength is increased. This means that, for example, if the value of the noise removal rate λjoint is increased between 0 and 1 in Equation 23 of item number 10-1, the value of keLL decreases and the synthesis ratio decreases.

(7) When the original image is a color difference component, the synthesis ratio of the high-frequency edge component to the low-frequency edge component is lowered as the noise removal strength is increased. This means that, for example, when the value of the noise removal rate λjoint is increased between 0 and 1 in Equation 24 of item number 10-1, the value of keLL decreases and the synthesis ratio decreases.

(8) At this time, the low-frequency edge component and the high-frequency edge component are set to the same synthesis ratio as the noise removal strength is reduced. When the noise removal rate λjoint varies between 0 ≦ λjoint ≦ 1, when the edge enhancement strength ζjoint is also sufficiently small, the value approaches 1 when the noise removal strength λjoint decreases, In both cases, the weights of the low-frequency edge component and the high-frequency edge component are all close to 1 and have the same composition ratio.

(9) An image processing method for removing noise from an original image generates a set of at least one low frequency band limited image and a high frequency band limited image from the original image, and applies a noise removal filter to each band limited image. Extract each low frequency noise component and high frequency noise component, generate one noise component by synthesizing the low frequency noise component and high frequency noise component, and remove the noise of the original image based on the generated noise component The synthesis ratio of low frequency noise component and high frequency noise component is changed according to the strength of noise removal.

(10) At this time, when the original image is a luminance component, the synthesis ratio of the low-frequency noise component to the high-frequency noise component is lowered as the noise removal strength is reduced.

(11) When the original image is a color difference component, the synthesis ratio of the high-frequency noise component to the low-frequency noise component is lowered as the noise removal strength is reduced.

(12) Further, as the strength of noise removal is increased, the low-frequency noise component and the high-frequency noise component are set to the same synthesis ratio.

(13) An image processing method for performing edge enhancement of an original image inputs an original image composed of a plurality of pixels, decomposes the input original image, and sequentially generates a low-frequency image having a low resolution and sequentially Generate a high-frequency image with a low resolution, apply an edge extraction filter to each of the low-frequency image and the high-frequency image to extract the edge component, generate a corresponding low-frequency edge component and high-frequency edge component, and generate the generated low-frequency image Multiply at least one of the frequency edge component and the high frequency edge component by the weighting coefficient to modulate the weight between the frequency bands of the edge component, and synthesize the modulated low frequency edge component and the high frequency edge component to sequentially increase the resolution. Is integrated into one edge component in sequence, the integrated edge component is multiplied by the edge enhancement factor to adjust its strength, and the adjusted edge component is added to the original image Performs edge enhancement of the original image, the value of weighting factor to modulate the weights between the frequency band of the edge component, was changed in accordance with the edge enhancement rate. The weighting coefficient here corresponds to the weight described above. Adjustment here means to increase or decrease the edge component in accordance with the edge enhancement rate.

(14) At this time, the value of the weighting coefficient for modulating the weight between the frequency bands of the edge component is increased, and the difference in the weight between the low frequency edge component and the high frequency edge component is widened as the edge enhancement rate value is small. The larger the rate value, the smaller the difference. In Expression 23, the weight keLL of the low frequency edge component changes in conjunction with ζjoint, and the fixed values 1 are set for the weights keLH, keHL, and keHH of the other high frequency edge components. Therefore, when the edge enhancement rate ζjoint varies between 0 ≦ ζjoint ≦ 1, the smaller the value of the edge enhancement rate (ζjoint), the wider the difference in weight between the low frequency edge component and the high frequency edge component, and the edge enhancement rate As the value of (ζjoint) increases, the difference is reduced. The same can be said for Formula 24.

(15) When the original image is a luminance component, the weighting coefficient value for modulating the weight between the frequency bands of the edge component is set such that the weight of the low frequency edge component is set smaller as the value of the edge emphasis rate increases, The weight of the low frequency edge component is set to be larger as the value of the edge enhancement rate is larger.

(16) When the original image is a color difference component, the weighting coefficient value for modulating the weight between the frequency bands of the edge component is set such that the higher the edge enhancement rate, the smaller the high frequency edge component weight is set. The weight of the high frequency edge component is set larger as the rate value increases.

(17) An image processing method for removing noise contained in an original image is a method of inputting an original image composed of a plurality of pixels, decomposing the input original image, and sequentially generating a low-frequency image having a low resolution and sequentially A high-frequency image with a very low resolution, extracting noise components contained in each of the low-frequency image and the high-frequency image, generating a low-frequency noise component and a high-frequency noise component corresponding to each, and generating the generated low-frequency A weighting factor is applied to at least one of the noise component and high-frequency noise component to modulate the weight between the frequency bands of the noise component, and the modulated low-frequency noise component and high-frequency noise component are combined to successively increase the resolution. It integrates one noise component in sequence, multiplies the integrated noise component by a noise removal rate to attenuate its intensity, and subtracts the attenuated noise component from the original image. Performs noise removal from the value of the weighting factor to modulate the weights between the frequency band of the noise component, was changed in accordance with the noise removal rate.

(18) At this time, the weighting coefficient value for modulating the weight between the frequency bands of the noise component is increased so that the difference in the weight between the low frequency noise component and the high frequency noise component is increased as the noise removal rate value is decreased. The larger the rate value, the smaller the difference.

(19) Further, when the original image is a luminance component, the weighting coefficient value for modulating the weight between the frequency bands of the noise component is set such that the weight of the low-frequency noise component is decreased as the noise removal rate value is decreased. The weight of the low frequency noise component is set to be larger as the noise removal rate is larger.

(20) When the original image is a color difference component, the weighting coefficient value for modulating the weight between the frequency bands of the noise component is set so that the weight of the high-frequency noise component is set smaller as the noise removal rate value is smaller. The weight of the high frequency noise component is set larger as the rate value increases.

(21) Filtering the original image to sequentially generate a plurality of band-limited images having a low resolution, extracting an edge component from each band-limited image, and for the extracted edge components of each band-limited image, By combining weights between resolutions, they are integrated into one edge component, and based on the integrated edge components, edge enhancement of the original image is performed. The center of gravity was changed. This is represented by Equations 27 and 29 in item number 10-2. According to Equation 29, μ changes according to the strength of edge enhancement, and kej changes according to Equation 27 according to the changing μ. The calculation results are shown in Equations 28 and 30. That is, when the edge enhancement rate ζjoint changes from 1 ≦ ζjoint ≦ ∞, the gravity center position of the weight changes so as to change from Equation 28 to Equation 30.

(22) Specifically, it is shown that the position of the center of gravity of the weight is shifted to the high resolution side as the strength of edge enhancement increases.

(23) In this case, in an area where the edge enhancement is low, the intermediate resolution is combined with the weight between the resolutions having the weight centroid.

(24) As shown in Equation 27, the distribution function of the weight between resolutions uses a Poisson distribution.

(25) Expressions 27 and 29 mean that the distribution characteristics are changed so that the average value of the Poisson distribution is monotonously moved toward the high resolution side as the strength of edge enhancement increases.

(26) When a noise component is extracted from the original image and noise removal of the original image is performed together with edge enhancement based on the extracted noise component, the weight centroid between the resolutions of the edge component is further calculated and the noise removal strength is further increased. I changed it according to. This means that λjoint is also considered in Equation 29.

(27) In this case, as expressed by Equations 27 to 30, as the noise removal strength λjoint increases, the center of gravity position is shifted to the high resolution side.

  With the above configuration, processing with consideration of the difference in image quality effect between the low-frequency image and the high-frequency image is performed in noise removal or edge enhancement with a simple configuration, and an image with optimum image quality can be provided.

  In particular, by clarifying the role of image quality effect in each of noise reduction and edge enhancement of multi-resolution low frequency subband and high frequency subband, and incorporating the relationship between parameters that give the optimum image quality based on that role, The number of parameters opened to the user can be reduced to facilitate handling.

  And, by realizing the usage ratio of low frequency subband and high frequency subband in noise removal in conjunction with the noise removal rate, it realizes the function that maximizes the original noise removal effect and harmful measures while reducing the number of parameters It becomes possible to do. Similarly, in the case of edge enhancement, it has been realized in conjunction with the edge enhancement rate based on the image quality improvement effect achieved by each low frequency subband and high frequency subband. It is possible to provide an interface function and an edge enhancement process that maximizes the image restoration effect.

  Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other embodiments conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention. Various combinations of the above-described embodiments and modifications are also valid as long as they are within the scope of the technical idea of the present invention.

DESCRIPTION OF SYMBOLS 1 Personal computer 2 Digital camera 3 Recording medium 4 Computer 5 Electrical communication line 11 CPU
DESCRIPTION OF SYMBOLS 12 Memory 13 Peripheral circuit 100 Digital camera 101 Subject 102 Shooting lens 103 Image pick-up element 104 Control apparatus 105 Memory

Claims (10)

Input an original image consisting of multiple pixels,
The input original image is decomposed, and a set consisting of a series of a low-frequency image having the lowest resolution and a high-frequency image having a plurality of resolutions that can completely reconstruct the original image, and a series of low-frequency images having a plurality of resolutions. to redundant frequency space representation than that equivalent representing an original image using both sets, generating a plurality of high-frequency images with a plurality of low-frequency images with successively lower resolution, sequential low resolution And
An edge extraction filter is applied to each of the low frequency image and the high frequency image to extract an edge component, and a low frequency edge component and a high frequency edge component corresponding to each are generated,
Modulating the weight between the frequency bands of the edge component by multiplying at least one of the generated low frequency edge component and high frequency edge component by a weighting coefficient,
Combining the modulated low frequency edge component and high frequency edge component sequentially into one edge component with high resolution,
Multiplying the integrated edge component by an edge enhancement rate arbitrarily set by the user to adjust its strength,
By adding the adjusted edge component to the original image, edge enhancement of the original image is performed,
An image processing method, wherein a value of a weighting coefficient for modulating a weight between frequency bands of the edge component is changed according to the edge enhancement rate. The image processing method according to claim 1 ,
As the value of the weighting coefficient for modulating the weight between the frequency bands of the edge components increases, the difference in weight between the low frequency edge component and the high frequency edge component increases as the value of the edge enhancement rate increases, and the value of the edge enhancement rate decreases. An image processing method characterized by changing so as to reduce the difference. The image processing method according to claim 1 ,
When the original image is a luminance component, the weighting coefficient value for modulating the weight between the frequency bands of the edge component is set such that the weight of the low-frequency edge component is set smaller as the value of the edge enhancement ratio increases, An image processing method, wherein the weight of the low frequency edge component is set larger as the value of becomes smaller. The image processing method according to claim 1 ,
When the original image is a color difference component, the weighting coefficient value for modulating the weight between the frequency bands of the edge component is set such that the higher the edge enhancement factor, the smaller the high frequency edge component weight, An image processing method, wherein the weight of a high-frequency edge component is set larger as the value is smaller. Input an original image consisting of multiple pixels,
The input original image is decomposed, and a set consisting of a series of a low-frequency image having the lowest resolution and a high-frequency image having a plurality of resolutions that can completely reconstruct the original image, and a series of low-frequency images having a plurality of resolutions. to redundant frequency space representation than that equivalent representing an original image using both sets, generating a plurality of high-frequency images with a plurality of low-frequency images with successively lower resolution, sequential low resolution And
Extracting the noise component contained in each of the low frequency image and the high frequency image, and generating a low frequency noise component and a high frequency noise component corresponding to each,
A weighting coefficient is applied to at least one of the generated low frequency noise component and high frequency noise component to modulate a weight between frequency bands of the noise component,
Combining the modulated low frequency noise component and high frequency noise component sequentially into one noise component with high resolution sequentially,
Multiplying the integrated noise component by a noise removal rate arbitrarily set by the user to attenuate the intensity,
By subtracting the attenuated noise component from the original image, noise is removed from the original image,
An image processing method, wherein a value of a weighting coefficient for modulating a weight between frequency bands of the noise component is changed according to the noise removal rate. The image processing method according to claim 5 ,
The value of the weighting coefficient that modulates the weight between the frequency bands of the noise component, the smaller the noise removal rate value, the wider the weight difference between the low frequency noise component and the high frequency noise component, the larger the noise removal rate value An image processing method characterized by changing so as to reduce the difference. The image processing method according to claim 5 ,
When the original image is a luminance component, the weighting coefficient value for modulating the weight between the frequency bands of the noise component is set such that the weight of the low frequency noise component is set smaller as the noise removal rate value is smaller. An image processing method characterized in that the weight of the low frequency noise component is set larger as the value of becomes larger. The image processing method according to claim 5 ,
When the original image is a color difference component, the weighting coefficient value for modulating the weight between the frequency bands of the noise component is set such that the weight of the high frequency noise component is set smaller as the noise removal rate value is smaller. An image processing method, wherein the weight of a high-frequency noise component is set larger as the value becomes larger. The image processing program to be executed by the image processing method a computer or an image processing apparatus according to any one of claims 1 to 8. An image processing apparatus in which the image processing program according to claim 9 is installed.

Images