JP2013165482A - Image processor, image processing program and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform highly precise interpolation which provides an interpolation image with high contrast while performance of a conventional image prediction space where contrast resolution which is superior on average can be performed in the whole gradation region is kept in a maximum.SOLUTION: An image processor generates three color interpolation images by using a prediction space by three interpolation gammas Γ, Γand Γ, sets a color interpolation image by the interpolation gamma Γ, which is positioned in the middle, as a reference, extracts a laplacian component between hierarchies in the same pixel from the three color interpolation images and adds it to a reference image.

Description

  The present invention relates to an image processing device, an image processing program, and an imaging device.

  Image data captured with a single-plate image sensor such as a Bayer array is usually input as linear gradation data having gradation characteristics proportional to the amount of light. As a method for performing color interpolation on this data, using the fact that the world's subject image is close to an achromatic color on average, interpolation is performed in consideration of other color components when interpolating a conventional color component. Technology is used.

  For example, in Patent Document 1, an interpolation value based on an average value of a certain color component is other than the linear Bayer input Bayer data so that the color difference is constant, that is, R−G = constant and B−G = constant. Discloses a method of calculating an interpolation value while adding a correction term based on the color components. As another method, the interpolated value is corrected by correcting the interpolated value based on the average value of one color component with other color components so that the color ratio is constant, that is, R / G = constant and B / G = constant. A method of calculating is also disclosed.

  As shown in Patent Document 2, an interpolation method with a constant color ratio shifts an image having a linear gradation characteristic to a gamma space having a logarithmic gradation characteristic, and sets an interpolation value so as to maintain a constant color difference in the image processing space. Equivalent to calculating. This is not limited to logarithmic gradation characteristics. If interpolation is performed so that the color difference is kept constant by shifting to some interpolation gamma space, the interpolation value prediction and the color ratio are assumed to be constant in the actual linear gradation space. This means that an arbitrary prediction space that passes between the predicted interpolation value predictions can be changed by the setting method of the gradation characteristic of the interpolation gamma.

  Methods for optimizing the gradation characteristics of such interpolation gamma and providing a space for predicting the best interpolation value are disclosed in Patent Documents 3 and 4 by the same inventor as the present applicant. In other words, using the assumption that the color difference is constant, in order to obtain clear contrast and resolution in all areas of the low-brightness, halftone, and high-brightness areas in an arbitrary image, a square root gradation gamma space for interpolation is used. Point out that it is better to move to Furthermore, when dark noise is present in high-sensitivity shooting, a technique has been introduced to always maintain uniform noise in an interpolating gamma space with a positive offset added to the square root. Since these square root image processing spaces provide a uniform noise space, it realizes an optimal interpolation value prediction space with a constant color difference, maximizes the direction judgment performance during interpolation, and minimizes noise amplification of the interpolation result The effect is also realized at the same time.

US Pat. No. 5,552,827 US Pat. No. 4,642,678 Japanese Patent No. 4239483 US Pat. No. 7,957,588

  The logarithmic gradation characteristic gamma space described above provides a prediction space that places great importance on the contrast reproduction of the low-brightness part, while the high-brightness part has a knee characteristic and the gradation is slanted. Contrast reproducibility has the disadvantage of a very low prediction space. On the other hand, interpolation with a constant color difference with linear gradation as disclosed in Patent Document 1 provides a prediction space that places the highest importance on contrast reproduction in high-luminance areas when linear gradation is considered as one of the gamma spaces for interpolation. The gradation area assigned to the low luminance part and the halftone is extremely narrow, and provides an image processing space in which the contrast reproduction capability of the area is extremely low. That is, if importance is placed on contrast reproduction as an interpolation gamma space, there is always a trade-off relationship in which some contrast reproduction is sacrificed. The solution shown to average the best contrast of all gradation areas from low luminance part to halftone and high luminance part as one gamma space for interpolation is the square root type. The gradation space.

  However, although the square root type gradation space provides a high-quality interpolated image quality having the best contrast on average as a prediction space by one interpolation gamma, depending on the image, a prediction space by another interpolation gamma is tried a little more. As a result, there has been a problem that the potential of being able to obtain clearer image quality with higher contrast cannot be maximized. That is, as an overall impression of the image, there is an image in which there is a general tendency that the image is covered or cloudy, the dark portion is slightly tightened, or the sharpness is slightly insufficient.

(1) The invention according to claim 1 converts an image signal in an input color space into an image signal in an image processing space for image data including at least two color components and having a missing color component, An image processing apparatus that performs interpolation processing of missing color components in an image processing space and converts the image signal after the interpolation processing into an image signal in an output color space, at least two floors having different first and second curvatures A tone converting unit that converts the image signal into a tonal characteristic, a color interpolating unit that generates at least two types of interpolated images by interpolating the missing color component of each tone-converted image signal, and at least two types A tone matching unit that converts an interpolated image into an image in an image processing space having a single tone characteristic to match the tone characteristics, and a tone matching that is interpolated using the first and second tone characteristics and that matches one tone characteristic. Using the first and second interpolated images A difference value extraction unit that extracts a difference value between at least a pixel value of the second interpolation image and a pixel value of the first interpolation image between the same pixels of the two images; And an addition unit for adding the difference value to the same pixel.
(2) The invention according to claim 4 converts an image signal in the input color space into an image signal in the image processing space for image data that includes at least two color components and includes missing color components, An image processing apparatus that performs interpolation processing of missing color components in an image processing space and converts the image signal after the interpolation processing into an image signal in an output color space, and the curvatures are different in the first, second, and third order. A gradation conversion unit that converts the image signal into three gradation characteristics, a color interpolation unit that generates three types of interpolated images by interpolating the missing color components of the gradation-converted image signals, and three types The interpolated image is converted into an image in an image processing space having one gradation characteristic, and a gradation matching unit that adjusts the gradation characteristic is interpolated with the first to third gradation characteristics and adjusted to one gradation characteristic. Using the combined first to third interpolated images, the same of the three images A Laplacian extraction unit that extracts a Laplacian component between the pixel value of the first interpolation image and the pixel value of the third interpolation image with respect to the pixel value of the second interpolation image; On the other hand, an addition unit that adds a Laplacian component to the same pixel is provided.
(3) The image pickup apparatus according to the invention described in claim 9 performs image processing on an image pickup unit that picks up a subject image and an image picked up by the image pickup unit. And an image processing apparatus.

  According to the present invention, a high-definition image that provides a higher-contrast interpolation image while maintaining the performance of a conventional image prediction space capable of performing excellent contrast resolution on average in all gradation regions while maximizing the performance of the conventional image prediction space. Interpolation can be performed.

It is a figure which shows a Bayer arrangement | sequence. It is a figure which shows three gradation characteristics. It is a figure which shows three gradation characteristics. It is a figure which shows three gradation characteristics. It is a figure which shows three gradation characteristics. It is a figure explaining a computer apparatus. It is a flowchart explaining the flow of the image processing which CPU performs. It is a flowchart explaining the detail of the Bayer interpolation process with respect to a reference | standard image. It is a flowchart explaining the detail of the process which correct | amends the Laplacian component between gradations. It is a flowchart explaining the flow of an edge emphasis process. It is a block diagram which illustrates the composition of the camera carrying an image processing device. It is the figure which represented the luminance resolving power color resolving power in a Bayer arrangement | sequence in frequency space. It is a flowchart explaining the flow of the image processing which CPU performs. It is a flowchart explaining the detail of an internal weighting image synthetic | combination process.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In this description, an example of image processing for performing color interpolation on an image captured through a color filter in which R (red), G (green), and B (blue) are arranged in a Bayer array will be described. The color of the image is represented by the RGB color system.

  FIG. 1 is a diagram showing the Bayer array. An image signal output from an image sensor including such a color filter has information on any one color component of RGB per pixel. That is, the R color information is output from the pixel corresponding to the R color filter, the G color information is output from the pixel corresponding to the G color filter, and the B color information is output from the pixel corresponding to the B color filter. Is output. Taking pixel information corresponding to the R color filter as an example, there is no information on the G component and B component only by the information on the R component. For this reason, in order to obtain information for all the RGB colors at all pixel positions constituting the image, information on the color components that are insufficient at each pixel position is calculated by interpolation processing (Bayer interpolation).

<Three kinds of gradation spaces>
First, three interpolation gamma spaces used in the following description will be described. In this embodiment, three image processing spaces are prepared as interpolation gamma spaces, and three types of interpolation results are obtained. Specifically, the following three gradation spaces are defined as three gradation curves, and are named Γ ₁ , Γ ₂ , and Γ ₃ , respectively. Assume that the input signal is x, the output signal is y, and both the gradation of the input signal and the gradation of the output signal are defined in the range [0, 1]. The gradation curve (gamma curve) is defined so that the input / output characteristics pass through (x, y) = (0,0) and (1,1). If the maximum value of the actual input gradation X is Xmax and the maximum value of the output gradation Y is Ymax, x = X / Xmax, y = Y / Ymax, and gradation conversion is performed by the following equation (1). Is called.

(I) Linear gradation characteristics (Γ ₁ )
This gradation characteristic provides a prediction space having the capability of firmly decomposing the contrast structure of the high luminance part because most of the gradation widths after gradation conversion are assigned to the high luminance part region. The gradation characteristic of y = f (x) in the above equation (1) is y = x.

上式（２）における正のオフセット値εは、暗電流ノイズ成分が増える高感度の撮影条件になるほど大きな値を設定する。 (II) Square root tone characteristics with offset (Γ ₂ )
This gradation characteristic provides a prediction space that reproduces the contrast structure of all gradation areas in a natural way. In the early Munsell color space and Hunter's uniform perceptual color space, the gradation characteristic of the square root is assumed to be the human visual sensitivity characteristic. Therefore, it is an excellent gradation space that provides a uniform noise space while realizing a gradation space close to human visibility. The gradation characteristic of y = f (x) in the above equation (1) is the following equation (2).

The positive offset value ε in the above equation (2) is set to a larger value as the imaging condition with higher sensitivity in which the dark current noise component increases.

(III) Cubic root gradation characteristics, logarithmic gradation characteristics (Γ ₃ )
This tone characteristic provides a prediction space that has the ability to specifically resolve the contrast structure of the low-brightness part. The cubic root gradation characteristic is a space adopted as a gradation characteristic that realizes a uniform color space of CIE L * a * b *. As a human visual sensitivity characteristic, in addition to the gradation adopted in Γ ₂ , it can be expected to be decomposed at a contrast defined by Γ ₃ , and a gradation space having a contrast resolution corresponding to that is provided. The gradation characteristic of y = f (x) in the above equation (1) is the following equation (3).

Note that a logarithmic gradation space capable of strongly decomposing the contrast of the low luminance portion may be used instead of the above equation (3). In this case, the following equation (4) is used as the gradation characteristic of y = f (x) in the above equation (1).
y = log (kx + 1) / log (k + 1) (4)
However, k is a constant, and the value is set smaller as the imaging sensitivity increases. For example, when the linear gradation data is represented by 14 bits, the lowest ISO sensitivity takes a value such as k = 2 ¹⁴ = 16384. If the value of k is set to ½ as the ISO sensitivity is further increased, a tone prediction space without the risk of noise amplification can be provided.

FIG. 2 is a diagram illustrating the above-described three gradation characteristics. In the three gradation characteristic curves, it can be said that the curvature increases in the order of Γ ₁ , Γ ₂ , and Γ ₃ and the difference from the linear gradation increases. The method of selecting the three gradation characteristics is arbitrary, and is not limited to these. The shape of the curve can be adjusted according to the purpose. FIG. 2 shows an example in which a linear gradation, a square root gradation with an offset, and a cubic root gradation are compared. FIG. 3 to FIG. 5 show how the logarithmic gradation is used instead of the cubic root gradation, and the offset value of the square root gradation and the constant of the logarithmic gradation are changed according to the ISO sensitivity.

  In this description, an example in which color interpolation is performed on an image based on the Bayer array will be described. However, the present invention can be similarly applied to an image based on the delta array shown in Patent Document 3 or any other array.

<Preliminary explanation>
Before describing the embodiments in detail, the essence of the present invention will be described here. It is assumed that the RGB color components in the Bayer array are converted into the above three gradation characteristics Γ ₁ , Γ ₂ , and Γ ₃ , and the color components are color-interpolated by a known demosaic technique in each image processing space. Then, R ^(Γ1) (x, y), G ^(Γ1) (x, y), and B ^(Γ1) (x, y) are set as the color planes of the interpolation result in the Γ ₁ space. Similarly, R ^(Γ2) (x, y), G ^(Γ2) (x, y), and B ^(Γ2) (x, y) are set as the color planes of the interpolation result in the Γ ₂ space. Further, the color planes of the interpolation result in the Γ ₃ space are R ^(Γ3) (x, y), G ^(Γ3) (x, y), and B ^(Γ3) (x, y). All of these color planes are converted into an image having gradation characteristics in the standard Γ ₂ space. The images thus obtained are represented by the following symbols.

RGB images predicted in Γ ₁ space and shifted to Γ ₂ space: R ^{(Γ1 → Γ2)} (x, y), G ^{(Γ1 → Γ2)} (x, y), B ^{(Γ1 → Γ2)} (x, y)
RGB image has shifted to the predicted gamma ₂ space gamma ₂ ^{space: R (Γ2 → Γ2) (} x, y), G (Γ2 → Γ2) (x, y), B (Γ2 → Γ2) (x, y)
RGB images predicted in Γ ₃ space and shifted to Γ ₂ space: R ^{(Γ3 → Γ2)} (x, y), G ^{(Γ3 → Γ2)} (x, y), B ^{(Γ3 → Γ2)} (x, y)
Actually, gradation conversion processing is required only for an image predicted in the Γ ₁ space and an image predicted in the Γ ₃ space.

  One way of thinking is that the results of interpolation (prediction) using three prediction spaces may be weighted and synthesized by distinguishing the gradation areas that are good and bad in each space. However, with this concept, the contrast resolution required for interpolation raises the question of what gradation space should be used when a bright part and a dark part exist adjacent to each other. In order to solve this problem, it is necessary to determine, on a pixel-by-pixel basis, whether a pixel to be interpolated for which an interpolation value does not exist is a bright region, a dark region, or a halftone region using information on surrounding pixels. Yes, we face the challenge of not being able to easily identify the criteria.

  In order to easily avoid such a problem, the present invention projects the spatial contrast predicted by the above three interpolation results in the three gradation spaces to the final result to the maximum extent. That is, an inter-tone Laplacian component using a difference between prediction spaces in the same pixel is derived using three interpolation images, and added to the interpolation result of the reference gamma space to perform contrast correction.

If expressed by an expression, it can be expressed as the following expression (5).
R ' ^{(Γ2 → Γ2)} (x, y) = R ^{(Γ2 → Γ2)} (x, y) + k * [2 * R ^{(Γ2 → Γ2)} (x, y)-R ^{(Γ1 → Γ2)} (x , y)-R ^{(Γ3 → Γ2)} (x, y)]
G ' ^{(Γ2 → Γ2)} (x, y) = G ^{(Γ2 → Γ2)} (x, y) + k * [2 * G ^{(Γ2 → Γ2)} (x, y)-G ^{(Γ1 → Γ2)} (x , y)-G ^{(Γ3 → Γ2)} (x, y)]… (5)
B ' ^{(Γ2 → Γ2)} (x, y) = B ^{(Γ2 → Γ2)} (x, y) + k * [2 * B ^{(Γ2 → Γ2)} (x, y)-B ^{(Γ1 → Γ2)} (x , y)-B ^{(Γ3 → Γ2)} (x, y)]
Here, the constant k usually takes a value of about 1.

  In this way, while preserving the performance of the conventional gamma space for interpolation with the square root gradation characteristics, the maximum width of the contrast of the original image that could not be predicted with the square root gradation characteristics is derived and output in other prediction spaces. It can be reflected in the image. As a result, it is possible to obtain a high-definition interpolation image quality in which the overall cloudiness is lost as an image, and the transparency, stereoscopic effect, and resolution are increased. These provide the ability to describe a fine contrast as a large amplitude component in an important part that could not be obtained by conventional interpolation. Also, in the prediction space other than the square root gradation, for example, in the linear gradation space, the contrast resolution in the dark part cannot be obtained at all, and the image melts, or severe overshoot or false color around the bright spot in the high luminance part It is possible to obtain a pure improvement effect without incurring the dangers of failure such as generating a noise, or excessively amplifying the noise in the dark portion at the cubic root or logarithmic gradation.

When the inter-tone Laplacian component of the above equation (5) is decomposed, for example, the R component can be divided into two parts as shown in the following equation (6).
[R ^{(Γ2 → Γ2)} (x, y)-R ^{(Γ1 → Γ2)} (x, y)] + [R ^{(Γ2 → Γ2)} (x, y)-R ^{(Γ3 → Γ2)} (x, y) ] (6)
This means that even if only one of the two parts is used, the utility of maximizing the interpolation contrast can be obtained as long as the image in which the component exists is handled. In other words, it is possible to create a Laplacian component of the correction term simply by preparing a prediction space in which only one interpolation gamma is shifted from the reference gradation and examining the interpolation contrast to see the difference from the reference interpolation. means.

<First embodiment>
Hereinafter, the first embodiment of the present invention will be described in detail. In the first embodiment, during the Bayer interpolation, inter-tone Laplacians are extracted from all color planes of the RGB components, and the interpolation contrast is corrected for the three components of each color plane.

  The present embodiment provides an image processing apparatus by causing the computer apparatus 100 shown in FIG. 6 to execute an image processing program for performing Bayer interpolation processing. When the program is loaded into the personal computer 100, the program is loaded on the data storage device of the personal computer 100 and then executed to use the program as an image processing apparatus.

  The program may be loaded by setting a recording medium 104 such as a CD-ROM storing the program in the personal computer 100, or by loading the program into the personal computer 100 via a communication line 101 such as a network. Good. When going through the network 101, the program is stored in the hard disk device 103 of the server computer 102 connected to the network 101. In this manner, the program is supplied as various types of computer program products such as provision via the recording medium 104 or the communication line 101.

  Here, as the best example of describing the well-known Bayer interpolation technique other than the interpolation gamma space, the interpolation shown in US Patent Application Publication No. 2010/021853 of the same inventor as the present inventor is described. There is an algorithm. This Bayer interpolation technique includes a technique for increasing the direction determination resolution of the same inventor as the present applicant (US Pat. No. 6,836,572), a technique for preventing oblique line jaggies when calculating an interpolation value (US Pat. No. 7,236,628), color Adaptive false color countermeasure technology based on judgment method and technology to increase direction resolution (US Pat. No. 7,565,007), adaptive false color countermeasure technology based on color gradient judgment method (US Pat. No. 7,391,903) and direction resolution It is equipped with the best high-performance demosaic technology that is comprehensively applied to technology. Therefore, in this embodiment as well, it is necessary to use these results in order to best achieve the purpose of the function of the additional portion. Therefore, the entire text is re-quoted as “interpolation algorithm A”.

  FIG. 7 is a flowchart for explaining the flow of image processing executed by the CPU of the computer apparatus 100. In step S1 of FIG. 7, the CPU (hereinafter simply referred to as CPU) of the computer apparatus 100 inputs the Bayer image and proceeds to step S2. In this description, the linear gray scale Bayer array plane is represented by M (x, y). M (multiplexed) represents that R, G, B components are spatially multiplexed. The input Bayer image data is stored in the work memory of the computer device 100 as an interpolation processing target. If each color component constituting the Bayer image is expressed in a gamma-corrected gradation space, the subsequent processing is performed after returning to the linear gradation data before the gamma correction.

In step S2, the CPU converts the data of the linear gradation Bayer image into the above-described three interpolation gamma spaces (Γ ₁ , Γ ₂ , Γ ₃ ). As a result, M ^(Γ1) (x, y), M ^(Γ2) (x, y), and M ^(Γ3) (x, y) are generated. In the present embodiment, an image converted into a Γ ₂ space having an intermediate gradation among the three gamma spaces is used as a reference image. This is because color interpolation processing is performed using the uniform noise space as an image processing space. Therefore, since the noise width becomes uniform at all brightness levels, the performance of direction determination is maximized, and noise amplification can be minimized in the interpolation value calculation using other color components.

In step S3, the CPU performs Bayer interpolation (one way) on the reference image. That, Bayer array surface ^{M (Γ2) (x, y} ) of the interpolation gamma space is a reference tone (gamma ₂ space) with respect to, the "interpolation algorithm A" using the interpolated image R ^(.GAMMA.2) ( x, y), G ^(Γ2) (x, y), and B ^(Γ2) (x, y) are generated. The contents of “interpolation algorithm A” will be described below. However, (x, y) is described using the symbol [i, j]. The G component on the M plane is referred to using G, and the R and B components on the M plane are referred to using Z.

  FIG. 8 is a flowchart for explaining the details of the Bayer interpolation process performed with the “interpolation algorithm A” on the reference image. In the “interpolation algorithm A”, a CrCb surface is generated in steps S31 to S41, and a G surface is generated in steps S42 to S45. In step S31 in FIG. 8, the CPU performs a process of “vertical / horizontal direction determination 1”. The processing of “vertical / horizontal direction determination 1” includes (S31-1) similarity calculation and (S31-2) similarity determination. Specifically, the following calculation is performed for the R and B pixel positions.

(S31-1) Calculation of similarity The calculation of similarity is performed by (a) calculating the similarity between different colors and (b) adding the surroundings of the similarity.
(a) Calculation of similarity between different colors The CPU calculates a similarity component between GR (GB) by the following equations (7) and (8).
Cv0 [i, j] = (| G [i, j-1]-Z [i, j] | + | G [i, j + 1]-Z [i, j] |) / 2 (7)
Ch0 [i, j] = (| G [i-1, j]-Z [i, j] | + | G [i + 1, j]-Z [i, j] |) / 2 (8)

The CPU further performs (b) similarity addition according to the following equations (9) and (10). This process is for increasing the accuracy of similarity by considering continuity with surrounding pixels, and may be omitted when priority is given to simplicity.
Cv [i, j] = (4 * Cv0 [i, j]
+ 2 * (Cv0 [i-1, j-1] + Cv0 [i + 1, j-1] + Cv0 [i-1, j + 1] + Cv0 [i + 1, j + 1])
+ Cv0 [i, j-2] + Cv0 [i, j + 2] + Cv0 [i-2, j] + Cv0 [i + 2, j]) / 16 (9)
Ch [i, j] = (4 * Ch0 [i, j]
+ 2 * (Ch0 [i-1, j-1] + Ch0 [i + 1, j-1] + Ch0 [i-1, j + 1] + Ch0 [i + 1, j + 1])
+ Ch0 [i, j-2] + Ch0 [i, j + 2] + Ch0 [i-2, j] + Ch0 [i + 2, j]) / 16… (10)

(S31-2) Similarity determination The CPU determines similarity according to the following equation (11) based on the calculated similarity.
if | Cv [i, j]-Ch [i, j] | = <Th0 HVd [i, j] = 0 Unknown vertical / horizontal similarity
else if Cv [i, j] <Ch [i, j] HVd [i, j] = 1 Vertical similarity
else HVd [i, j] = -1 horizontal similarity (11)
However, the threshold value Th0 takes a value of around 10 for 256 gradations, and is set higher when there is a lot of image noise.

In step S32, the CPU performs “color difference generation”. The “color difference generation” process includes (S32-1) R-position Cr plane generation and (S32-2) Cr plane interpolation.
(S32-1) R position Cr plane generation The CPU generates an R position Cr plane by the following equation (12).
if [i, j] is a R site in a Bayer plane {
if HVd [i, j] = 1 Cr [i, j] = R [i, j]
-{(G [i, j-1] + G [i, j + 1]) / 2
+ (2 * R [i, j]-R [i, j-2]-R [i, j + 2]) / 4}
else if HVd [i, j] = -1 Cr [i, j] = R [i, j]
-{(G [i-1, j] + G [i + 1, j]) / 2
+ (2 * R [i, j]-R [i-2, j]-R [i + 2, j]) / 4}
else Cr [i, j] = R [i, j]
-{(G [i, j-1] + G [i, j + 1] + G [i-1, j] + G [i + 1, j]) / 4
+ (4 * R [i, j]-R [i, j-2]-R [i, j + 2]-R [i-2, j]-R [i + 2, j]) / 8}
} (12)

Alternatively, the R position Cr plane is calculated by the following equation (13) instead of the above equation (12).
if [i, j] is a R site in a Bayer plane {
if HVd [i, j] = 1 Cr [i, j] = R [i, j]
-{(G [i, j-1] + G [i, j + 1]) / 2
+ (2 * G [i-1, j]-G [i-1, j-2]-G [i-1, j + 2]
+ 2 * G [i + 1, j]-G [i + 1, j-2]-G [i + 1, j + 2]) / 8}
else if HVd [i, j] = -1 Cr [i, j] = R [i, j]
-{(G [i-1, j] + G [i + 1, j]) / 2
+ (2 * G [i, j-1]-G [i-2, j-1]-G [i + 2, j-1]
+ 2 * G [i, j + 1]-G [i-2, j + 1]-G [i + 2, j + 1]) / 8}
else Cr [i, j] = R [i, j]
-{(G [i, j-1] + G [i, j + 1] + G [i-1, j] + G [i + 1, j]) / 4
+ (2 * G [i-1, j]-G [i-1, j-2]-G [i-1, j + 2]
+ 2 * G [i + 1, j]-G [i + 1, j-2]-G [i + 1, j + 2]
+ 2 * G [i, j-1] -G [i-2, j-1] -G [i + 2, j-1]
+ 2 * G [i, j + 1] -G [i-2, j + 1] -G [i + 2, j + 1]) / 16}
} (13)
Here, the reason why the direction index HVd [i, j], in which the similarity is determined only by the similarity between different colors, is used because the generation of the false color near the Nyquist frequency can be significantly suppressed.

(S32-2) Cr surface interpolation The CPU interpolates the Cr surface according to the following equations (14) to (16).
B sites
Cr [i, j] = (Cr [i-1, j-1] + Cr [i-1, j + 1] + Cr [i + 1, j-1] + Cr [i + 1, j + 1 ]) / 4 ... (14)
G sites (same lines as R rows)
Cr [i, j] = (Cr [i-1, j] + Cr [i + 1, j]) / 2 (15)
G sites (same lines as B rows)
Cr [i, j] = (Cr [i, j-1] + Cr [i, j + 1]) / 2 (16)
The CPU performs the same for the Cb surface. That is, R position Cb surface generation and Cb surface interpolation are performed.

  In step S33, the CPU performs “provisional color difference correction 1”. There are plenty of false colors on the color difference plane obtained by the color difference generation in step S32, which causes color moire and color spot noise at high ISO sensitivity. These should be adaptively removed in distinction from the color structure. However, provisional removal is performed in advance so that a false color boundary is not erroneously identified as a color boundary when a color gradient described later is distinguished as an index. Although two methods are illustrated below, it may be performed by Method 1 or Method 2. Note that this method is not necessarily limited to this method.

Method 1 (Low-pass processing)
A 9 × 9 separable filter is exemplified as the low-pass filtering process. Expressions (17) and (18) correspond to the horizontal direction and vertical direction of the pixel arrangement, respectively.
Horizontal lowpass filtering
tmp_Cr [i, j] = (70 * Cr [i, j] + 56 * (Cr [i-1, j] + Cr [i + 1, j])
+ 28 * (Cr [i-2, j] + Cr [i + 2, j]) + 8 * (Cr [i-3, j] + Cr [i + 3, j])
+ (Cr [i-4, j] + Cr [i + 4, j])} / 256 (17)
Vertical lowpass filtering
TCr1 [i, j] = (70 * tmp_Cr [i, j] + 56 * (tmp_Cr [i, j-1] + tmp_Cr [i, j + 1])
+ 28 * (tmp_Cr [i, j-2] + tmp_Cr [i, j + 2])
+ 8 * (tmp_Cr [i, j-3] + tmp_Cr [i, j + 3])
+ tmp_Cr [i, j-4] + tmp_Cr [i, j + 4]} / 256… (18)
Although the description has been given for the Cr plane, TCb1 [i, j] is calculated in the same manner for the Cb plane.

Method 2 (median processing)
In the case of median processing, calculation is performed by the following equation (19). Similarly, TCb1 [i, j] is calculated for the Cb plane.
TCr1 [i, j] = Median {Cr [i + m, j + n]} (19)
However, m = 0, ± 1, ± 2, ± 3, ± 4 n = m = 0, ± 1, ± 2, ± 3, ± 4

  In step S34, the CPU performs “color gradient analysis”. Specifically, the false color and the actual color are discriminated by examining the color gradient so as not to destroy the color structure portion. Even if the process of “provisional color difference correction 1” in step S33 is applied to the actual color, the contrast is likely to remain as compared with the false color, so that it is possible to distinguish the actual color with very high accuracy from its statistical properties. In order to protect the color structure as accurately as possible, a color index surface is created that increases the color contrast between the existing colors and decreases the color contrast between the false colors. In general, since false colors tend to appear between opposite colors, it is preferable to generate a color index between primary colors.

That is, the color index Cdiff [i, j] for evaluating the color of each pixel by the following equation (20) from the temporary color difference signals TCr1 [i, j] and TCb1 [i, j] obtained as described above. Convert to
Cdiff [i, j] = (| TCr1 [i, j] | + | TCb1 [i, j] | + | TCr1 [i, j]-TCb1 [i, j] |) / 3 (20)
The color index uses all the color difference information that can be combined between the three primary colors of RGB to improve the protection performance of the color structure. When the definition of the color difference is expanded, the following equation (21) is obtained.
Cdiff = (| RG | + | GB | + | BR |) / 3 (21)

Next, the CPU examines a color gradient representing the degree of color change within the plane in which the color contrast information is pressed into a single color index plane. At this time, the size of the differential filter for detecting the color gradient is made the same as the size of the provisional color difference correction filter so that the entire range where there is a possibility of destruction of the color structure portion is examined, thereby improving the non-destructive nature of the color structure. The color gradient is expressed by the following equation (22).

  In addition, in order to speed up the calculation, it may be reduced to the inter-pixel differentiation that is thinned out a little more without examining the differentiation from all the surrounding pixels. In addition, although a 9 × 9 size filter is used together with “provisional color difference correction 1” in step S33, a 5 × 5 size filter may be used for both. In general, the 9 × 9 size provides a more powerful false color and real color separation performance.

In step S35, the CPU performs “adaptive color difference correction 1”. Specifically, whether to perform color difference correction processing is switched according to the magnitude of the color gradient. The original Cr [i, j] is used as it is for the color boundary with a large color gradient. First, a color gradient determination map idx_Cgrad [i, j] is created by the following equation (23).
if Cgrad [i, j] ≦ ThG {idx_Cgrad [i, j] = 1}
else {idx_Cgrad [i, j] = 0} (23)
Then, according to the following equation (24), the correction process is not performed on the area where the color gradient exists in the color gradient determination map, and the correction process is performed on the area where the color gradient does not exist.
if idx_grad [i, j] = 0 {Cr [i, j] = TCr1 [i, j], Cb [i, j] = TCb1 [i, j]} (24)
The threshold ThG is set to a very small value, and the color spot noise and color moire generation region corresponds to the color difference correction target region regardless of the chromatic color portion and the achromatic color portion, and the color structure portion is well excluded. The threshold ThG takes a value of about 5 or less for 256 gradations.

Thereafter, a uniform low pass process of about 3 × 3 size is applied to the entire surface. The low-pass process is expressed by the following equation (25), for example.
Cr [i, j] = {4 * Cr [i, j]
+ 2 * (Cr [i-1, j] + Cr [i + 1, j] + Cr [i, j-1] + Cr [i, j + 1])
+ 1 * (Cr [i-1, j-1] + Cr [i + 1, j-1] + Cr [i-1, j + 1] + Cr [i + 1, j + 1])} / 16 (25)
Although the above formula (25) has been exemplified for the Cr plane, Cb [i, j] is similarly calculated for the Cb plane.

In step S <b> 36, the CPU performs a “temporary color difference correction 2” process. Specifically, in order to accurately perform color evaluation, the above-described color moire generated on the color difference surface is further removed. Although a color difference median filter is also conceivable, the object can be achieved at high speed only by using a low-pass filter exemplified in the following equation (26).
TCr2 [i, j] = {4 * Cr [i, j]
+ 2 * (Cr [i-2, j] + Cr [i + 2, j] + Cr [i, j-2] + Cr [i, j + 2])
+ 1 * (Cr [i-2, j-2] + Cr [i + 2, j-2] + Cr [i-2, j + 2] + Cr [i + 2, j + 2])} / 16 ... (26)
Although the above equation (26) has been described for the Cr surface, TCb2 [i, j] is calculated in the same manner for the Cb surface.

In step S <b> 37, the CPU performs “derivation of color index”. The color difference component calculated as described above is a scalar quantity that can almost completely eliminate structural factors of the image and can measure the actual hue of the image in units of pixels. As the color index, the color difference between RGs indicated by the Cr component and the color difference between BGs indicated by the Cb component have already been obtained. However, since this is a scalar amount in units of pixels, the color difference between RBs can be further evaluated. Color evaluation close to visual judgment can be made. The following formula (27) shows the derivation formula of the color index Cdiff.
Cdiff [i, j] = (| TCr2 [i, j] | + | TCb2 [i, j] | + | TCr2 [i, j]-TCb2 [i, j] |) / 3 (27)

  Therefore, it is possible to obtain a very accurate color index by separating the disturbing factors associated with the actual color and structural factors that were problematic in the prior art color level and evaluating the color difference between all color components. be able to. This color index can be said to represent the degree of color.

  Here, in order to further improve the accuracy of the color index, the same process as that performed by the peripheral addition of the similarity may be performed for the color index.

In step S38, the CPU performs “color determination” using the following equation (28). In “color determination”, the continuous color index is subjected to threshold determination and converted to a discrete color index BW [i, j] as a color determination result.
if Cdiff [i, j] = <Thbc BW [i, j] = 'b'
else BW [i, j] = 'c' coloring part (28)
However, the threshold value Thbc takes a value of about 15 at 256 gradations.

In step S39, the CPU performs “adaptive color difference correction 2”. Specifically, as illustrated in the following equation (29), since the Nyquist false color generated before the correction is also reduced in the color difference plane subjected to the temporary correction process, the “color determination” result in S38. Based on the above, the false color of the achromatic color part is suppressed as much as possible, and the color structure of the chromatic color part is left.
if BW [i, j] ≠ 'c' {Cr [i, j] = TCr2 [i, j], Cb [i, j] = TCb2 [i, j]} (29)

In step S40, the CPU performs “derivation of color index”. The color index Cdiff is derived once again using the color difference surface that is cleanly and adaptively subjected to false color removal as described above. Thereby, the discrimination accuracy between the false color and the actual color for switching between the same color correlation and the different color correlation for generating the luminance plane is increased to the limit. The following formula (30) shows the derivation formula for the color index Cdiff.
Cdiff [i, j] = (| Cr [i, j] | + | Cb [i, j] | + | Cr [i, j]-Cb [i, j] |) / 3 (30)
Here, in order to further improve the accuracy of the color index, the same process as that performed by the peripheral addition of the similarity may be performed for the color index.

In step S41, the CPU performs “color determination” using the following equation (31). In “color determination”, the continuous color index is subjected to threshold determination and converted to a discrete color index BW [i, j] as a color determination result.
if Cdiff [i, j] = <Thab BW [i, j] = 'a' Achromatic part (31)
However, the threshold value Thab takes a value of about 5 at 256 gradations.

  In step S42, the CPU performs a process of “vertical / horizontal direction determination 2”. The processing of “Vertical / Horizontal Direction Determination 2” includes (S42-1) similarity calculation and (S42-2) similarity determination. Specifically, the following calculation is performed for the R and B pixel positions.

(S42-1) Calculation of similarity The similarity is calculated by (a) calculating the similarity between the same colors and (b) adding the peripherals of the similarity.
(a) Calculation of similarity between same colors The CPU calculates a similarity component between GG and a similarity component between BB (RR) by the following equations (32) to (37).
Cv0 [i, j] = (Cv1 + Cv2) / 2 (32)
Ch0 [i, j] = (Ch1 + Ch2) / 2 (33)
However, the similarity component between GG is
Cv1 = | G [i, j-1]-G [i, j + 1] | (34)
Ch1 = | G [i-1, j]-G [i + 1, j] |… (35)
And the similarity component between BB (RR) is
Cv2 = (| Z [i-1, j-1]-Z [i-1, j + 1] | + | Z [i + 1, j-1]-Z [i + 1, j + 1] | ) / 2 (36)
Ch2 = (| Z [i-1, j-1]-Z [i + 1, j-1] | + | Z [i-1, j + 1]-Z [i + 1, j + 1] | ) / 2 (37)
It is.

(b) Peripheral Addition of Similarity The CPU further performs (b) peripheral addition of similarity according to the following equations (38) and (39). This process is intended to increase the accuracy of similarity by considering continuity with surrounding pixels, and may be omitted when priority is given to simplicity, as in the case of similarity between different colors.
Cv [i, j] = (4 * Cv0 [i, j]
+ 2 * (Cv0 [i-1, j-1] + Cv0 [i + 1, j-1] + Cv0 [i-1, j + 1] + Cv0 [i + 1, j + 1])
+ Cv0 [i, j-2] + Cv0 [i, j + 2] + Cv0 [i-2, j] + Cv0 [i + 2, j]) / 16… (38)
Ch [i, j] = (4 * Ch0 [i, j]
+ 2 * (Ch0 [i-1, j-1] + Ch0 [i + 1, j-1] + Ch0 [i-1, j + 1] + Ch0 [i + 1, j + 1])
+ Ch0 [i, j-2] + Ch0 [i, j + 2] + Ch0 [i-2, j] + Ch0 [i + 2, j]) / 16… (39)

(S42-2) Similarity determination The CPU determines similarity according to the following equation (40) based on the calculated similarity.
if | Cv [i, j]-Ch [i, j] | = <Th1 HVs [i, j] = 0 Unknown vertical / horizontal similarity
else if Cv [i, j] <Ch [i, j] HVs [i, j] = 1 Vertical similarity
else HVs [i, j] = -1 horizontal similarity (40)
However, the threshold value Th1 is approximately the same as Th0.

In step S43, the CPU performs “direction index selection”. Specifically, based on the color determination result, the direction determination result based on the similarity between different colors and the direction determination result based on the similarity between the same colors are properly used according to the following equation (41).
if BW [i, j] = 'a' HV [i, j] = HVd [i, j]
else HV [i, j] = HVs [i, j] (41)

  In step S44, the CPU performs “diagonal direction determination”. The “diagonal direction determination” process includes (S44-1) calculation of similarity and (S44-2) similarity determination. Specifically, the following calculation is performed for the R and B pixel positions.

(S44-1) Calculation of similarity The calculation of similarity is performed by (a) calculating different colors and similarities between colors, and (b) performing peripheral addition of similarities.
(a) Calculation of similarity The CPU calculates a similarity component according to the following equations (42) to (49).
C45_0 [i, j] = (a1 * C45_1 + a2 * C45_2 + a3 * C45_3) / (a1 + a2 + a3)… (42)
C135_0 [i, j] = (a1 * C135_1 + a2 * C135_2 + a3 * C135_3) / (a1 + a2 + a3)… (43)
However, the BR (RB) similarity component is
C45_1 = (| Z [i + 1, j-1]-Z [i, j] | + | Z [i-1, j + 1]-Z [i, j] |) / 2 (44)
C135_1 = (| Z [i-1, j-1]-Z [i, j] | + | Z [i + 1, j + 1]-Z [i, j] |) / 2 (45)
And the GG similarity component is
C45_2 = (| G [i, j-1]-G [i-1, j] | + | G [i + 1, j]-G [i, j + 1] |) / 2 (46)
C135_2 = (| G [i, j-1]-G [i + 1, j] | + | G [i-1, j]-G [i, j + 1] |) / 2 (47)
The similarity component between BB (RR) is
C45_3 = | Z [i + 1, j-1]-Z [i-1, j + 1] |… (48)
C135_3 = (| Z [i-1, j-1]-Z [i + 1, j + 1] |… (49)
It is. The values of the constants a1, a2, and a3 are preferably set to a1 = a2 = a3 = 1, a1 = a2 = 2, a3 = 1, and the like.

(b) Peripheral addition of similarities The CPU further performs (b) peripheral addition of similarities according to the following equations (50) and (51). This process is intended to increase the accuracy of similarity by considering continuity with surrounding pixels, and may be omitted when priority is given to simplicity, as in the case of similarity in the vertical and horizontal directions.
C45 [i, j] = (4 * C45_0 [i, j]
+ 2 * (C45_0 [i-1, j-1] + C45_0 [i + 1, j-1]
+ C45_0 [i-1, j + 1] + C45_0 [i + 1, j + 1])
+ C45_0 [i, j-2] + C45_0 [i, j + 2] + C45_0 [i-2, j] + C45_0 [i + 2, j]) / 16… (50)
C135 [i, j] = (4 * C135_0 [i, j]
+ 2 * (C135_0 [i-1, j-1] + C135_0 [i + 1, j-1]
+ C135_0 [i-1, j + 1] + C135_0 [i + 1, j + 1])
+ C135_0 [i, j-2] + C135_0 [i, j + 2] + C135_0 [i-2, j] + C135_0 [i + 2, j]) / 16… (51)

(S44-2) Similarity determination The CPU determines the similarity according to the following equation (52) based on the calculated similarity.
if | C45 [i, j]-C135 [i, j] | = <Th2 DN [i, j] = 0 Unknown diagonal similarity
else if C45 [i, j] <C135 [i, j] DN [i, j] = 1 45 ° similar
else DN [i, j] = -1 diagonally 135 ° similar (52)
However, the threshold value Th2 has the same value as Th0 and Th1.

In step S <b> 45, the CPU performs “G plane generation” according to the following equation (53).
if [i, j] is not a G site in a Bayer plane {
if HV [i, j] = 0 {
if DN [i, j] = 0 G [i, j] = (Gv + Gh) / 2
else if DN [i, j] = 1 G [i, j] = (Gv45 + Gh45) / 2
else G [i, j] = (Gv135 + Gh135) / 2
}
else if HV [i, j] = 1 {
if DN [i, j] = 0 G [i, j] = Gv
else if DN [i, j] = 1 G [i, j] = Gv45
else G [i, j] = Gv135
}
else {
if DN [i, j] = 0 G [i, j] = Gh
else if DN [i, j] = 1 G [i, j] = Gh45
else G [i, j] = Gh135
}
} ... (53)
here,
Gv = (G [i, j-1] + G [i, j + 1]) / 2
+ (2 * Z [i, j]-Z [i, j-2]-Z [i, j + 2]) / 8
+ (2 * G [i-1, j]-G [i-1, j-2]-G [i-1, j + 2]
+ 2 * G [i + 1, j]-G [i + 1, j-2]-G [i + 1, j + 2]) / 16 ... (54)
Gv45 = (G [i, j-1] + G [i, j + 1]) / 2
+ (2 * Z [i, j]-Z [i, j-2]-Z [i, j + 2]) / 8
+ (2 * Z [i-1, j + 1]-Z [i-1, j-1]-Z [i-1, j + 3]
+ 2 * Z [i + 1, j-1]-Z [i + 1, j-3]-Z [i + 1, j + 1]) / 16… (55)
Gv135 = (G [i, j-1] + G [i, j + 1]) / 2
+ (2 * Z [i, j]-Z [i, j-2]-Z [i, j + 2]) / 8
+ (2 * Z [i-1, j-1]-Z [i-1, j-3]-Z [i-1, j + 1]
+ 2 * Z [i + 1, j + 1]-Z [i + 1, j-1]-Z [i + 1, j + 3]) / 16… (56)
Gh = (G [i-1, j] + G [i + 1, j]) / 2
+ (2 * Z [i, j]-Z [i-2, j]-Z [i + 2, j]) / 8
+ (2 * G [i, j-1]-G [i-2, j-1]-G [i + 2, j-1]
+ 2 * G [i, j + 1]-G [i-2, j + 1]-G [i + 2, j + 1]) / 16… (57)
Gh45 = (G [i-1, j] + G [i + 1, j]) / 2
+ (2 * Z [i, j]-Z [i-2, j]-Z [i + 2, j]) / 8
+ (2 * Z [i + 1, j-1]-Z [i-1, j-1]-Z [i + 3, j-1]
+ 2 * Z [i-1, j + 1]-Z [i-3, j + 1]-Z [i + 1, j + 1]) / 16… (58)
Gh135 = (G [i-1, j] + G [i + 1, j]) / 2
+ (2 * Z [i, j]-Z [i-2, j]-Z [i + 2, j]) / 8
+ (2 * Z [i-1, j-1]-Z [i-3, j-1]-Z [i + 1, j-1]
+ 2 * Z [i + 1, j + 1]-Z [i-1, j + 1]-Z [i + 3, j + 1]) / 16… (59)
And

As described above, since Cr [i, j], Cb [i, j], and G [i, j] are obtained at each pixel position, a color image is completed. In step S46, when the CPU uses the RGB color system, the CPU performs color conversion according to the following equation (60).
R [i, j] = Cr [i, j] + G [i, j]
B [i, j] = Cb [i, j] + G [i, j] (60)

  In step S4 of FIG. 7, the CPU obtains the process of step S3 three times even for the interpolation image that is actively generated using the three interpolation gamma spaces. However, obtaining a direction determination index or a switching index for color difference surface smoothing in a space other than the uniform noise space is unreliable and dangerous. This is because if the noise is not uniform, the direction may be erroneously influenced by noise. This is because, when the color is extremely deviated from a cubic root gradation or a square root gradation which can be called a uniform color space, there is a possibility that a correct color suitable for vision cannot be evaluated.

  In order to deal with such a situation, the final direction determination indexes HV [i, j] and DN [i, j] generated in step S3, and the color gradient determination that is an adaptive false color removal switching index The map idx_Cgrad [i, j] and the color determination map BW [i, j] are held as reference data as they are, and a common index is used for all. In this way, not only the amount of calculation is reduced, but also the three interpolation images refer to the common direction determination index and the same false color removal index, so that the images in the same direction without any contradiction among the three are also included. Interpolation results obtained through image and false color suppression processes are obtained.

The interpolated images in the Γ ₁ , Γ ₂ , and Γ ₃ spaces thus obtained are represented by the following equation (61), respectively.
Γ ₁ space: R ^(Γ1) (x, y), G ^(Γ1) (x, y), B ^(Γ1) (x, y)
Γ ₂ space: R ^(Γ2) (x, y), G ^(Γ2) (x, y), B ^(Γ2) (x, y) (61)
Γ ₃ space: R ^(Γ3) (x, y), G ^(Γ3) (x, y), B ^(Γ3) (x, y)

In step S5, the CPU performs gradation matching of the interpolation image. In order to measure how much the interpolation values predicted in three different gamma spaces for interpolation (Γ ₁ , Γ ₂ , Γ ₃ ) actually differ, we moved to a common gamma space Above, we need to see the difference. Otherwise, the interpolation value is directly affected by the difference in gamma characteristics. Therefore, a gamma space for interpolation (Γ ₂ space), which is a reference gradation, is selected as a common gamma space.

Each interpolated image shifted to the reference interpolation gamma space (Γ ₂ space) is represented by the following equation (62).
Γ ₁ space → Γ ₂ space: R ^{(Γ1 → Γ2)} (x, y), G ^{(Γ1 → Γ2)} (x, y), B ^{(Γ1 → Γ2)} (x, y)
Γ ₂ space → Γ ₂ space: R ^{(Γ2 → Γ2)} (x, y), G ^{(Γ2 → Γ2)} (x, y), B ^{(Γ2 → Γ2)} (x, y) (62)
Γ ₃ space → Γ ₂ space: R ^{(Γ3 → Γ2)} (x, y), G ^{(Γ3 → Γ2)} (x, y), B ^{(Γ3 → Γ2)} (x, y)
However, the Γ ₂ space → Γ ₂ space need not be processed. That is, R ^{(Γ2 → Γ2)} (x, y) = R ^(Γ2) (x, y), G ^{(Γ2 → Γ2)} (x, y) = G ^(Γ2) (x, y), B ^{(Γ2 → Γ2)} (x, y) = B ^(Γ2) (x, y).

In step S6, the CPU performs Laplacian extraction between gradations. Interpolated images obtained in _{three different} gamma spaces for interpolation (Γ ₁ , Γ ₂ , Γ ₃ ) and transferred to the reference Γ ₂ space are used in the same pixel of each color plane. Differences in the contrast of the interpolation values that are actively predicted using the three interpolation value prediction spaces are extracted in the form of a second derivative with respect to the result of the reference image, that is, a Laplacian component.

The Laplacian component is expressed by the following equation (63).
△ R ^(Γ2) (x, y) = 2 * R ^{(Γ2 → Γ2)} (x, y)-R ^{(Γ1 → Γ2)} (x, y)-R ^{(Γ3 → Γ2)} (x, y)
△ G ^(Γ2) (x, y) = 2 * G ^{(Γ2 → Γ2)} (x, y)-G ^{(Γ1 → Γ2)} (x, y)-G ^{(Γ3 → Γ2)} (x, y)… ( 63)
△ B ^(Γ2) (x, y) = 2 * B ^{(Γ2 → Γ2)} (x, y)-B ^{(Γ1 → Γ2)} (x, y)-B ^{(Γ3 → Γ2)} (x, y)

The Laplacian component extracted in this way uses a common index for the direction determination result used for the interpolation values predicted in the _three interpolation gamma spaces (Γ ₁ , Γ ₂ , Γ ₃ ). A stable fine correction component is extracted without worrying about generating a strange pattern due to. The same can be said because the false color suppression effect is also in phase.

In step S7, the CPU corrects the reference interpolation image. Interpolation in interpolation gamma space obtained in step S3 (gamma ₂ space) image ^{R (Γ2) (x, y} ), G (Γ2) (x, y), B and ^{(Γ2) (x, y)} , Laplacian Processing for correcting the contrast with the components is performed to maximize the gradation contrast that can be extracted by interpolation (the following equation (64)).
R ' ^(Γ2) (x, y) = R ^{(Γ2 → Γ2)} (x, y) + k * △ R ^(Γ2) (x, y)
G ' ^(Γ2) (x, y) = G ^{(Γ2 → Γ2)} (x, y) + k * ^{ΔG (Γ2)} (x, y) (64)
B ' ^(Γ2) (x, y) = B ^{(Γ2 → Γ2)} (x, y) + k * △ B ^(Γ2) (x, y)
However, the value of the constant k usually takes a value of 1 to 0.5.

In step S8, the CPU performs output color space conversion of the interpolated image. Specifically, the tone characteristics of the interpolated image (RGB image interpolated in the Γ ₂ space) after contrast correction according to the above equation (64) is inversely converted back to linear tone, and then an arbitrary color conversion process is performed. And output gamma conversion processing and output as an output color space image. A typical output color space is, for example, an sRGB color space.

According to the first embodiment described above, the following operational effects can be obtained.
(1) The image processing apparatus converts linear tone image signals into three tone characteristics Γ ₁ , Γ having different curvatures from image data consisting of RGB color components and having missing color components. ₂ and Γ ₃ are converted into image signals in the image processing space (S2), and the missing color components of each of the tone-converted image signals are interpolated to generate three Bayer interpolation images (S4). The CPU and the three kinds of interpolated images are converted into an image in the image processing space having the reference gradation characteristic Γ ₂ to match the gradation characteristics (S5). The CPU and the three gradation characteristics Γ ₁ , Γ ₂ , Γ ₃ Using the three types of Bayer interpolation images interpolated in the above and tone-matched to the reference tone characteristics Γ ₂ , the pixel values and gradations of the interpolated image of the gradation characteristics Γ ₂ are calculated between the same pixels of the three images. difference values, and the pixel value and the gradation of tone characteristic gamma ₂ interpolation image between the pixel value of the characteristic gamma ₁ interpolation image A difference value between the pixel value of sexual gamma ₃ interpolation image extracting respectively (S6) CPU, relative to the tone characteristics gamma ₂ interpolation image, adds the difference value to the same pixel (S7) CPU And a CPU that converts the image signal after the addition into an image signal in the output color space (S8). This enables high-definition interpolation that can provide a higher-contrast interpolated image while maximizing the performance of the conventional image prediction space that can perform excellent contrast resolution on average in all gradation regions. be able to. That is, three types of interpolation results are generated using the prediction space by the _three interpolation gammas Γ ₁ , Γ ₂ , and Γ ₃ , and the same between the three parties based on the interpolation result by the interpolation gamma Γ ₂ located in the middle. A method of extracting a differential component twice between gradations in a pixel and adding it to a reference image was introduced. Accordingly, the contrast predicted by the other gamma spaces Γ ₁ and Γ ₃ that cannot be extracted in the gamma space Γ ₂ of the reference image is reflected to the maximum while the interpolation quality of the conventional reference image is maximized. It became possible. This maximizes the contrast inherent in the image that can be extracted during the interpolation process, thus eliminating the cloudiness and fogging of the entire image and increasing the overall transparency, resolution, and stereoscopic effect. A fine interpolated image can be obtained.

(2) In the image processing apparatus of (1), gradation conversion is performed. (S2) The CPU uses the linear gradation characteristic Γ ₁ , the square root gradation characteristic Γ ₂ , and the cubic root gradation characteristic Γ ₃ , respectively. Therefore, the contrast predicted in the square root tone gamma space Γ ₁ and the cubic root tone gamma space Γ ₃ which could not be extracted in the linear tone gamma space Γ ₂ can be reflected to the maximum.

(3) In the image processing apparatus according to (1) or (2), the color components are interpolated to generate three Bayer interpolation images. (S4) The CPU uses the missing color component interpolation values as missing colors. Since the calculation is performed using not only the color information of the components but also the color information of other color components, the color components can be appropriately interpolated.

(4) The image processing apparatus converts the linear gradation image signal into three gradation characteristics Γ ₁ and Γ having different curvatures with respect to image data including RGB color components and having missing color components. ₂ and Γ ₃ are converted into image signals in the image processing space (S2), and the missing color components of each of the tone-converted image signals are interpolated to generate three Bayer interpolated images (S4). The CPU and the three kinds of interpolated images are converted into an image in the image processing space having the reference gradation characteristic Γ ₂ to match the gradation characteristics (S5). The CPU and the three gradation characteristics Γ ₁ , Γ ₂ , Γ ₃ Using the three types of Bayer interpolation images interpolated in the above and matched in tone to the reference tone characteristic Γ ₂ , the tone for the pixel value of the interpolated image of the tone characteristic Γ ₂ is obtained between the same pixels of the three images. Laplacian between the pixel value of the characteristic gamma ₁ of the interpolated image and the pixel value of the interpolated image tone characteristic gamma ₃ Extracting a partial and (S6) CPU, relative to the tone characteristics gamma ₂ of interpolated images, and so comprises adding Laplacian components in the same pixel and (S7) CPU, a. This enables high-definition interpolation that can provide a higher-contrast interpolated image while maximizing the performance of the conventional image prediction space that can perform excellent contrast resolution on average in all gradation regions. be able to.

(5) In the image processing apparatus of (4), gradation conversion is performed (S2). The CPU uses the linear gradation characteristic Γ ₁ , the square root gradation characteristic Γ ₂ , and the cubic root gradation characteristic Γ ₃ , respectively. Therefore, the contrast predicted in the square root tone gamma space Γ ₁ and the cubic root tone gamma space Γ ₃ which could not be extracted in the linear tone gamma space Γ ₂ can be reflected to the maximum.

(6) In the image processing apparatus according to (4) or (5), the color component is interpolated to generate three Bayer interpolation images. (S4) The CPU uses the missing color component interpolation value as the missing color. Since the calculation is performed using not only the color information of the components but also the color information of other color components, the color components can be appropriately interpolated.

(Modification 1)
In the above description, three Bayer interpolation images interpolated with the three gradation characteristics Γ ₁ , Γ ₂ , and Γ ₃ and gradation-matched with the reference gradation characteristic Γ ₂ are used to identify the same three images. between pixels, it extracts the Laplacian component between the pixel values of the gradation characteristic gamma ₁ of the interpolated image pixel values and the tone characteristics gamma ₃ of the interpolation image with respect to the pixel value of the gradation characteristic gamma ₂ interpolation image (S6) The example in which the Laplacian component is added to the same pixel with respect to the interpolation image having the gradation characteristic Γ ₂ has been described (S8). Instead, the difference value between the pixel value of the interpolation image having the gradation characteristic Γ ₂ and the pixel value of the interpolation image having the gradation characteristic Γ ₁ , or the pixel value and the gradation of the interpolation image having the gradation characteristic Γ ₂ A difference value between the pixel value of the interpolation image having the characteristic Γ ₃ may be extracted, and the difference value may be added to the same pixel with respect to the interpolation image having the gradation characteristic Γ ₂ . Even in this case, the contrast predicted in the gamma space Γ ₁ or the gamma space Γ ₃ that cannot be extracted in the reference tone gamma space Γ ₂ can be reflected to the maximum extent.

<Second Embodiment>
In the first embodiment, an example has been described in which Laplacian between gradations is extracted from all color planes of RGB components during Bayer interpolation, and interpolation contrast is corrected for the three components of each color plane. In the second embodiment, an example in which this is simplified a little more and interpolation contrast correction is performed only on the luminance surface will be described. In the following description, the description of the part that performs the same processing as in the first embodiment will be omitted, and the description will focus on different processing.

<Generation of Laplacian component of luminance surface>
The CPU performs inter-tone Laplacian components ΔR ^{(Γ 2)} (x, y), Δ G ^{(Γ 2)} (x, y) based on the equation (63) extracted in step S 6 of FIG. ), △ from ^{B (Γ2) (x, y} ), generated by the gradation between Laplacian component of the luminance component ^{△ Y (Γ2) (x,} y) the following equation (65).
△ Y ^(Γ2) (x, y) = 0.3 △ R ^(Γ2) (x, y) + 0.6 △ G ^(Γ2) (x, y) + 0.1 △ B ^(Γ2) (x, y)… (65)

  In step S7, the CPU corrects the reference interpolation image as follows. The CPU temporarily converts the RGB interpolated image obtained in step S3 of FIG. 7 into a luminance plane and a color difference plane, corrects the luminance plane with an inter-tone Laplacian component, and then returns to the original RGB plane. . The flow of processing for correcting the reference interpolation image will be described with reference to the flowchart illustrated in FIG.

In step S81 in FIG. 9, the CPU converts the RGB interpolated image into a luminance / color difference plane by the following equations (66) to (68).
Y (x, y) = 0.3R ^{(Γ2 → Γ2)} (x, y) + 0.6G ^{(Γ2 → Γ2)} (x, y) + 0.1B ^{(Γ2 → Γ2)} (x, y)… (66)
Cr (x, y) = R ^{(Γ2 → Γ2)} (x, y)-Y (x, y) (67)
Cb (x, y) = B ^{(Γ2 → Γ2)} (x, y)-Y (x, y) (68)

In step S82, the CPU corrects the luminance surface by the following equation (69).
Y ′ (x, y) = Y (x, y) + ^{ΔY (Γ2)} (x, y) (69)

In step S83, the CPU performs inverse conversion to the original RGB plane by the following equation (70).
R ' ^(Γ2) (x, y) = Cr (x, y) + Y' (x, y)
B ′ ^(Γ2) (x, y) = Cb (x, y) + Y ′ (x, y) (70)
G ' ^(Γ2) (x, y) = [Y' (x, y)-0.3R ' ^(Γ2) (x, y)-0.1B' ^(Γ2) (x, y)] / 0.6

  According to the second embodiment described above, it is simpler than the first embodiment by correcting the interpolation contrast only for the luminance plane instead of correcting the interpolation contrast for the three components of each color plane. it can.

<Third embodiment>
In the first embodiment and the second embodiment, the method of actively moving the interpolation gamma space in the demosaic processing such as the Bayer array and extracting the maximum contrast amplitude obtained as the interpolation prediction value has been described. The same idea can be applied when image processing is performed on a general image that has been interpolated. For example, when edge enhancement is performed, edge detection is performed with the gradation characteristics of the input image, and processing is performed such that the extracted edge component is multiplied by a gain and added to the original image. Alternatively, it is converted to a uniform color space such as L * a * b *, edge detection is performed on the luminance plane L *, and edge components are added. Alternatively, as shown in U.S. Pat. No. 7,940,983 by the same inventor as the present applicant, it is converted into a uniform color / uniform noise space and the same processing is performed on the luminance plane.

  In these cases, since the tone characteristic of the color plane for edge detection is fixed to a curved line, for example, if a gradation space of a curve where the knee characteristic of the high-luminance part is strongly lying is used, This causes a problem that the edge contrast of the part is not sufficiently extracted. In order to deal with such a problem, as shown in the first embodiment and the second embodiment, the gradation space is actively changed, and the contrast component between the gradations is extracted by the edge component extracted by each. The idea of correcting the edge component extracted in the reference gradation space with it is established.

  In the third embodiment, edge enhancement processing for a monochrome image will be described as an example. In the case of a color image, whether the monochrome image processing is the same for each of the R, G, and B surfaces, or is the same processing as for the monochrome image only for the luminance surface after conversion to the luminance / color difference surface Any one of the above may be performed.

The reference gradation space Γ ₂ for performing edge enhancement has a square root type gradation characteristic that is a uniform noise space as in the first embodiment. The Γ ₁ and Γ ₃ spaces have the same gradation characteristics as in the first embodiment. These gradation characteristic setting methods need not be limited to these. For example, when the uniform color space L * is used as the reference gradation space, the Γ ₂ space has a cubic root gradation characteristic, so a square root type uniform noise space is used as the Γ ₁ space, and a logarithmic gradation is used as the Γ ₃ space. Can also be set.

<Generation of Laplacian component of luminance surface>
The flow of edge emphasis processing performed by the CPU of the third embodiment will be described with reference to the flowchart illustrated in FIG. In step S91 of FIG. 10, the CPU inputs a monochrome image M (x, y) with a standard gamma curve.

In step S92, the CPU performs three kinds of gradation conversion on the monochrome image. Specifically, after the gamma characteristic of the input image data is returned to the linear gradation, it is converted into the above-described three image processing gamma spaces (Γ ₁ , Γ ₂ , Γ ₃ ). As a result, M ^(Γ1) (x, y), M ^(Γ2) (x, y), and M ^(Γ3) (x, y) are generated.

In step S93, the CPU extracts three types of edge components. Specifically, on the image plane in each gradation space in the _three image processing gamma spaces (Γ ₁ , Γ ₂ , Γ ₃ ), the image smoothed by the following equations (71) to (73) and the original The edge component is extracted by taking the difference value from the image.
Γ ₁ space: E ^(Γ1) (x, y) = M ^(Γ1) (x, y)-<M ^(Γ1) (x, y)>… (71)
Γ ₂ space: E ^(Γ2) (x, y) = M ^(Γ2) (x, y)-<M ^(Γ2) (x, y)>… (72)
Γ ₃ space: E ^(Γ3) (x, y) = M ^(Γ3) (x, y)-<M ^(Γ3) (x, y)> (73)
The symbol <> represents a local average of a plurality of pixels.

In step S94, the CPU performs Laplacian extraction between the gradations of the edge component and correction of the edge component. Specifically, the Laplacian between the edge component in the Γ ₁ space and the edge component in the Γ ₃ space with respect to the edge component in the Γ ₂ space within the same pixel from the edge components extracted in the three image processing gamma spaces. The component is extracted, thereby correcting the contrast of the edge component in the Γ ₂ space (the following equation 74).
E ' ^(Γ2) (x, y) = E ^(Γ2) (x, y) + k * [2 * E ^(Γ2) (x, y) ^{-E (Γ1)} (x, y) ^{-E (Γ3)} (x, y)] ... (74)
Here, k is a constant multiple. By this correction, in the reference image processing space Γ ₂ , the edge contrast component that is difficult to extract the contrast is picked up as an edge component.

In step S95, the CPU performs edge enhancement processing of the reference image. That is, edge enhancement processing represented by the following equation (75) is performed using the edge component after contrast correction.
M ' ^(Γ2) (x, y) = M ^(Γ2) (x, y) + α * E' ^(Γ2) (x, y) (75)
Here, the constant α is a gain factor for edge enhancement.

In step S96, the CPU performs output color space conversion of the edge-enhanced image and ends the processing in FIG. Specifically, the tone characteristics of the image edge-enhanced in the reference image processing space Γ ₂ space are inversely transformed to return to linear tones, and then output gamma transformation processing is performed and output as an output space image.

  According to the third embodiment described above, the concept described in the first embodiment and the second embodiment (interpolated gamma space is actively moved in the demosaic processing such as Bayer array to obtain an interpolation predicted value. The maximum contrast amplitude that can be extracted) is also introduced during edge emphasis processing, so that the tone space is actively changed, and the contrast component between the tones is extracted with the edge components extracted by each. The edge component extracted in the gradation space is corrected by it. As a result, for example, even when a gradation space having a curve in which the knee characteristic of the high luminance part is strongly lying is used, edge extraction can be sufficiently performed in the high luminance part, and high-definition edge enhancement processing is performed. be able to.

(Modification 2)
In the above description, an example in which the computer apparatus 100 (FIG. 6) executes the image processing program has been described. However, the image processing apparatus may be mounted on the imaging apparatus. FIG. 11 is a block diagram illustrating the configuration of a camera equipped with an image processing apparatus. The camera 200 includes an operation member 201, an image sensor 202, a control device 203, a memory card slot 204, and a monitor 205. The operation member 201 includes various input members operated by the user, such as a power button, a release button, a zoom button, and the like.

  The image sensor 202 is configured by a CMOS image sensor or the like. The image sensor 202 captures a subject image formed by a lens (not shown). Then, the image data obtained by imaging is output to the control device 203.

  The control device 203 includes a memory and other peripheral circuits, and controls a photographing operation performed by the camera 200. Memory constituting the control device 203 includes SDRAM and flash memory. The SDRAM is a volatile memory, and is used as a work memory for the control device 203 to develop a program when the program is executed. The SDRAM is also used as a buffer memory for temporarily storing data. On the other hand, the flash memory is a non-volatile memory in which data of a program executed by the control device 203, various parameters read at the time of program execution, and the like are recorded.

  The memory card slot 204 is a slot for inserting a memory card as a storage medium. The control device 203 writes and records the image file generated by performing the photographing process on the memory card inserted in the memory card slot 204. Further, the control device 203 reads an image file recorded in the memory card inserted in the memory card slot 204.

  The monitor 205 is a liquid crystal monitor (rear monitor) mounted on the back surface of the camera 200. The monitor 205 displays a reproduction image based on the image file recorded on the memory card, a setting menu for setting the camera 200, and the like. When the user operates the operation member 201 to set the camera 200 to the shooting mode, the control device 203 outputs image data for display of images acquired in time series from the image sensor 202 to the monitor 205. As a result, a through image is displayed on the monitor 205.

On the image pickup surface of the image pickup element 202, R, G, and B color filters are arranged in a Bayer array. That is, the image data output from the image sensor 202 to the control device 203 is expressed in the RGB color system. The control device 203 performs the above-described processing on this image data.
For delta arrangements other than Bayer arrangement, the same idea can be applied by color interpolation of arbitrary arrangement such as honeycomb arrangement.

Before describing the fourth embodiment, the demosaic technique will be described again. What has been described in the first embodiment and the second embodiment is a case where the cubic root gradation characteristic (Γ ₃ ) is adopted as the gradation curve having the largest curvature. As shown in FIG. 2, this does not deviate significantly from the square root gradation characteristic (Γ ₂ ) of the reference gradation, and thus the above argument holds. However, as shown in FIGS. 3 to 5, when a gradation curve having an extremely large curvature such as a logarithmic gradation characteristic (Γ ₃ ) is adopted, the situation is slightly different, and the image quality effect is another unique new one. Produce a thing of nature. The inter-gradient Laplacian plays different roles depending on the gradation characteristics to be adopted. The following describes such an embodiment.

  In paragraph 0023, it was stated that the concept of weighted synthesis of the interpolation results of a plurality of prediction spaces by discriminating the gradation areas of the pros and cons of each space is difficult because the judgment criteria do not exist easily. . However, the following is a description of a new concept in which the judgment criteria are extremely rational and appropriate.

<Difference in image quality due to interpolation gamma space (prediction space)>
Here, the properties of the image interpolated by the three kinds of methods of the linear gradation space (Γ ₁ ), the square root gradation space (Γ ₂ ), and the logarithmic gradation space (Γ ₃ ) will be described again with specific examples. To do. Interpolation gamma space when a subject with a lot of colors and characters, such as a `` signboard chart '' in a transmission standard chart for HDTV (high-definition television), is shot at a resolution close to the Nyquist resolution limit. The difference in image quality due to

For example, in a subject image in which black characters are contained in a red background, an image interpolated in the linear gradation space (Γ ₁ ) has a resolution in which the black characters are melted in the red background and the characters are almost lost. When interpolated in the square root gradation space (Γ ₂ ), the black characters are somehow resolved, but there is an impression of blurring in red, resulting in an unclear color resolution. When interpolated in the other logarithmic gradation space (Γ ₃ ), black characters clearly appear and color resolution is maximized. Therefore, in the case of a combination such as a black character on a red background, an interpolating gamma space having a logarithmic gradation in which the contrast of a dark part is increased is an advantageous characteristic.

On the other hand, the opposite phenomenon occurs in a subject image in which a thin white frame is included in a red background. That is, in the case of an interpolating gamma space (Γ ₃ ) having a logarithmic gradation, the surrounding red color oozes out into a white frame line, resulting in a white line colored red. In the case of the interpolating gamma space (Γ ₂ ), which is a square root gradation, the red blur still remains in the white line, although it is considerably suppressed. On the other hand, in the case of an interpolating gamma space (Γ ₁ ) having a linear gradation, no red blur occurs in the white line, and the color resolving power in this color combination is the highest. Therefore, in the case of a combination such as a white line on a red background, an interpolating gamma space which is a linear gradation in which a wide gradation contrast is assigned to a bright gradation area is an advantageous characteristic.

<Method for synthesizing three gamma prediction results>
In such a situation, by correcting by the inter-tone Laplacian while performing weighted synthesis of the interpolation values predicted in the three interpolation gamma spaces by the following equations (76) to (78), Consider calculating an interpolation value. However, it is assumed that the result of interpolation in the three gamma spaces for interpolation is converted into a gamma space for interpolation having a square root gradation (Γ ₂ ) as a common gamma space. RGB is synthesized with a common weight.

  The first term of the equations (76) to (78) can be called interpolation or internal division interpolation because the internal division points of the three interpolation results can be predicted. Since the second term can predict the outer dividing point of three interpolation results, it can be said that it plays the role of extrapolation or outer dividing point interpolation. The first term is weighted average interpolation, and the second term is inter-tone Laplacian.

<Effect of internal weighting and Laplacian between gradations>
When G _sqrt (x, y) is corrected with a Laplacian between gradations (only external correction) as in the first embodiment, the internal weighting coefficient w is determined by the method described in detail below. The image quality is compared for the three cases where only one term is used (internal weighting only) and when the first term and second term are used together (internal weighting + outside weight correction).

  In the case of a black character subject image on a red background, an image that has been subjected to “external compensation only” is contrary to the effect of the first embodiment, the black character is covered with a red background and the resolution of the black character is rather a square root gradation. As a result, a phenomenon of lowering the interpolation result of only the interpolation gamma space is generated. However, the result of interpolation using only “internal weighting” with an internal weighting factor determined so as to basically maximize the resolution contrast maximizes the advantage of the logarithmic gradation interpolation gamma space. Incorporated to show strong contrast resolution performance. As a result, the “internal weighting + outer part correction” method using both the first term and the second term is more comprehensive than the conventional interpolation result of only the interpolation gamma space, which is a square root gradation, in terms of color resolution. High-quality interpolation results with increased color contrast can be obtained.

  Next, in the case of a subject image with a thin white line on a red background, the interpolation result with “internal weighting only” is close to the result of interpolation using the logarithmic gray scale interpolation gamma space, and the white line is strong against red. The result is that the color reproducibility is distorted with blurring and high saturation. The interpolation result of “external component correction only” in the other first embodiment provides an interpolation result in which the white line does not blur and the saturation of the entire red background portion is low. Therefore, the “internal weighting + outer part correction” method combining the first term and the second term is well harmonized and exhibits appropriate color reproducibility, while interpolating only the gamma space for interpolation, which is a square root gradation. Interpolation results with high color resolving power in which white lines are clearly resolved than the results are obtained.

  In other words, the method that uses both internal weighting and external weight correction, both of which are good and bad, are both “black text on red background” and “white line on red background”. While deriving, it is possible to cancel each other's harmful effects and generate a higher-performance interpolation result than the prediction result by one conventional interpolation gamma space.

  In the above description, an example of black characters and white lines on a red background has been described. However, the same improvement effect can be derived for other various color combinations.

<Weighted composite index>
Here, how to determine the internal weighting coefficient will be described. For the three types of interpolation results obtained using the three prediction spaces, a “gain (Figure of Merit)” is calculated, and synthesis is performed with emphasis on the interpolation result having a large gain. As a criterion for determining the gain, it is first required to increase the resolution (spatial resolution). The resolving power is roughly divided into two kinds of concepts of luminance resolving power and color resolving power.

  As an index regarding whether the luminance resolution of each interpolated image is high, it is checked whether the luminance contrast is high. The luminance contrast is estimated to give an appropriate determination index when Laplacian that is a local edge component is extracted. This is because if you have an image structure that replaces black and white in units of pixels close to the Nyquist frequency, the Laplacian of the second derivative extracted in the vicinity has a large value when it has the ability to decompose the image structure. Because it is thought that it will be. Therefore, a luminance plane is generated, a Laplacian of the luminance plane is extracted, and luminance contrast is evaluated.

  Similarly, an indicator regarding whether the color resolution of the interpolated image is high examines whether the color contrast is high. The color contrast is estimated to give an appropriate judgment index when a color difference plane or a saturation plane is created and a Laplacian in the plane is extracted.

<Trade-off relationship>
However, the interpolation image contains a noise component and a false color component. Maximizing the brightness contrast as described above may amplify noise. In addition, when the color contrast is maximized, there is a risk of adopting an interpolation result with a strong false color. That is, in general, in image processing for color interpolation, noise removal, edge enhancement, etc., luminance contrast and noise are in a conjugate relationship, so that increasing the luminance contrast amplifies the noise, and reducing the noise reduces the luminance contrast. There are significant tradeoffs. In other words, the uncertainty principle works.

  Similarly, there is a conjugate relationship between color contrast and false color (or color noise), and there is a trade-off that increases false color when trying to increase color contrast and decreases color contrast when trying to reduce false color. To do.

<Gain (Figure of Merit)>
Based on this fact, it is necessary to consider what kind of index to give the best “Figure of Merit”. The solution to this is to set an index to simultaneously maximize brightness contrast while suppressing noise. Similarly, setting an index to simultaneously maximize the color contrast while suppressing false colors. Furthermore, an index is set so that the maximum gain of the luminance plane and the maximum gain of the color difference plane are synchronized.

こうして求めた「Figure of Merit」を重み係数に比例するように設定する(式(８１))。

各画素毎に３つの補間画像に対する重み係数の間で、規格化を行うと加重係数となる。 Therefore, the `` Figure of Merit '' at each pixel is I (x, y), the Laplacian of the luminance plane Y is △ Y (x, y), the Laplacian of the color difference plane C is △ C (x, y), R, The local standard deviation of G and B surfaces is σ _R (x, y), σ _G (x, y), σ _B (x, y), the color difference between RG, BG, and RB is | R ( x, y) -G (x, y) |, | B (x, y) -G (x, y) |, | R (x, y) -B (x, y) | 79) and (80).
I = (luminance contrast), (noise suppression), (color contrast), (false color suppression) ... (79)

The “Figure of Merit” obtained in this way is set to be proportional to the weighting coefficient (Formula (81)).

When normalization is performed among the weighting factors for the three interpolated images for each pixel, the weighting factor is obtained.

<Image quality improvement effect>
The composite image of the three types of interpolation results obtained by using both the internal weighting and the external weight correction obtained in this way is compared to the interpolation result using a single prediction space that is a conventional square root tone, as shown below. Performance can be achieved. That is,
a) The sharpness of the entire image is improved.
b) Statistically, the color resolution is dramatically improved among most color combinations.
c) Suppression of false color around the high luminance part is improved while maintaining the conventional color moiré suppression capability.
d) Suppress chromatic aberration.

  The improvement in color resolution has the effect of extending the color resolution limit to a high frequency range that exceeds the color resolution range of the normal Bayer array. This point will be described with reference to FIG. FIG. 12 is a diagram showing the resolving power (luminance resolving power) and the chromatic color resolving power (color resolving power) of an achromatic object in the Bayer array in a frequency space (k space). In FIG. 12, a square area surrounded by a line crossing ± π / a of the kx axis and the ky axis is the range of the achromatic resolution unique to the Bayer arrangement. A square area surrounded by a line crossing ± π / (2a) of the kx axis and the ky axis is a range of chromatic color resolution unique to the Bayer array.

  The above extension of the color resolution limit exceeds the frame indicating the color resolving power of the R and B components in FIG. 12 (the chromatic color resolving range unique to the Bayer array) and is close to or near the frame indicating the color resolving power of the G component. As described above, there are many cases where a color combination showing a color resolving power having an area twice or more than that in the frequency space can be obtained.

<Fourth embodiment>
-Bayer interpolation: Internal weighting and gradation Laplacian of each RGB surface-
In the fourth embodiment, three types of gamma space for interpolation are used: linear tone (Γ ₁ ), square root tone with offset (Γ ₂ ), and logarithmic tone (Γ ₃ ). The interpolation value is calculated by correcting the interpolated Laplacian while performing weighted synthesis of the interpolation value predicted in space.

  FIG. 13 is a flowchart for explaining the flow of image processing executed by the CPU of the computer apparatus 100 according to the fourth embodiment. Here, the same processes as those in the flowchart of the first embodiment (FIG. 7) are denoted by the same step numbers and description thereof is omitted. In the same steps as those in the first embodiment, the same operational effects as those in the first embodiment can be obtained. 13 differs from FIG. 7 (first embodiment) in that step S70 and step S80 are performed between step S6 and step S8. Therefore, these differences will be mainly described.

  In step S70 of FIG. 13, the CPU synthesizes the internally weighted image. FIG. 14 is a flowchart for explaining the details of the internally weighted image composition processing. In FIG. 14, “gain calculation” is performed in steps S71 to S75, and “conversion to weighted coefficients” is performed in steps S76 to S78. In steps S79 to S7B, “internal weighted composition” is performed.

"Gain calculation"
The CPU performs the following index calculation for each of the three types using the three types of interpolated RGB planes. Since the processing contents are the same, any one of the three interpolated RGB planes is represented by R (x, y), G (x, y), B (x, y).

(S71) Luminance Contrast Index The CPU generates a luminance contrast index in step S71 of FIG. Specifically, the luminance plane is generated by the following equation (82).
Y (x, y) = [R (x, y) + 2G (x, y) + B (x, y)] / 4 (82)

ＣＰＵは、輝度面のラプラシアン成分を次式（８６）により抽出する。

そして、式（８７）により、輝度コントラストの指標としてラプラシアン成分の二乗を採用する。
（輝度コントラスト指標）= |△Y(x,y)|² …（８７） Next, the CPU performs Gaussian blurring on the luminance surface. As will be apparent from the reference pixel range necessary for interpolation immediately after generating the color difference plane and performing the color difference plane interpolation as described in the demosaic, it becomes the core of color interpolation. The reference pixel range is basically about 7 × 7 pixels. Therefore, it is necessary to collect information on the extent of this extent in order to distinguish between the three interpolation results. Therefore, in this embodiment, Gaussian blur of 7 × 7 size is performed. Here, it is easy to use a separation type filter (following equations (83) to (85)) that convolves a Gaussian blur in the horizontal direction and a Gaussian blur in the vertical direction. Here, 2r _th = 7 is set.

The CPU extracts the Laplacian component of the luminance surface by the following equation (86).

Then, the square of the Laplacian component is adopted as the luminance contrast index according to the equation (87).
(Luminance contrast index) = | △ Y (x, y) | ² (87)

(S72) Color Contrast Index In step S72, the CPU generates a color contrast index. Specifically, the saturation plane is generated by the following equation (88).
C (x, y) = [| R (x, y) -G (x, y) | + | B (x, y) -G (x, y) | + | R (x, y) -B ( x, y) |] ... (88)

そして、式（９１）により、色コントラストの指標としてラプラシアン成分の二乗を採用する。
（色コントラスト指標）= |△C(x,y)|² …（９１） Next, the CPU performs Gaussian blur on the color difference plane. As in the case of the luminance surface, the measurement is performed in a 7 × 7 size range. Therefore, the definition formula of the Gaussian blur filter is expressed by the following formula (89).

Then, the square of the Laplacian component is adopted as an index of color contrast according to the equation (91).
(Color contrast index) = | ΔC (x, y) | ² (91)

(S73) Noise index The CPU generates a noise index in step S73. Specifically, a local standard deviation is adopted as an index representing the noise fluctuation width, the standard deviation is obtained for each of the R, G, and B planes, and the sum of squares thereof is used as a noise index of the interpolation result. Taking the R plane as an example, it is expressed by the following equations (92) and (93). Note that the region for examining the local standard deviation is also set to about 7 × 7 size, which is the core of the reference pixel range for interpolation value calculation. Therefore, 2r _th = 7 is set.

The CPU similarly obtains the G plane and the B plane, and adopts the following equation (94) as a noise index.

(S74) Saturation index In step S74, the CPU generates a saturation index. Specifically, the achromaticity of each pixel is measured as a saturation index. The achromatic chromaticity increases as the occurrence of false colors decreases. The saturation index is expressed by the following equation (95).

(S75) Integrated Index In step S75, the CPU generates an integrated index. The CPU represents the integrated index I (x, y) in each pixel as a product and a quotient of the four indices as in the following equation (96).
(Integrated index) = [(Luminance contrast index) / (Noise index)] / [(Color contrast index) / (Saturation index)]

"Conversion to weighting factor"
In the above formulas (76) to (78), the case of the common internal weighting coefficient has been described. However, in actual operation, the color moiré that is a false color of the chrominance plane is not amplified. The same value is assigned to the internal weighting coefficient for the luminance plane and the internal weighting coefficient for the chrominance plane, and a slight correction is made with the low-pass filter for the weighting coefficient for the chrominance plane. Perform the process. The following explanation is a procedure along this flow.

と表す。 (S76) Standardization of integrated index In step S76, the CPU standardizes the integrated index. Specifically, in the square root tone characteristics (Γ ₂ ) of the interpolated image demodulated through each of the linear tone (Γ ₁ ), the square root tone (Γ ₂ ), and the logarithmic tone (Γ ₃ ). Each integrated index calculated in

It expresses.

When a weighting coefficient for performing internal weighting is represented by w (x, y), each weighting coefficient is represented by the following equations (97) to (99).

(S77) Setting of Weighting Factor for Luminance Surface In step S77, the CPU sets a weighting factor for luminance surface. The CPU sets the weighting coefficients according to the above equations (97) to (99) as they are for the luminance surface as the luminance surface weighting factors (equations (100) to (102)).

ただし、ｂは、例えば３×３サイズの微小なローパスフィルタである。上式右辺において○と×を組合わせた演算記号は、コンボリューションを表す。 (S78) Setting of Weighting Coefficient for Color Difference Surface In step S78, the CPU sets a weighting coefficient for the color difference surface. The CPU multiplies the weighting coefficients according to the above equations (97) to (99) by a small low-pass filter b as the weighting coefficient for the color difference surface, and weights for the color difference surface according to the following equations (103) to (105). Set the coefficient.

However, b is a minute low-pass filter of 3 × 3 size, for example. In the right side of the above formula, an operation symbol combining ○ and × represents convolution.

"Internally weighted composition"
In step S79, the CPU performs conversion from the RGB space to the GCbCr space. Specifically, each of the three interpolated RGB planes is converted into a luminance plane and a color difference plane. However, the luminance surface remains the G component. Therefore, the color difference plane only needs to subtract the G component from the R and B components. The luminance / color difference conversion method is not limited to this.

The CPU obtains the luminance plane and color difference plane predicted by the interpolation gamma space (Γ ₁ ), which is a linear gradation, using the following equations (106) to (108).
G ^{(Γ1 → Γ2)} (x, y) (106)
Cr ^{(Γ1 → Γ2)} (x, y) = R ^{(Γ1 → Γ2)} (x, y) -G ^{(Γ1 → Γ2)} (x, y) (107)
Cb ^{(Γ1 → Γ2)} (x, y) = B ^{(Γ1 → Γ2)} (x, y) -G ^{(Γ1 → Γ2)} (x, y) (108)

The CPU obtains the luminance plane and color difference plane predicted by the interpolation gamma space (Γ ₂ ), which is the square root gradation, by the following equations (109) to (111).
G ^{(Γ2 → Γ2)} (x, y) (109)
Cr ^{(Γ2 → Γ2)} (x, y) = R ^{(Γ2 → Γ2)} (x, y) -G ^{(Γ2 → Γ2)} (x, y) (110)
Cb ^{(Γ2 → Γ2)} (x, y) = B ^{(Γ2 → Γ2)} (x, y) -G ^{(Γ2 → Γ2)} (x, y) (111)

The CPU obtains the luminance plane and the color difference plane predicted by the interpolating gamma space (Γ ₃ ) that is logarithmic gradation, using the following equations (112) to (114).
G ^{(Γ3 → Γ2)} (x, y) (112)
Cr ^{(Γ3 → Γ2)} (x, y) = R ^{(Γ3 → Γ2)} (x, y) -G ^{(Γ3 → Γ2)} (x, y) (113)
Cb ^{(Γ3 → Γ2)} (x, y) = B ^{(Γ3 → Γ2)} (x, y) -G ^{(Γ3 → Γ2)} (x, y) (114)

次式（１１６）は、Ｃｒ面の内分加重合成を表す式である。

次式（１１７）は、Ｃｂ面の内分加重合成を表す式である。

In step S7A, the CPU performs internally weighted synthesis of the luminance / color difference plane. The following equation (115) is an equation representing the internally weighted synthesis of the luminance plane.

The following equation (116) is an equation representing the internally weighted synthesis of the Cr surface.

The following equation (117) is an equation representing the internally weighted synthesis of the Cb surface.

In step S7B, the CPU performs conversion from the GCbCr space to the RGB space. Specifically, by inverse color space conversion, R, G, B planes obtained by internally weighted synthesis in a reference interpolation gamma space (Γ ₂ space) are obtained. However, the G plane may be left as it is.

The CPU obtains the R and B planes obtained by internally weighted synthesis using the following equations (118) to (119).

In step S80 of FIG. 13, the CPU corrects the internally weighted image. The CPU performs processing for correcting the contrast of the internally weighted synthesized image in the Γ ₂ space obtained in step S70 with the inter-tone Laplacian obtained in step S6 (FIG. 13) (the following equation (120)).
R ' ^(Γ2) (x, y) = R _w ^{(Γ2 → Γ2)} (x, y) + k * △ R ^(Γ2) (x, y)
G ' ^(Γ2) (x, y) = G _w ^{(Γ2 → Γ2)} (x, y) + k * ^{ΔG (Γ2)} (x, y) (120)
B ' ^(Γ2) (x, y) = B _w ^{(Γ2 → Γ2)} (x, y) + k * △ B ^(Γ2) (x, y)
However, the value of the constant multiple k usually takes a value of 1 to 0.5.

According to the fourth embodiment described above, the following operational effects can be obtained.
(1) The image processing apparatus converts linear tone image signals into three tone characteristics Γ ₁ , Γ having different curvatures from image data consisting of RGB color components and having missing color components. ₂ and Γ ₃ are converted into image signals in the image processing space (S2), and the missing color components of each of the tone-converted image signals are interpolated to generate three Bayer interpolation images (S4). The CPU and the three kinds of interpolated images are converted into an image in the image processing space having the reference gradation characteristic Γ ₂ to match the gradation characteristics (S5). The CPU and the three gradation characteristics Γ ₁ , Γ ₂ , Γ ₃ Using the three types of Bayer interpolation images interpolated in the above and tone-matched to the reference tone characteristics Γ ₂ , the pixel values and gradations of the interpolated image of the gradation characteristics Γ ₂ are calculated between the same pixels of the three images. difference values, and the pixel value and the gradation of tone characteristic gamma ₂ interpolation image between the pixel value of the characteristic gamma ₁ interpolation image Equalize the difference value between the pixel value of sexual gamma ₃ interpolation image extracting respectively (S6) CPU, relative to the tone characteristics gamma ₂ of interpolated images, the difference value with a weighted combining three interpolated images CPU (S70, S80) for adding the image signal to the pixel, and a CPU for converting the image signal after the addition into an image signal in the output color space (S9). This enables high-definition interpolation that can provide a higher-contrast interpolated image while maximizing the performance of the conventional image prediction space that can perform excellent contrast resolution on average in all gradation regions. be able to.

(2) In the image processing apparatus of (1), gradation conversion is performed. (S2) The CPU uses the linear gradation characteristic Γ ₁ , the square root gradation characteristic Γ ₂ , and the logarithmic gradation characteristic Γ ₃ , respectively. Therefore, the contrast predicted in the gamma space Γ ₁ having the square root gradation and the gamma space Γ ₃ having the logarithmic gradation that cannot be extracted in the linear gradation gamma space Γ ₂ can be reflected to the maximum extent.

(3) In the image processing apparatus according to (1) or (2), the color components are interpolated to generate three Bayer interpolation images. (S4) The CPU uses the missing color component interpolation values as missing colors. Since the calculation is performed using not only the color information of the components but also the color information of other color components, the color components can be appropriately interpolated.

(4) The image processing apparatus converts the linear gradation image signal into three gradation characteristics Γ ₁ and Γ having different curvatures with respect to image data including RGB color components and having missing color components. ₂ and Γ ₃ are converted into image signals in the image processing space (S2), and the missing color components of each of the tone-converted image signals are interpolated to generate three Bayer interpolated images (S4). The CPU and the three kinds of interpolated images are converted into an image in the image processing space having the reference gradation characteristic Γ ₂ to match the gradation characteristics (S5). The CPU and the three gradation characteristics Γ ₁ , Γ ₂ , Γ ₃ Using the three types of Bayer interpolation images interpolated in the above and matched in tone to the reference tone characteristic Γ ₂ , the tone for the pixel value of the interpolated image of the tone characteristic Γ ₂ is obtained between the same pixels of the three images. Laplacian between the pixel value of the characteristic gamma ₁ of the interpolated image and the pixel value of the interpolated image tone characteristic gamma ₃ Extracting a partial and (S6) CPU, relative to the tone characteristics gamma ₂ interpolation image, while the weighted synthesized three interpolation image, adds the Laplacian components in the same pixel and (S70, S80) CPU, a I prepared. This enables high-definition interpolation that can provide a higher-contrast interpolated image while maximizing the performance of the conventional image prediction space that can perform excellent contrast resolution on average in all gradation regions. be able to.

(5) In the image processing apparatus of (4), gradation conversion is performed (S2) so that the CPU uses the linear gradation characteristic Γ ₁ , the square root gradation characteristic Γ ₂ , and the logarithmic gradation characteristic Γ ₃ , respectively. Therefore, the contrast predicted in the gamma space Γ ₁ having the square root gradation and the gamma space Γ ₃ having the logarithmic gradation that cannot be extracted in the linear gradation gamma space Γ ₂ can be reflected to the maximum extent.

(6) In the image processing apparatus according to (4) or (5), the color component is interpolated to generate three Bayer interpolation images. (S4) The CPU uses the missing color component interpolation value as the missing color. Since the calculation is performed using not only the color information of the components but also the color information of other color components, the color components can be appropriately interpolated.

(Modification 3)
A protrusion removal filter may be applied to the Bayer plane subjected to logarithmic gradation conversion. Since the interpolation gamma space having a logarithmic gradation amplifies a dark portion with high contrast, there is a possibility of amplifying a protruding point caused by a dark current. In this case, as a result, there are cases where black scratches or white scratches that occur during long-second exposure photography appear. The protruding point removal filter is effective in dealing with such a situation.

  The protruding point removal filter is disclosed in the first embodiment of Japanese Patent Application Laid-Open No. 2011-135666 by the same inventor as that of the present applicant, or described in Japanese Patent Application No. 2012-281167 in which this is further enhanced. Use a good one. Specifically, in the above-described embodiment, a logarithmic gradation Bayer plane is formed between the gradation conversion (three ways) of the Bayer image in “Step S2” and the Bayer interpolation (three ways) in “Step S4”. Apply the protruding point removal filter.

  The above-described principle is valid not only in the illustrated Bayer array, but also in other primary color system arrays and complementary color system arrays.

  The above description is merely an example, and is not limited to the configuration of the above embodiment. You may combine the said embodiment and modified example suitably.

DESCRIPTION OF SYMBOLS 100 ... Personal computer 200 ... Camera 201 ... Operation member 202 ... Imaging element 203 ... Control apparatus 205 ... Monitor

Claims (9)

For image data consisting of at least two color components and having missing color components, an image signal in the input color space is converted into an image signal in the image processing space, and interpolation processing of the missing color components in the image processing space is performed In the image processing apparatus that converts the image signal after the interpolation processing into an image signal in the output color space,
A gradation conversion unit that converts the image signal into two gradation characteristics having different first and second curvatures;
A color interpolation unit that interpolates missing color components of each of the tone-converted image signals to generate at least two types of interpolation images;
A gradation matching unit that converts the at least two kinds of interpolated images into an image in an image processing space having one gradation characteristic and matches the gradation characteristic;
Using the first and second interpolated images interpolated with the first and second gradation characteristics and gradation-matched with one gradation characteristic, at least a second value is obtained between the same pixels of the two images. A difference value extraction unit that extracts a difference value between the pixel value of the interpolation image and the pixel value of the first interpolation image;
An adding unit that adds the difference value to the same pixel with respect to the second interpolation image;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The gradation converter
As the first gradation characteristic, a linear gradation characteristic, a cubic root gradation characteristic, or a logarithmic gradation characteristic,
The second gradation characteristic is a square root gradation characteristic or a gradation characteristic with a positive offset in the square root.
An image processing apparatus using each of them. The image processing apparatus according to claim 1 or 2,
The color interpolation unit calculates an interpolation value of a missing color component using not only the color information of the missing color component but also the color information of another color component. Interpolation processing of the missing color component in the image processing space by converting an image signal in the input color space into an image signal in the image processing space for image data including at least two color components and having the missing color component In the image processing apparatus that converts the image signal after the interpolation processing into an image signal in the output color space,
A gradation conversion unit that converts the image signal into three gradation characteristics having different curvatures in the order of the first, second, and third;
A color interpolation unit that interpolates missing color components of each of the tone-converted image signals to generate three types of interpolated images;
A gradation matching unit that converts the three types of interpolation images into an image in an image processing space having a single gradation characteristic to match the gradation characteristic;
Using the first to third interpolation images interpolated with the first to third gradation characteristics and gradation-matched to one gradation characteristic, the second interpolation is performed between the same pixels of the three images. A Laplacian extraction unit that extracts a Laplacian component between the pixel value of the first interpolation image and the pixel value of the third interpolation image with respect to the pixel value of the image;
An adder that adds the Laplacian component to the same pixel for the second interpolated image;
An image processing apparatus comprising: The image processing apparatus according to claim 4.
The gradation converter
Linear gradation characteristics as the first gradation characteristics,
The second gradation characteristic is a square root gradation characteristic or a gradation characteristic with a positive offset in the square root.
As the third gradation characteristic, a cubic root gradation characteristic or a logarithmic gradation characteristic,
An image processing apparatus using each of them. The image processing apparatus according to claim 4 or 5,
The color interpolation unit calculates an interpolation value of a missing color component using not only the color information of the missing color component but also the color information of another color component. For image data composed of at least two color components and having missing color components, an image signal in the input color space has at least two gradation characteristics having different first and second curvatures. Conversion processing for gradation conversion to an image signal in the image processing space;
Interpolating the missing color component of each of the image signals subjected to the gradation conversion to generate at least two types of interpolated images; and
A gradation matching process for converting the at least two kinds of interpolation images into an image in an image processing space having one gradation characteristic to match the gradation characteristic;
Using the two types of interpolated images interpolated with the different gradation characteristics and gradation-matched with the one gradation characteristic, at least the pixel value of the second interpolation image between the same pixels of the two images A calculation process for calculating a difference value between pixel values of the first interpolation image;
An addition process for adding the difference value to the same pixel with respect to the second interpolation image;
An image processing program for causing a computer to execute. The image data is composed of at least two color components, and the image data of the input color space is different from the image data in which the missing color component exists in three different curvatures in the order of at least the first, second, and third. A conversion process that performs gradation conversion into an image signal in an image processing space of gradation characteristics;
Generation processing for interpolating the missing color component of each image signal subjected to the gradation conversion to generate three types of interpolation images;
A gradation matching process for converting the three kinds of interpolated images into an image in an image processing space having one gradation characteristic to match the gradation characteristic;
Using the first to third interpolated images interpolated with the first to third gradation characteristics and gradation-matched to the one gradation characteristic, the second pixel is extracted between the same pixels of the three images. A calculation process for calculating a Laplacian component between the pixel value of the first interpolation image and the pixel value of the third interpolation image with respect to the pixel value of the interpolation image;
An addition process for adding the Laplacian component to the same pixel for the second interpolation image;
An image processing program for causing a computer to execute. An imaging unit that captures a subject image;
The image processing apparatus according to any one of claims 1 to 6, wherein image processing is performed on an image captured by the imaging unit;
An imaging apparatus comprising:

Images