JP4703666B2 - Image processing apparatus, photographing apparatus, image processing method, image processing program, and recording medium - Google Patents

Description

  According to the present invention, from a color signal output from a solid-state imaging device in which a plurality of types of color filters are discretely arranged in each pixel unit, all color signals in each pixel are obtained, and color signals of a target pixel and its neighboring pixels are obtained. Image processing apparatus generated by interpolation processing used, single-plate digital camera using the same, photographing apparatus such as video camera and camera-equipped mobile phone apparatus, image processing method using the image processing apparatus, and execution of this on a computer The present invention relates to an image processing program for causing a computer to read and a computer readable recording medium storing the image processing program.

   Conventionally, only information relating to the brightness of light is generally output from a solid-state imaging device such as a CCD or CMOS used in, for example, a digital camera or a video camera.

   Therefore, in a single-plate camera with one solid-state image sensor, a color filter that passes only one color component for each pixel included in the CCD is placed in front of the CCD in order to obtain color information. Each pixel unit is configured to output a corresponding color component. For example, in the case of a primary color filter, there are three types of color components: R (red), G (green), and B (blue) of the three primary colors.

   Several types of arrangement of each color component of these color filters have been proposed. A typical example is a Bayer array as shown in FIG. In this Bayer array, G having a large contribution to the luminance signal is arranged in a checkered pattern, and R and B are arranged in a checkered pattern in the remaining portions.

   At the time when a signal is output from the CCD covered with such a color filter such as a Bayer array, only color information for one color among the RGB color components is obtained for each pixel.

   Therefore, the image processing unit performs interpolation processing for obtaining the remaining color information by estimating and calculating the information for the remaining two colors for each pixel from the color signal values of the surrounding pixels.

   Furthermore, interpolation processing may be performed on the color information for one color for each pixel obtained by the CCD from the color signal value of the pixel of interest itself and the color signal value of the surrounding pixels (neighboring pixels). .

   Below, the bilinear method which is the conventional interpolation processing method is demonstrated using FIG.

   FIG. 12A is a diagram illustrating a Bayer array when the color component of the target pixel is G, and FIG. 12B is a diagram illustrating the Bayer array when the color component of the target pixel is R or B. is there. The symbols P and Q represent the color components R and B, or B and R, respectively. When P represents R, Q represents B. Conversely, when P represents B, Q represents R. .

The interpolation formula (Formula 1) in the bilinear method when the color filter is a Bayer array is as follows.
(Formula 1)
When the pixel of interest is G33 as shown in FIG. 12A (the color element of the pixel of interest is G), Equation 1 is
Go = G33 or Go = (G33 × 4 + G22 + G42 + G24 + G44) / 8 Po = (P32 + P34) / 2
Qo = (Q23 + Q43) / 2
It is.

As shown in FIG. 12B, when the target pixel is P33 (the color element of the target pixel is R or B), Equation 1 is
Go = (G32 + G23 + G43 + G34) / 4
Po = P33
Qo = (Q22 + Q42 + Q24 + Q44) / 4
It is.

However, Go: Output G signal of the target pixel Po: Output P signal of the target pixel Qo: Output Q signal of the target pixel At this time, (P, Q) = (R, B) or (B, R)
However, when color signal interpolation processing is performed on an image that changes from white to black using the conventional bilinear method, a false color signal (hereinafter referred to as a false color signal) is generated at the edge portion. It has been known.

   As a method for reducing such false color signals, for example, Patent Document 1 proposes an interpolation processing method that suppresses the generation of false color signals by utilizing the correlation of colors.

   As a general image feature, “in the local region of the color signal, the ratio of the low-frequency component of the specific color component to the high-frequency component is the same in other color components” and “the color signal In the local region, the difference between the G signal and the R signal, or the difference between the G signal and the B signal is substantially equal between adjacent pixels.

By utilizing these properties, the interpolation processing method of Patent Document 1 detects the correlation between the vertical direction (vertical direction) and the horizontal direction (horizontal direction) from the color signals arranged in the Bayer array. Interpolation processing is performed using characteristics. Specifically, the color signal by the interpolation process when there is a correlation in the vertical direction and the color signal by the interpolation process when there is a correlation in the horizontal direction are calculated in advance, and the correlation in the vertical direction and the correlation in the horizontal direction are calculated. In accordance with the intensity, the color signal obtained by the interpolation processing for each direction is mixed.
Japanese Patent No. 2931520

   However, even with the conventional interpolation processing method described in Patent Document 1, the false color signal is reduced as compared with the case where the bilinear method is used. However, in a general image, there are many edges in various directions other than the vertical direction (longitudinal direction) and the horizontal direction (lateral direction). In the interpolation processing method of Patent Document 1, with respect to a large number of edge directions as described above, it is obtained by interpolation processing in two directions according to two types of correlation data (correlation strength) in the vertical direction and the horizontal direction. Since the received color signals are mixed, a false color signal may be generated by an oblique edge.

   For example, a case where a solid-state imaging device such as a CCD has a color filter as shown in FIG. 13B and a monochrome color signal as shown in FIG. 13A is obtained by A / D conversion will be described. .

FIG. 13A corresponds to a part of an edge portion that is slightly inclined to the upper right side from the vertical direction. When interpolation processing is performed on the color signal value of FIG. 13A by correlation using a color difference of 5 pixels × 3 pixels as described in Patent Document 1, each color signal is
Rv = G33 + (R32 + R34) / 2− (G31 + 2 × G33 + G35) / 4
= 64
Gv = G33
= 2
Bv = G33 + B23-G23
= G33 + B23− {B23 + (G22 + G24) / 2− (B21 + 2 × B23 + B25) / 4}
= -60
Rh = G33 + R32-G32
= G33 + R32- {R32 + (G22 + G42) / 2- (R12 + 2 × R32 + R52) / 4}
= 64
Gh = G33
= 2
Bh = G33 + (B23 + B43) / 2− (G13 + 2 × G33 + G53) / 4
= 64
It becomes.

  However, the color signals Rv, Gv, and Bv are color signals when processing suitable for the case where the correlation in the vertical direction is strong is performed. The color signals Rh, Gh, and Bh are color signals when processing suitable for a case where the correlation in the horizontal direction is strong is performed.

   When the calculation formula for the directional correlation coefficient described in Patent Document 1 is used, the horizontal correlation coefficient is 0. Therefore, when obtaining the color signal, the color signal only in the vertical direction is referred to. Will be.

Therefore,
Ro = Rv = 64
Go = Gv = 2
Bo = Bv = −60
It becomes.

   Originally, a monochrome image has R = G = B, whereas a calculated color signal has Ro ≠ Go ≠ Bo. This indicates that a false color signal is generated.

   This will be examined in more detail.

  In the interpolation processing using the color correlation described in Patent Document 1, the G component and the G component are included in the vertical direction and the horizontal direction of the color filters arranged in the Bayer array. Other color components of the pixel of interest are obtained by correlation with other color components. For example, the horizontal direction in FIG. 12A will be described as an example.

   As shown in FIG. 12A, the horizontal line including the target pixel G33 includes G and Q. Therefore, Q33 which is the Q component of the target pixel can be obtained from these values. A horizontal line including P32 on one line includes G and P. Therefore, G32 which is the G component of the target pixel can be obtained from these values.

In the interpolation processing method described in Patent Document 1, as described above, paying attention to the pixels P32 and G33 shown in FIG. 12A, the difference between G and P in the local region is approximately equal. (P33-G33 = P32-G32)
Or the property that the ratio of G and P in the local region is almost equal (P33 / G33 = P32 / G32)
Is used to obtain P33.

   Since edges are in various directions with respect to a general image, it is preferable to detect not only two directions, vertical and horizontal, but also a large number of directions.

   However, with the interpolation processing method described in Patent Document 1, it is not possible to perform an interpolation process in consideration of the color correlation in the oblique direction.

   This is because, for example, in FIG. 12A, only the G component exists in the direction of 45 degrees diagonally, and in FIG. 12B, the P component and the Q component (B component) exist in the direction of 45 degrees diagonally. And only the R component).

   The present invention solves the above-described conventional problems. From a color signal output from a solid-state image pickup device in which a plurality of color filters are discretely arranged in each pixel unit, such as a Bayer array, in each pixel. When all the color signals are generated by interpolation processing using the color signals of the target pixel and its neighboring pixels, the false color signal of the false color signal is obtained by utilizing the correlation of the color signals with respect to edges in a plurality of directions including the oblique direction. Image processing apparatus capable of further suppressing generation, photographing apparatus using the same, image processing method using the image processing apparatus, image processing program for causing a computer to execute the same, and computer storing the image processing program It is an object to provide a readable recording medium that can be read.

   The image processing apparatus according to the present invention, from a color signal output from a solid-state image pickup device in which a plurality of types of color filters are discretely arranged in each pixel unit, converts all color signals in each pixel into a target pixel and its neighboring pixels In the image processing apparatus having the data interpolation unit that is generated by the data interpolation process using the color signal of the color signal, the data interpolation unit quantifies the directionality of the edge with respect to a plurality of directions including the oblique direction near the target pixel. Directionality data calculation means for calculating data, directionality detection means for detecting the directionality of the edge near the target pixel using the directionality data for the plurality of directions calculated by the directionality data calculation means, Directional color signal calculation means for calculating a color signal corresponding to the directionality of one or a plurality of edges detected by the directionality detection means, and calculation by the directionality color signal calculation means Been provided with the target pixel color signal calculation means for calculating a color signal of the pixel of interest using one or more color signals, the object is achieved.

   Preferably, the directionality data calculation means in the image processing apparatus of the present invention calculates a value corresponding to the degree of approximation between the designated direction and the actual edge direction in the vicinity of the target pixel as the directionality data.

   Further preferably, the directionality data calculation means in the image processing apparatus of the present invention calculates a smaller value as the degree of approximation as the directionality data.

   Further preferably, in the image processing apparatus according to the present invention, the plurality of directions including the oblique direction are a plurality of directions including at least one of an upper right oblique direction and an oblique lower right direction in the vicinity of the target pixel.

   Further preferably, the directionality data calculation means in the image processing apparatus of the present invention calculates the directionality data with respect to four directions including a right upper direction, a horizontal direction, a right lower direction and a vertical direction passing through the target pixel. To do.

   Further preferably, the directionality data calculation means in the image processing apparatus of the present invention is configured so that the pixel of interest has an upper direction, an oblique upper right direction, a right direction, an oblique lower right direction, a downward direction, an oblique lower left direction, a left direction and an oblique direction. The directionality data is calculated for the eight directions in the upper left direction.

   Further preferably, when the directionality data calculation means in the image processing apparatus of the present invention quantifies the directionality of the edge with respect to a plurality of directions in the vicinity of the target pixel, the color signal that changes depending on the presence or absence of the edge is used. Consider the strength of the correlation.

   Further preferably, the directionality detecting means in the image processing apparatus of the present invention has no brightness gradient near the target pixel depending on whether or not the maximum value of the directionality data for the plurality of directions is smaller than a predetermined threshold value. Is determined to be a gentle flat portion, and “no direction” is detected in the case of the flat portion, and “directional” is detected in the case of the flat portion.

   Further preferably, when the directionality detecting means in the image processing apparatus of the present invention detects the “with directionality”, the direction in which the value of the directionality data is the minimum value is set as the edge direction in the vicinity of the target pixel. To detect.

   Further preferably, even when the directionality detecting means in the image processing apparatus of the present invention detects the “with directionality”, there are a plurality of directions in which the value of the directionality data has a minimum value. Detects “no directionality”.

   Further preferably, the directionality detection means in the image processing apparatus of the present invention detects one or more directions as the directionality candidates of the edges in the vicinity of the target pixel from the respective directionality data for a plurality of directions in ascending order of values.

   Further preferably, the directional color signal calculation means in the image processing apparatus of the present invention calculates a difference between two different color signals in consideration of the directionality of the edge at an arbitrary position near the target pixel. And a color signal calculation means for calculating a color signal using the correlation of the color signal difference calculated by the color difference calculation means.

   Further preferably, the directional color signal calculation means in the image processing apparatus of the present invention calculates a ratio of two different color signals in consideration of the directionality of the edge at an arbitrary position near the target pixel. And color signal calculation means for calculating a color signal using the correlation of the ratios of the color signals calculated by the color ratio calculation means.

   Further preferably, the directional color signal calculation means in the image processing apparatus of the present invention is arranged such that the position of the target pixel, the center position of the vicinity pixel of the target pixel, and the target position are arbitrary positions in the vicinity of the target pixel. Any of intersections between adjacent pixels in the vicinity of the pixel is used.

   Further preferably, the directional color signal calculation means in the image processing apparatus of the present invention detects the directional color signal when the pixel of the color component to be calculated does not exist with respect to the direction detected by the directional detection means. A color signal corresponding to the directionality of the edge is calculated with reference to the pixel of the color component to be calculated from the neighboring pixels of the target pixel other than the determined direction.

   Further preferably, the directional color signal calculation means in the image processing apparatus of the present invention uses an average value of the color components to be calculated in a plurality of neighboring pixels of the target pixel other than the detected direction.

   Further preferably, the pixel-of-interest color signal calculation means in the image processing apparatus of the present invention is calculated by the directional color signal calculation means for the pixels detected as “directional” by the directional detection means. The obtained color signal is calculated as the color signal in the target pixel.

   Further preferably, the pixel-of-interest color signal calculation means in the image processing apparatus of the present invention uses the bilinear method for the pixel signal detected by the directionality detection means as “no directionality”, and the color signal at the pixel of interest. Is calculated.

   Further preferably, the pixel-of-interest color signal calculation means in the image processing apparatus of the present invention calculates an average color of each color signal corresponding to a plurality of directions detected by the directionality detection means as a color signal in the pixel of interest. Average color calculation means is included.

   Further preferably, the target pixel color signal calculation means in the image processing apparatus of the present invention comprises: an average color calculation means for calculating an average color of each color signal corresponding to a plurality of directions detected by the directionality detection means; A multi-color correlated color signal calculating unit that calculates a color signal at the target pixel using the correlation between the average color calculated by the average color calculating unit and the color of the target pixel;

   The imaging device of the present invention includes a solid-state imaging device that can image a subject, an analog / digital conversion unit that performs analog / digital conversion on imaging data from the solid-state imaging device, and a digital signal from the analog / digital conversion unit. The image processing apparatus according to claim 1, which performs signal processing, achieves the object.

   In the image processing method of the present invention, from a color signal output from a solid-state imaging device in which a plurality of types of color filters are discretely arranged in each pixel unit, all color signals in each pixel are converted to the target pixel and its neighboring pixels. A directionality data calculation step for calculating directionality data in which the directionality of edges with respect to a plurality of directions including an oblique direction in the vicinity of the target pixel is quantified A directionality detection step for detecting the directionality of an edge in the vicinity of the target pixel using the directionality data for the plurality of directions calculated in the directionality data calculation step, and one or more detected in the directionality detection step A directional color signal calculation step for calculating a color signal corresponding to the directionality of a plurality of edges, and one or more calculated in the directional color signal calculation step. And a target pixel color signal calculation step of calculating a color signal of the pixel of interest by using the color signal, the object is achieved.

   Preferably, the directionality data calculation step in the image processing method of the present invention calculates a value corresponding to the degree of approximation between the designated edge direction and the actual edge direction near the target pixel as the directionality data.

   Further preferably, the directionality data calculation step in the image processing method of the present invention calculates a smaller value as the degree of approximation as the directionality data.

   Further preferably, in the directionality data calculation step in the image processing method of the present invention, the directionality data is calculated with respect to four directions including a right upper direction, a horizontal direction, a right lower direction and a vertical direction passing through the target pixel. To do.

   Further preferably, the directionality data calculation step in the image processing method of the present invention is performed in the upward direction, the diagonally upper right direction, the right direction, the diagonally lower right direction, the downward direction, the diagonally lower left direction, the left direction and the diagonal direction with the target pixel as the center. The directionality data is calculated for the eight directions in the upper left direction.

   Further preferably, in the directionality data calculation step in the image processing method of the present invention, when quantifying the directionality of the edge with respect to a plurality of directions in the vicinity of the target pixel, the color signal that changes depending on the presence or absence of the edge Consider the strength of the correlation.

   Further preferably, in the directionality detecting step in the image processing method of the present invention, the luminance gradient has no edge near the target pixel depending on whether or not the maximum value of the directionality data for a plurality of directions is smaller than a predetermined threshold value. It is determined whether or not it is a gentle flat portion, and “no direction” is detected in the case of the flat portion, and “directional” is detected in the case of not being the flat portion.

   Further preferably, in the directionality detection step in the image processing method of the present invention, when the “with directionality” is detected, a direction in which the value of the directionality data is a minimum value is set as an edge direction in the vicinity of the target pixel. To detect.

   Further preferably, in the directionality detection step in the image processing method of the present invention, even when the “with directionality” is detected, there are a plurality of directions in which the value of the directionality data has a minimum value. Detects “no directionality”.

   Further preferably, in the directionality detection step in the image processing method of the present invention, one or more directions are detected as directionality candidates in the edge direction in the vicinity of the target pixel from the directionality data for a plurality of directions in ascending order.

   Further preferably, in the directional color signal calculation step in the image processing method of the present invention, the color difference calculation for calculating a difference between two different color signals in consideration of the directionality of the edge at an arbitrary position in the vicinity of the target pixel. And a color signal calculation step of calculating a color signal using the correlation of the color signal difference calculated in the color difference calculation step.

   Still preferably, in the image processing method of the present invention, the directional color signal calculation step calculates a ratio between two different color signals in consideration of the directionality of the edge at an arbitrary position in the vicinity of the target pixel. A calculation step, and a color signal calculation step of calculating a color signal using the correlation of the ratio of the color signals calculated in the color ratio calculation step.

   Further preferably, the directional color signal calculation step in the image processing method of the present invention includes the position of the target pixel, the center position of the neighboring pixel of the target pixel, and the target position as an arbitrary position in the vicinity of the target pixel. Any of intersections between adjacent pixels in the vicinity of the pixel is used.

   Further preferably, in the directional color signal calculation step in the image processing method of the present invention, when there is no pixel of a color component to be calculated with respect to the detection direction detected by the directional detection means, A color signal corresponding to the directionality of the edge is calculated from a plurality of neighboring pixels of the target pixel other than the detected direction with reference to the pixel of the color component to be calculated.

   Further preferably, the directional color signal calculation step in the image processing method of the present invention uses an average value of the color components to be calculated in a plurality of neighboring pixels of the target pixel other than the detected direction.

   Further preferably, in the image processing method of the present invention, the pixel-of-interest color signal calculation step calculates the color signal calculated in the directionality detection step for the pixel detected as “directional” in the directionality detection step. Is calculated as a color signal in the target pixel.

   Still preferably, in the image processing method of the present invention, the pixel-of-interest color signal calculation step calculates a color signal at the pixel of interest using a bilinear method for the pixel detected as “no directionality” in the directionality detection step. calculate.

   More preferably, the pixel-of-interest color signal calculation step in the image processing method of the present invention calculates an average color of each color signal corresponding to a plurality of directions detected in the directionality detection step as a color signal in the pixel of interest. An average color calculating step;

   Further preferably, the pixel-of-interest color signal calculation step in the image processing method of the present invention includes an average color calculation step of calculating an average color of each color signal corresponding to a plurality of directions detected in the directionality detection step; A multi-color correlated color signal calculating step for calculating a color signal at the target pixel using the correlation between the average color calculated at the average color calculating step and the color of the target pixel.

   An image processing program of the present invention is for causing a computer to execute each processing step of the image processing method according to any one of claims 22 to 40, thereby achieving the object.

   A readable recording medium according to the present invention is a computer-readable recording medium storing the image processing program according to claim 41, whereby the above object is achieved.

   The operation of the present invention will be described below with the above configuration.

   As described above, a general image has a characteristic in the local region that “the difference in G, R, and B and the ratio of G, R, and B are almost constant because there is little change in the color signal in the local region. There is a nature of “is.

   The conventional interpolation processing method uses the color correlation in the horizontal direction and the vertical direction by using the feature in the local region of the image and the feature that the vertical and horizontal directions of the Bayer array always include the G component. Interpolation was performed.

   On the other hand, in the present invention, as in the case of the conventional method described above, the feature in the local region of the image is used, and when performing the interpolation in the oblique direction, not only the oblique direction but also a wider range of pixels. In addition, if necessary, the color signal at an arbitrary point (for example, the intersection of the pixel and the pixel) is calculated regardless of the center of the pixel, and the color correlation with the calculated color signal is used. By calculating the color signal of the pixel of interest, it is possible to perform an interpolation process that also uses the correlation of the color signal with respect to the edge in the oblique direction.

   As described above, in the Bayer array interpolation processing, it is possible to reduce the error of the color signal by checking more directionality and calculating the color signal adapted to each directionality. Can be further suppressed.

  That is, in the present invention, since the color signal can be calculated using the color correlation in the oblique direction, it is more accurate than the conventional interpolation processing method using only the color correlation in the vertical and horizontal directions. Therefore, it is possible to detect the direction of the correct image, and to calculate the color signal while suppressing the generation of the false color signal.

  Further, by performing directionality detection in eight directions centering on the pixel of interest and calculating a color signal using color correlation, interpolation processing with fewer false color signals can be performed in consideration of edge points. Is possible.

  Furthermore, it is possible to further reduce the false color signal by calculating the color signal using the color signal using the color signal calculated using the characteristics of the color correlation.

   As described above, according to the present invention, generation of a false color signal can be achieved by performing the interpolation process using the feature of the local region of the image and also using the correlation of the color signal with the edge in the oblique direction. Further, it is possible to calculate a suppressed color signal.

  In addition, by performing color signal calculation by detecting directionality in eight directions with the pixel of interest at the center, interpolation processing with fewer false color signals can be performed.

  Furthermore, the false color signal can be further reduced by calculating the color signal using the color correlation using the color signal calculated using the characteristics of the color correlation.

Hereinafter, a case where the first to fifth embodiments of the image processing apparatus of the present invention are applied to a photographing apparatus will be described in detail with reference to the drawings.
(Embodiment 1)
FIG. 1 is a block diagram illustrating a configuration example of a photographing apparatus using the first embodiment of the image processing apparatus of the present invention.

   As illustrated in FIG. 1, the imaging device 10 according to the first embodiment includes a solid-state imaging device 1 that can image a subject, an A / D conversion unit 2 that performs A / D conversion on imaging data from the solid-state imaging device 1, and the like. And an image processing unit 3 as an image processing apparatus for performing various signal processing on the digital signal from the A / D conversion unit 2.

   In the solid-state imaging device 1, subject light is incident on a photodiode portion through a predetermined color filter, and is photoelectrically converted into an analog electric signal representing a color size corresponding to each pixel by the photodiode portion and output. Note that the color filter in this case is a Bayer array with three primary colors of RGB.

   The A / D conversion unit 2 converts an analog electric signal output from the solid-state imaging device 1 into a digital signal (hereinafter referred to as Bayer data) and outputs the digital signal.

   The image processing unit 3 includes a white balance unit 4, a gamma correction unit 5, and a data interpolation unit 6. The Bayer data output from the A / D conversion unit 2 is input, and white balance, gamma correction, and data are input. After various image processing such as interpolation is performed, image data including all RGB data for each pixel unit is output from the image processing unit 3.

The white balance unit 4 receives the Bayer data output from the A / D conversion unit 2 and multiplies each pixel value of the Bayer data by a corresponding RGB gain to perform white adjustment. Bayer data is output.

   The gamma correction unit 5 receives the Bayer data after white adjustment output from the white balance unit 4, and adjusts the luminance gradation (gamma correction) for each pixel value of the Bayer data so as to correspond to the display unit. And Bayer data after gamma correction is output.

   The data interpolation unit 6 receives the Bayer data after the gamma correction output from the gamma correction unit 5 and inputs an edge (luminance value) in the vicinity of the target pixel for each pixel unit arranged in a two-dimensional matrix. The presence or absence of the edge and the directionality of the edge are detected by arithmetic processing using the color signal of the target pixel and its neighboring pixels, and the presence or absence of this edge and the detected edge Data interpolation processing according to directionality is performed, and RGB signals are output for each pixel.

   FIG. 2 is a block diagram illustrating a configuration example of the data interpolation unit 6 of FIG.

   As shown in FIG. 2, the data interpolation unit 6 controls the directional data calculation unit 61, the directional detection unit 62, the directional color signal calculation unit 63, and the target pixel color signal calculation unit 64. A controller 65 for detecting the presence / absence of an edge and the directionality of the edge, and performing data interpolation processing according to the detected directionality of the edge. The directionality data calculation unit 61 and the control unit 65 constitute a directionality data calculation unit. The directionality detection unit 62 and the control unit 65 constitute a directionality detection unit, and the directionality color signal calculation unit 63 and the control unit 65 constitute a directionality color signal calculation unit, and the target pixel color signal calculation unit. 64 and the control unit 65 constitute a pixel color signal calculation unit of interest.

   The directionality data calculation unit 61 has, for each pixel of the target Bayer data, directionality data that decreases in value as the degree of approximation between the specified direction including at least an oblique direction and the actual edge direction in the vicinity of the target pixel increases. , Respectively, for a plurality of predetermined designated directions.

   The directionality detection unit 62 uses the respective directionality data for a plurality of designated directions calculated by the directionality data calculation unit 61, and the presence / absence of an edge and the edge in the vicinity of the target pixel (including the vicinity pixel passing through the target pixel). Directionality is detected.

   The directional color signal calculation unit 63 calculates a color signal corresponding to the directionality of the edge detected by the directionality detection unit 62. In the first embodiment, the directional color signal calculation unit 63 is a color difference calculation unit that calculates a difference between two different color signals in consideration of the directionality at an arbitrary point (position on the screen) near the target pixel. And a color signal calculation unit (not shown) as color signal calculation means for calculating a color signal using the correlation of the color signal difference calculated by the color difference calculation unit. have.

   The pixel-of-interest color signal calculation unit 64 calculates the color signal of the pixel of interest using the directionality of the edge detected by the directionality detection unit 62 and the color signal calculated by the directionality color signal calculation unit 63. The

   The control unit 65 is connected to the directional data calculation unit 61, the directional detection unit 62, the directional color signal calculation unit 63, and the target pixel color signal calculation unit 64, respectively. Control the flow of processing.

   Hereinafter, the processing procedure of the data interpolation processing by the data interpolation unit 6 of FIGS. 1 and 2 will be described in order with reference to FIG.

  FIG. 3 is a flowchart for explaining the data interpolation operation of the data interpolation unit 6 of FIG. Note that the processing in steps S1 to S4 in FIG. 3 is repeated by the number of pixels in the interpolation designation area in the input Bayer data area in step S5. In addition, as shown in FIG. 4, the interpolation designation area surrounded by a chain line fits within the Bayer data area surrounded by a solid line, and outside the interpolation designation area, Bayer data having a width of 2 pixels or more in both the upper, lower, left and right directions. Shall exist. This is to make it possible to obtain information on neighboring pixels as much as necessary for the interpolation processing with respect to the pixels at the boundary portion of the interpolation designated area.

   As shown in FIG. 3, first, in step S <b> 1, the directionality data calculation unit 61 calculates directionality data for each target pixel for a plurality of predetermined designated directions.

   In the first embodiment, as shown in FIG. 5, four directions of upper right, horizontal, lower right, and vertical are used as a plurality of designated directions. Not only these designated directions but also designated directions other than those mentioned above are conceivable, but in a very small area, it can be approximated to the above four directions.

   Further, as a plurality of neighboring pixels located in the vicinity of the pixel of interest, in the first embodiment, for example, as shown in FIG. 12A, eight neighboring pixels adjacent to the pixel of interest are further surrounded. 16 neighboring pixels are used, but any number of neighboring pixels is used as long as it is a pixel necessary for data interpolation processing and is located so as to surround the target pixel. be able to.

   The directionality data calculation unit 61 quantifies the directionality of edges in a plurality of designated directions (here, four directions) including the upper right direction, the horizontal direction, the lower right direction, and the vertical direction in the vicinity of the target pixel. Calculate the data. That is, as the directionality data, a value corresponding to the degree of approximation between the designated direction (designated direction) and the actual edge direction in the vicinity of the target pixel is calculated. For example, the directionality data is calculated such that the value decreases as the degree of approximation between the specified direction (specified direction) and the actual edge direction in the vicinity of the target pixel increases. Various types of directional data calculation formulas are conceivable. For example, the following calculation formula (Formula 2) can be used.

However, DDru: Directional data calculation formula in the upper right direction
DDho: Horizontal directionality data calculation formula
DDrd: Directional data calculation formula in the diagonally lower right direction
DDve: Formula for calculating directionality data in the vertical direction.
(Formula 2)
When the target pixel is G33 (see FIG. 12A)
DDru = {(| G51−G33 | + | G42−G33 | + | G24−G33 | + | G15−G33 |
+ | G22−G31 | + | G22−G13 | + | G44−G53 | + | G44−G35 |) / 2
+ (| P32−P14 | + | P52−P34 |)
+ (| Q41−Q23 | + | Q43−Q25 |)} / 2
DDho = {(| G13−G33 | + | G53−G33 | + | G22−G42 | + | G24−G44 |)
+ (| Q23−Q43 |) × 2
+ (| P12−P32 | + | P52−P32 | + | P14−P34 | + | P54−P34 |) / 2} / 2
DDrd = {(| G11−G33 | + G22−G33 | + | G44−GG33 | + | G55−G33 |
+ | G42−G31 | + | G42−G53 | + | G24−G13 | + | G24−G35 |) / 2
+ (| P32−P54 | + | P12−P34 |)
+ (| Q21−Q43 | + | Q23−Q45 |)} / 2
DDve = {(| G31−G33 | + | G35−G33 | + | G22−G24 | + | G42−G44 |)
+ (| P32−P34 |) × 2
+ (| Q21-Q23 | + | Q25-Q23 | + | Q41-Q43 | + | Q45-Q43 |) / 2} / 2
When the target pixel is P33 (see FIG. 12B)
DDru = (| P15−P33 | + | P51−P33 |)
+ (| Q42−Q24 |)
+ (| G32−G23 | + | G43−G34 |) / 2
DDho = (| P13−P33 | + | P53−P33 |)
+ (| G23−G43 |)
+ (| Q22-Q42 | + | Q24-Q44 |) / 2
DDrd = (| P11−P33 | + | P55−P33 |)
+ (| Q44−Q22 |)
+ (| G32−G43 | + | G23−G34 |) / 2
DDve = (| P31−P33 | + | P35−P33 |)
+ (| G32−G34 |)
+ (| Q22−Q24 | + | Q42−Q44 |) / 2
Next, returning to FIG. 3, in step S <b> 2, the directionality detection unit 62 detects the presence / absence of the edge of the target pixel and the directionality of the edge using the directionality data calculated in step S <b> 1.

   Below, each process sequence of the directionality detection process by the directionality detection part 62 of FIG. 2 is demonstrated using FIG.

  FIG. 6 is a flowchart for explaining an operation example of the directionality detection unit 62 in FIG. 2 (an example of step S2 in FIG. 3).

   As shown in FIG. 6, first, in step S21, the maximum value of the four types of direction data of FIG. 5 calculated in step S1 of FIG. 3 is detected by the direction detection unit 62.

   Next, in step S22, whether or not the maximum value of the directionality data detected in step S21 is smaller than the threshold value A stored in a register (not shown), that is, there is no edge near the target pixel and the luminance gradient is gentle. It is determined whether or not it is a flat portion. If this is a flat portion (Yes), a signal indicating “no directionality” is output in step S23, and the directionality detection processing of the pixel of interest ends. If it is not a flat part in step S22 (No), that is, if there is an edge part, it is determined that there is a direction, and the process proceeds to the next step S24. Thus, in step S22, the presence / absence of an edge is detected by detecting whether or not it is a flat portion. The range of values that can be taken by the threshold A in this case varies depending on the directionality data calculation formula, but as the threshold A is increased, the pixel data determined to be a flat portion increases, and as the threshold A is decreased, the flat portion is increased. Pixel data determined to be reduced.

   In step S24, the directionality with the smallest value is detected from the four types of directionality data calculated in step S1.

   In subsequent step S25, it is detected whether or not there are a plurality of directions having the direction data having the minimum value. If there are a plurality of direction data having the minimum value (Yes), the process of step S23 is performed. In step S23, a signal indicating “no directionality” is output, and the directionality detection processing of the target pixel ends. If there is not a plurality of directional data having a minimum value (No), the process proceeds to step S26.

   In step S26, the minimum directionality detected in step S24 is output, and the directionality detection processing of the target pixel ends.

   Next, returning to FIG. 3, in step S <b> 3, the directional color signal calculation unit 63 calculates a color signal corresponding to the edge direction (edge directionality) detected by the directionality detection unit 62. Since the directional color signal calculation processing by the directional color signal calculation unit 63 is a characteristic configuration of the first embodiment, it will be described in detail later.

   Next, in step S4, the target pixel color signal calculation unit 64 calculates the color signal calculated by the directional color signal calculation unit 63 in step S3 as a color signal in the target pixel. For the pixels detected as “no directionality” by the directionality detection unit 62 in step S2, a color signal is calculated using the bilinear method.

  Through steps S5, the processes in steps S1 to S4 are repeated until the interpolation process for all pixels is completed.

  Hereinafter, the color signal calculation method by the directional color signal calculation unit 63 (step S3 in FIG. 3) will be described in detail.

   There are many methods for calculating a color signal for each edge direction. Here, a color signal calculation formula will be described for the vertical direction and the upper right diagonal direction. The horizontal color signal calculation formula can be obtained by rotating the vertical color signal calculation formula by 90 degrees. Similarly, the color signal calculation formula in the lower right direction can be obtained by rotating the color signal calculation formula in the upper right direction by 90 degrees.

The following (Formula 3) is an example of a color signal calculation formula used by the directional color signal calculation unit 63.
(Formula 3)
When the target pixel is G33 (see FIG. 12A)
Gve = G33
Pve = (P32 + P34) / 2
Qve = G33 + (Q23−G23 ′ + Q43−G43 ′) / 2
However, G23 ′ = (G22 + G24) / 2
G43 ′ = (G42 + G44) / 2
Gru = (2 × G33 + G42 + G24) / 4
Pru = Gru + (P32−G32 ′ + P34−G34 ′) / 2
Qru = Gru + (Q23−G23 ′ + Q43−G43 ′) / 2
And

When the target pixel is P33 (see FIG. 12B)
Gve = (G32 + G34) / 2
Pve = P33
Qve = Gve + (Q23′−G23 + Q43′−G43) / 2
However, G32 ′ = (G22 + G42) / 2
G34 '= (G24 + G44) / 2
G23 ′ = (G22 + G24) / 2
G43 ′ = (G42 + G44) / 2
And

However,
Q23 '= (Q22 + Q24) / 2
Q43 '= (Q42 + Q44) / 2
Pru = P33
Qru = (Q42 + Q24) / 2
Gru = S33 + (Ga-Sa + Gb-Sb) / 2
And

However,
S33 = (2 × P33 + Q42 + Q24) / 4
Sa = (P33 + Q22) / 2
Ga = (G32 + G23) / 2
Sb = (P33 + Q44) / 2
Gb = (G34 + G43) / 2
Gve: G component color signal in the vertical direction Pve: P component color signal in the vertical direction Qve: Q component color signal in the vertical direction Gru: G component color signal in the upper right direction Pru: P component color signal in the upper right direction Qru : Q component color signal in the upper right direction.

   Hereinafter, how to derive the above (formula 3) will be described.

   First, a method for deriving a color signal in the vertical direction when the pixel of interest is the G component (when the detection edge direction is the vertical direction), a method of deriving a color signal in the vertical direction when the pixel of interest is the P component, The method of deriving the color signal in the upper right direction when the target pixel is the G component (when the detection edge direction is the upper right direction), and finally, the color in the upper right direction when the target pixel is the P component. The method of deriving signals will be described sequentially.

  First, how to derive the vertical color signal calculation formula (Formula 3) when the pixel of interest V33 is the G component will be described with reference to FIGS. 7 and 12A showing the pixel positions.

   In the above (Formula 3), the G component, R component, and B component of the color signal in the vertical direction are represented by Gve, Pve, and Qve. However, Pve is R component or B component, and Qve is B component or R component.

   The value G33 is used as the G signal Gve for the pixel of interest V33.

   As the P signal Pve for the target pixel V33, an average value of pixel components adjacent to one pixel in the vertical direction (V32 and V34) is used.

   Further, the Q signal Qve of the pixel of interest V33 is calculated using the correlation related to the color signal difference in the vertical direction.

Here, the correlation relating to the difference between the color signals is a property that “the change in the color signals is small in the local region and the difference between the different color components is substantially constant”. In other words, the Q component Q ′ and the G component G ′ of the neighboring pixels in consideration of the directionality are calculated, and the following (formula 4) is used to calculate the attention from the relationship between the Q component Q and the G component G of the target pixel. The Q component Q of the pixel can be derived. The same applies to the P component.
(Formula 4)
Q = G + (Q′−G ′)
P = G + (P′−G ′)
Here, paying attention to the pixel V23, the pixel V43, and the pixel V33, the difference between the G component G23 ′ and the Q component Q23 in the pixel V23, the difference between the G component G43 ′ and the Q component Q43 in the pixel V43, and the difference in the pixel V33 It is assumed that the difference between the G component G33 and the Q component Q33 is substantially equal. The average of the difference between the Q component and the G component in the pixel V23 and the difference between the Q component and the G component in the pixel V43 is defined as (Q′−G ′) in (Equation 4), and the Q component Q33 in the target pixel V33. Is calculated.

  Next, how to derive the vertical color signal calculation formula (Formula 3) when the pixel of interest V33 is the P component will be described with reference to FIGS. 7 and 12B showing the pixel positions.

   As the G signal Gve of the target pixel V33, an average value of pixel components adjacent to one pixel (V32 and V34) in the vertical direction is used.

   The value P33 is used as the P signal Pve for the pixel of interest V33.

   Further, the Q signal Qve of the target pixel V33 is calculated using the correlation regarding the color signal difference in the vertical direction as in the case where the target pixel is the G component.

   Here, paying attention to the pixel V23, the pixel V43, and the pixel V33, the difference between the G component G23 and the Q component Q23 ′ in the pixel V23, the difference between the G component G43 and the Q component Q43 ′ in the pixel V43, and the difference in the pixel V33 It is assumed that the difference between the G component Gve and the Q component Q33 is almost equal. The average of the difference between the Q component and the G component in the pixel V23 and the difference between the Q component and the G component in the pixel V43 is defined as (Q′−G ′) in (Equation 4), and the Q component Q33 in the target pixel V33. Is calculated.

  Next, a method of deriving the color signal calculation formula (formula 3) in the upper right direction when the target pixel V33 is the G component will be described with reference to FIGS. 7 and 12A showing the pixel positions.

   In the above (Expression 3), the G component, the R component, and the B component of the color signal in the diagonally upper right direction are represented by Gru, Pru, and Qru. However, Pru is an R component or a B component, and Qru is a B component or an R component.

   The G component Gru of the target pixel V33 is calculated from the pixel V33, the pixel V42, and the pixel V24.

   In addition, the P component Pru of the target pixel V33 and the Q component Qru of the target pixel V33 are calculated using the correlation regarding the color signal difference in the upper right direction.

    First, a method for calculating a color signal using color correlation for the P component of the target pixel V33 will be described.

   Here, paying attention to the pixel V32, the pixel V34, and the pixel V33, the difference between the P component P32 and the G component G32 ′ in the pixel V32, the difference between the P component P34 and the G component G34 ′ in the pixel V34, and the difference in the pixel V33 Assume that the difference between the P component P33 and the G component G33 is substantially equal.

   One of the main characteristic configurations of the present invention is how to obtain the G components G32 'and G34'. In the conventional method, when trying to obtain the G component in the upper right direction of the pixel V32, the G component is calculated from the upper right pixel including the pixel V32. However, since only the P component and the Q component exist in the upper right direction including the pixel V32, the G component cannot be obtained.

   On the other hand, the first embodiment is characterized by referring to pixels other than the pixels in the diagonally upper right direction in order to obtain the value of the G component G32 'of the pixel V32.

   For example, the average value of the G component G22 and the G component G42 is used as the value of the G component G32 'in the pixel V32. Similarly, the G component, and the average value of G24 and G component G44 are used as the value of the G component G34 'in the pixel V34.

   Using the G component thus calculated, the difference between the P component and the G component in the pixel V32 and the difference between the P component and the G component in the pixel V34 are the same as in the case of the color correlation in the vertical direction. The P component P33 in the pixel of interest V33 is calculated using (P′−G ′) in (Equation 4) as an average value.

   Next, a method for calculating a color signal using color correlation for the Q component of the target pixel V33 will be described. Basically, the color signal is calculated in the same procedure as the P component.

   Here, paying attention to the pixel V23, the pixel V43, and the pixel V33, the difference between the Q component Q23 and the G component G23 ′ in the pixel V23, the difference between the Q component Q43 and the G component G43 ′ in the pixel V43, and the difference in the pixel V33 Assume that the difference between the Q component Q33 and the G component G33 is substantially equal.

   For example, the average value of the difference between the Q component and the G component in the pixel V23 and the difference between the Q component and the G component in the pixel V43 is defined as (Q′−G ′) in (Equation 4), and A Q component Q33 is calculated.

   Finally, how to derive the color signal calculation formula (formula 3) in the upper right direction when the target pixel V33 is the P component will be described with reference to FIGS. 7 and 12B showing the pixel positions.

   The value P33 is used as the P component Pru of the target pixel V33.

   Further, as the Q component Qru of the target pixel V33, an average value of pixel values adjacent to one pixel (V42 and V24) in the diagonally upper right direction is used.

   Further, the G component Gru of the target pixel V33 is calculated by using the correlation regarding the color signal difference in the upper right direction.

   In FIG. 7, a point (position) where four pixels of the pixel V22, the pixel V32, the pixel V23, and the pixel V33 intersect is defined as a point A. In addition, a point (position) where four pixels of the pixel V33, the pixel V43, the pixel V34, and the pixel V44 intersect is a point B.

   Here, attention is paid to the positions of point A and point B in addition to the pixel V33. The difference between the average value S33 and the G component G33 of the P component and the Q component in the pixel V33, the difference between the average value Sa and the G component Ga of the P component and the Q component at the point A, and the P component and the Q at the point B It is assumed that the difference between the component average Sb and the G component Gb is substantially equal.

Here, the following (Formula 5) is derived from the correlation related to the color signal difference in the local region, as in the above (Formula 4).
(Formula 5)
G = S + (G′−S ′)
G: G component of the pixel of interest S: Average value of the P component and Q component of the pixel of interest G ′: G component at the neighborhood point of the pixel of interest S ′: Average value of the P component and Q component at the neighborhood point of the pixel of interest
In G ′ and S ′, the neighboring points of the target pixel are set to the same position.

   As a result, the average value of the difference between the S value and the G component at the point A and the difference between the S value and the G component at the point B is set as (G′−S ′) in the above (Expression 5). A G component G33 is calculated.

   As described above, according to the first embodiment, it is possible to perform the interpolation process using the characteristics in the local region of the image and also using the correlation of the color signal with the edge in the oblique direction. Since the color signal can be calculated using the color correlation in the diagonal direction, the color signal can be detected more accurately than the conventional interpolation processing method using the color correlation in the vertical and horizontal directions. Can be calculated.

For example, in the Bayer array having the luminance value shown in FIG. 13A, the directional data calculation formula (Formula 2) according to the first embodiment is used for the target pixel G33.
DDru = 248
DDho = 744
DDrd = 744
DDve = 434
Thus, the directionality detection unit 62 detects the direction “DDru; diagonally right upward” in which the value of the directionality data is the minimum value as the edge direction in the vicinity of the target pixel.

When the color signal calculation formula (Formula 3) according to the first embodiment is used,
Gru = (2 × G33 + G24 + G42) / 4 = 64
Rru = (R32 + R34) / 2 + (2 × G33−G22−G44) / 4 = 64
Bru = (B23 + B43) / 2 + (2 × G33−G22−G44) / 4 = 64
And
R = G = B
Therefore, a color signal without a false color signal can be calculated.

   One of the characteristic configurations of the present invention uses the correlation of color signals at neighboring points other than neighboring pixels by utilizing the fact that “the color component of the pixel of interest is correlated with the color components of neighboring points”. In the point.

   In the conventional method, the color signal at the center of the pixel is calculated, and the color signal of the target pixel is calculated based on the correlation with the color component of the target pixel using the pixel value and the calculated color signal.

   In the present invention, the color component at an arbitrary point (position) near the target pixel is calculated, and the color signal is calculated based on the correlation with the color signal of the target pixel. In addition, various color signal calculation formulas can be derived.

   Hereinafter, examples of several color signal calculation formulas will be described.

First, with reference to FIG. 12A, the color signal calculation formulas in the upper right direction when the pixel of interest is the G component are shown in the following (formula 6) and (formula 7).
(Formula 6)
Gru = (2 × G33 + G42 + G24) / 4
Pru = Gru + (Pa-Ga + Pb-Gb) / 2
Qru = Gru + (Qa-Ga + Qb-Gb) / 2
However,
Ga = (G22 + G33) / 2
Pa = (3 × P32 + P14) / 4
Qa = (3 × Q23 + Q41) / 4
Gb = (G33 + G44) / 2
Pb = (3 × P34 + P52) / 4
Qb = (3 × Q43 + Q25) / 4
And

The above (Expression 6) is obtained by calculating the position of the intersection A of V22, V32, V23 and V33 shown in FIG. 7, the position of the intersection B of V33, V43, V34 and V44, and the G component, P component and Q at the target pixel. This is an equation for obtaining a color signal using a correlation regarding a difference in color signals.
(Formula 7)
Gru = (2 × G33 + G42 + G24) / 4
Pru = Gru + (Pa-Ga + Pb-Gb) / 2
Qru = Gru + (Qa-Ga + Qb-Gb) / 2
However,
Ga = (2 × G24 + G33 + G15) / 4
Pa = (P14 + P34) / 2
Qa = (Q23 + Q25) / 2
Gb = (2 × G42 + G33 + G51) / 4
Pb = (P32 + P52) / 2
Qb = (Q41 + Q43) / 2
And

   The above (Expression 7) is an expression in which a color signal is obtained by utilizing the correlation regarding the difference between the color signals for V24, V42 and the G component, P component, and Q component in the target pixel shown in FIG.

Next, with reference to FIG. 12B, the following equations (Equation 8) and (Equation 9) show the color signal calculation expressions in the diagonally upper right direction when the target pixel is the P component.
(Formula 8)
Pru = P33
Qru = (Q42 + Q24) / 2
Gru = P33 + (Ga−Pa + Gb−Pb) / 2
However,
Pa = (2 × P33 + P31 + P13) / 4
Ga = (G32 + G23) / 2
Pb = (2 × P33 + P35 + P53) / 4
Gb = (G34 + G43) / 2
And

The above (Equation 8) shows the difference in color signal between the position of the intersection A of V22, V32, V23 and V33, the position of the intersection B of V33, V43, V34 and V44, and the G component and P component at the target pixel. This is an equation for obtaining a color signal by utilizing the correlation of.
(Formula 9)
Pru = P33
Qru = (Q42 + Q24) / 2
Gru = S33 + (Ga-Sa + Gb-Sb) / 2
However,
S33 = (2 × P33 + Q42 + Q24) / 4
Sa = (2 × P33 + P31 + P13 + 2 × Q22 + Q24 + Q42) / 8
Ga = (G41 + 3 × G32 + 3 × G23 + G14) / 8
Sb = (2 × P33 + P35 + P53 + 2 × Q44 + Q24 + Q42) / 8
Gb = (G25 + 3 × G34 + 3 × G43 + G52) / 8
And

   The above (Expression 9) is the average value of the G component, the P component, and the Q component at the position of the intersection A of V22, V32, V23, and V33, the position of the intersection B of V33, V43, V34, and V44, and the target pixel. Is a formula for obtaining a color signal by utilizing the correlation with respect to the difference between the color signals.

In addition to the above (Expression 3) and (Expression 6) to (Expression 9), various color signal calculation expressions can be considered.
(Embodiment 2)
In the second embodiment, a case where the color ratio correlation is used is described, unlike the case where the process (step S3 in FIG. 3) by the directional color signal calculation unit 63 of the first embodiment uses the color difference correlation. That is, in the first embodiment, the color signal is calculated using the correlation related to the color signal difference, but in the second embodiment, the color signal is calculated using the correlation related to the ratio of the color signals.

  As shown in FIGS. 1 and 2, the image processing apparatus 10 </ b> A according to the second embodiment is a solid-state imaging device 1 that can image a subject, and A / D that performs A / D conversion on imaging data from the solid-state imaging device 1. A D conversion unit 2 and an image processing unit 3A as an image processing apparatus that performs various signal processing on the digital signal from the A / D conversion unit 2 are provided.

  The image processing unit 3A includes a white balance unit 4, a gamma correction unit 5, and a data interpolation unit 6A. The data interpolation unit 6A includes a directional data calculation unit 61, a directional detection unit 62, a directional color signal calculation unit 63A, a target pixel color signal calculation unit 64, and a control unit 65 that controls these. ing.

  The directional color signal calculation unit 63A includes a color difference calculation unit (not shown) that calculates a ratio between two different color signals in consideration of the directionality at an arbitrary point near the target pixel, and the color ratio calculation unit. A color signal calculation unit (not shown) that calculates a color signal by using the correlation of the calculated ratio of the color signal.

  Hereinafter, a color signal calculation method by the directional color signal calculation unit 63A (corresponding to step S3 in FIG. 3) according to the second embodiment will be described. The directionality data calculation unit 61 (step S1 in FIG. 3), the directionality detection unit 62 (step S2 in FIG. 3), and the target pixel color signal calculation unit 64 (step S4 in FIG. 3) are described in the first embodiment. Since the same processing as in the above case can be performed, the description thereof is omitted here.

Here, the correlation related to the ratio of the color signals is a property that “the change of the color signals is small in the local region and the ratio of the different color components is substantially constant”. That is, the Q component Q ′ and the G component G ′ of neighboring pixels are calculated in consideration of the directionality, and the following (formula 10) is used to calculate the attention from the relationship between the Q component Q and the G component G of the target pixel. The Q component Q of the pixel can be derived. The same applies to the P component.
(Formula 10)
Q = G × (Q ′ / G ′)
P = G × (P ′ / G ′)
There are many methods for calculating the color signal for each direction. Here, as in the case of the first embodiment, the color signal calculation formula will be described for the vertical direction and the diagonally upper right direction. The horizontal color signal calculation formula can be obtained by rotating the vertical color signal calculation formula by 90 degrees. Similarly, the color signal calculation formula in the lower right direction can be obtained by rotating the color signal calculation formula in the upper right direction by 90 degrees.

The following (Formula 11) is an example of a color signal calculation formula used by the directional color signal calculation unit 63A.
(Formula 11)
When the target pixel is G33 (see FIG. 12A)
Gve = G33
Pve = (P32 + P34) / 2
Qve = G33 × (Q23 / G23 ′ + Q43 / G43 ′) / 2
However,
G23 ′ = (G22 + G24) / 2
G43 ′ = (G42 + G44) / 2
Gru = (2 × G33 + G42 + G24) / 2
Pru = Gru × (P32 / G32 ′ + P34 / G34 ′) / 2
Qru = Gru × (Q23 / G23 ′ + Q43 / G43 ′) / 2
And

However,
G32 '= (G22 + G42) / 2
G34 '= (G24 + G44) / 2
G23 ′ = (G22 + G24) / 2
G43 ′ = (G42 + G44) / 2
And

When the target pixel is P33 (see FIG. 12B)
Gve = (G32 + G34) / 2
Pve = P33
Qve = Gve × (Q23 ′ / G23 + Q43 ′ / G43) / 2
However,
Q23 '= (Q22 + Q24) / 2
Q43 '= (Q42 + Q44) / 2
Pru = P33
Qru = (Q42 + Q24) / 2
Gru = S33 × (Ga / Sa + Gb / Sb) / 2
And

However,
S33 = (2 × P33 + Q42 + Q24) / 4
Sa = (P33 + Q22) / 2
Ga = (G32 + G23) / 2
Sb = (P33 + Q44) / 2
Gb = (G34 + G43) / 2
Gve: G component color signal in the vertical direction Pve: P component color signal in the vertical direction Qve: Q component color signal in the vertical direction Gru: G component color signal in the upper right direction Pru: P component color signal in the upper right direction Qru : Q component color signal in the upper right direction.

   In the following, how to derive the above (formula 11) will be described.

   First, a method for deriving a color signal in the vertical direction when the pixel of interest is the G component (when the detection edge direction is the vertical direction), a method of deriving a color signal in the vertical direction when the pixel of interest is the P component, The method of deriving the color signal in the upper right direction when the target pixel is the G component (when the detection edge direction is the upper right direction), and finally, the color in the upper right direction when the target pixel is the P component. A method for deriving signals will be described.

  First, how to derive the vertical color signal calculation formula (Formula 11) when the target pixel V33 is the G component will be described with reference to FIGS. 7 and 12A showing the pixel positions.

   In the above (Formula 11), the G component, R component, and B component of the color signal in the vertical direction are represented by Gve, Pve, and Qve. However, Pve is R component or B component, and Qve is B component or R component.

   The value G33 is used as the G signal Gve for the pixel of interest V33.

   As the P signal Pve for the target pixel V33, an average value of pixel components adjacent to one pixel in the vertical direction (V32 and V34) is used.

   Further, the Q signal Qve of the pixel of interest V33 is calculated using the correlation regarding the ratio of the color signals in the vertical direction.

   Here, paying attention to the pixel V23, the pixel V43, and the pixel V33, the ratio between the G component G23 ′ and the Q component Q23 in the pixel V23, the ratio between the G component G43 ′ and the Q component Q43 in the pixel V43, and the ratio in the pixel V33 Assume that the ratio of the G component G33 and the Q component Q33 is substantially equal. The average value of the ratio between the Q component and the G component in the pixel V23 and the ratio between the Q component and the G component in the pixel V43 is set to (Q ′ / G ′) in the above (Equation 10), and the Q component Q33 in the target pixel V33. Is calculated.

  Next, how to derive the vertical color signal calculation formula (formula 11) when the pixel of interest V33 is the P component will be described with reference to FIGS. 7 and 12B showing the pixel positions.

   As the G component Gve of the target pixel V33, an average value of pixel values adjacent to one pixel in the vertical direction (V32 and V34) is used.

   The value P33 is used as the P component Pve of the pixel of interest V33.

   Further, the Q component Qve of the target pixel V33 is calculated using the correlation regarding the ratio of the color signals in the vertical direction, as in the case where the target pixel is the G component.

   Here, paying attention to the pixel V23, the pixel V43, and the pixel V33, the ratio between the G component G23 and the Q component Q23 ′ in the pixel V23, the ratio between the G component G43 and the Q component Q43 ′ in the pixel V43, and the ratio in the pixel V33 Assume that the ratio between the G component Gve and the Q component Q33 is substantially equal. Assuming that the ratio of the Q component to the G component in the pixel V23 and the average ratio of the Q component to the G component in the pixel V43 are (Q ′ / G ′) in the above (Equation 10), the Q component Q33 in the target pixel V33 is Calculated.

  Next, a method of deriving the color signal calculation formula (Formula 11) in the upper right direction when the target pixel V33 is the G component will be described with reference to FIGS. 7 and 12A showing the position of the pixel.

   In the above (Formula 11), the G component, R component, and B component of the color signal when the detected edge direction is obliquely upward to the right are represented by Gru, Pru, and Qru. However, Pru is an R component or a B component, and Qru is a B component or an R component.

   The G component Gru of the target pixel V33 is obtained from the pixel V33, the pixel V42, and the pixel V24.

   In addition, the P component Pru of the target pixel V33 and the Q component Qru of the target pixel V33 are calculated using the correlation regarding the ratio of the color signals in the upper right diagonal direction.

   First, a method for calculating a color signal using color correlation for the P component of the target pixel V33 will be described.

   Here, paying attention to the pixel V32, the pixel V34, and the pixel V33, the ratio between the P component P32 and the G component G32 ′ in the pixel V32, the ratio between the P component P34 and the G component G34 ′ in the pixel V34, and the ratio in the pixel V33 Assume that the ratio of the P component P33 and the G component G33 is substantially equal.

   One of the main features of the present invention is how to obtain G32 'and G34'. In the conventional method, when the G component of the pixel V32 is to be obtained, the G component is calculated from the pixel in the upper right direction including the pixel V32. However, since only the P component and the Q component exist in the upper right direction including the pixel V32, the G component cannot be obtained.

   In contrast, the present invention is characterized by referring to pixels other than the pixels in the diagonally upper right direction in order to obtain the value of the G component G32 'of the pixel V32.

   For example, the average value of G22 and G42 is used as the value of the G component G32 'in the pixel V32. Similarly, the average value of G24 and G44 is used as the value of the G component G34 'in the pixel V34.

   Using the G component thus calculated, the ratio between the P component and the G component in the pixel V32 and the ratio between the P component and the G component in the pixel V34 are the same as in the case of the color correlation in the vertical direction. The P component P33 in the pixel of interest V33 is calculated with the average value of (P ′ / G ′) in (Equation 10) above.

   Next, a method for calculating a color signal using color correlation for the Q component of the target pixel V33 will be described. Basically, the color signal is calculated in the same procedure as in the case of the P component.

   Here, paying attention to the pixel V23, the pixel V43, and the pixel V33, the ratio between the Q component Q23 and the G component G23 ′ in the pixel V23, the ratio between the Q component Q43 and the G component G43 ′ in the pixel V43, and the ratio in the pixel V33 Assume that the ratio between the Q component Q33 and the G component G33 is substantially equal.

   For example, the average value of the ratio between the Q component and the G component in the pixel V23 and the ratio between the Q component and the G component in the pixel V43 is (Q ′ / G ′) in the above (Equation 10), and A Q component Q33 is calculated.

  Finally, a method of deriving the color signal calculation formula (Formula 11) in the upper right direction when the target pixel V33 is the P component will be described with reference to FIGS. 7 and 12B showing the pixel positions.

   The value P33 is used as the P component Pru of the target pixel V33.

   Further, as the Q component Qru of the target pixel V33, an average value of pixel values adjacent to one pixel (V42 and V24) in the diagonally upper right direction is used.

   Further, the G component Gru of the target pixel V33 is calculated using the correlation regarding the ratio of the color signals in the upper right direction.

   In FIG. 8, a point (position) where four pixels of the pixel V22, the pixel V32, the pixel V23, and the pixel V33 intersect is a point A. In addition, a point (position) where four pixels of the pixel V33, the pixel V43, the pixel V34, and the pixel V44 intersect is a point B.

   Here, attention is paid to point A and point B in addition to the pixel V33. The ratio of the average value S33 and the G component G33 of the P component and the Q component in the pixel V33, the ratio of the average value Sa and the G component Ga of the P component and the Q component at the point A, and the P component and the Q at the point B Assume that the ratio between the average Sb of the components and the G component Gb is substantially equal.

Here, the following (Formula 12) is derived in the same manner as the above (Formula 10) by the correlation regarding the ratio of the color signals in the local region.
(Formula 12)
G = S × (G ′ / S ′)
G: G component of the pixel of interest S: Average value of the P component and Q component of the pixel of interest G ′: G component at the neighborhood point of the pixel of interest S ′: Average value of the P component and Q component at the neighborhood point of the pixel of interest In G ′ and S ′, the neighboring points of the target pixel are set to the same position.

   As a result, the ratio of the S value and the G component at the point A and the average value of the ratio of the S value and the G component at the point B are set as (G ′ / S ′) in the above (Equation 12). A G component G33 is calculated.

   As described above, according to the second embodiment, it is possible to perform the interpolation process using the characteristics in the local region of the image and also using the correlation of the color signal with the edge in the oblique direction. Since the color signal can be calculated using the color correlation in the diagonal direction, the color signal can be detected more accurately than the conventional interpolation processing method using the color correlation in the vertical and horizontal directions. Can be calculated.

For example, in the Bayer array having the luminance value shown in FIG. 13A, the directional data calculation formula (Formula 2) according to the second embodiment is used for the target pixel G33.
DDru = 248
DDho = 744
DDrd = 744
DDve = 434
Thus, the directionality detection unit 62 detects the direction “DDru; diagonally right upward” in which the value of the directionality data is the minimum value as the edge direction in the vicinity of the target pixel.

When the color signal calculation formula (Formula 11) according to the second embodiment is used,
Gru = (2 × G33 + G24 + G42) / 2 = 128
Rru = Gru × [{(R32 / (G22 + G42) / 2} + {(R34 / (G24 + G44) / 2}]) = 128
Bru = Gru × [{(B23 / (G22 + G24) / 2} + {(B43 / (G42 + G44) / 2}]) = 128
And
R = G = B
Therefore, a color signal without a false color signal can be calculated.
(Embodiment 3)
In the third embodiment, the directionality data is calculated with respect to the target pixel in the eight directions of the upward direction, the oblique upper right direction, the right direction, the oblique lower right direction, the downward direction, the oblique lower left direction, the left direction, and the oblique upper left direction. A case will be described in which one or more directions are detected as the edge directionality candidates in the vicinity including the target pixel from the directional data for a plurality of directions in ascending order.

  As shown in FIGS. 1 and 2, the image processing apparatus 10 </ b> B according to the third embodiment is a solid-state imaging device 1 that can image a subject, and A / D that converts A / D conversion of imaging data from the solid-state imaging device 1. A D conversion unit 2 and an image processing unit 3B as an image processing apparatus that performs various signal processing on the digital signal from the A / D conversion unit 2 are provided.

  The image processing unit 3B includes a white balance unit 4, a gamma correction unit 5, and a data interpolation unit 6B. The data interpolation unit 6B includes a directional data calculation unit 61B, a directional detection unit 62B, a directional color signal calculation unit 63B, a target pixel color signal calculation unit 64B, and a control unit 65 that controls these. ing.

   The target pixel color signal calculation unit 64B includes an average color calculation unit (not shown) as an average color calculation unit that calculates the average color of each color signal corresponding to the plurality of directions detected by the directionality detection unit 62B. A multi-color correlated color signal calculating unit (multi-color correlated color signal calculating unit) that calculates a color signal using the correlation between the color calculated by the average color calculating unit (not shown) and the color of the pixel of interest (Not shown).

   FIG. 3 is also a flowchart for explaining an operation example of the data interpolation unit 6B of the third embodiment.

  Hereinafter, the processing procedure of the data interpolation process performed by the data interpolation unit 6B according to the third embodiment will be described in order according to each step. Each process of steps S1 to S4 is repeated as many times as the number of pixels in the input interpolation designated area in the Bayer data area as shown in FIGS.

   As shown in FIG. 3, first, in step S1, the directionality data calculation unit 61B calculates the directionality data of the target pixel for a plurality of predetermined edge directions (designated directions).

   In the directionality data calculation unit 61B, as shown in FIG. 8, as a plurality of edge directions, with the pixel of interest at the center, right upward direction, right direction, right downward direction, down direction, left downward direction, left direction , 8 directions of diagonally upward left and upward are used. A feature of the third embodiment is that edge data in consideration of an end point of an edge can be calculated by calculating directional data in eight directions around the target pixel.

In the directionality data calculation unit 61B, for example, the directionality data is calculated so that the value decreases as the degree of approximation between the designated direction and the actual edge direction in the vicinity of the target pixel increases. Various types of directional data calculation formulas are conceivable. For example, the following calculation formula (Formula 13) is used.
(Formula 13)
When the target pixel is G33 (see FIG. 12A)
DDru = (| G51−G33 | + | G42−G33 | + | P52−P34 | + | Q41−Q23 |)
DDri = (| G53−G33 | + | Q23−Q43 |)
+ (| G22−G42 | + | G24−G44 | + | P52−P32 | + | P54−P34 |) / 2
DDrd = (| G44−G33 | + | G55−G33 | + | P32−P54 | + | Q23−Q45 |)
DDdo = (| G35−G33 | + | P32−P34 |)
+ (| G22−G24 | + | G42−G44 | + | Q25−Q23 | + | Q45−Q43 |) / 2
DDld = (| G15−G33 | + | G24−G33 | + | P25−P43 | + | Q14−Q32 |)
DDle = (| G13−G33 | + | Q23−Q43 |)
+ (| G22−G42 | + | G24−G44 | + | P12−P32 | + | P14−P34 |) / 2
DDlu = (| G22−G33 | + | G11−G33 | + | P12−P34 | + | Q21−Q43 |)
DDup = (| G31−G33 | + | P32−P34 |)
+ (| G22-G24 | + | G42-G44 | + | Q21-Q23 | + | Q41-Q43 |) / 2
When the target pixel is P33 (see FIG. 12B)
DDru = (| P51−P33 | + | Q42−Q24 |)
+ (| G32−G23 | + | G41−G32 | + | G52−G43 | + | G43−G34 |) / 2
DDri = (| P53−P33 | + | G23−G43 |)
+ (| Q22-Q42 | + | G32-G52 | + | Q24-Q44 | + | G34-G54 |) / 2
DDrd = (| P55−P33 | + | Q44−Q22 |)
+ (| G32−G43 | + | G43−G54 | + | G23−G34 | + | G34−G45 |) / 2
DDdo = (| P35−P33 | + | G32−G34 |)
+ (| Q22-Q24 | + | G23-G25 | + | Q42-Q44 | + | G43-G45 |) / 2
DDld = (| P15−P33 | + | Q42−Q24 |)
+ (| G32−G23 | + | G43−G34 | + | G23−G14 | + | G34−G25 |) / 2
DDle = (| P13−P33 | + | G23−G43 |)
+ (| Q22-Q42 | + | Q24-Q44 | + | G12-G32 | + | G14-G34 |) / 2
DDlu = (| P11−P33 | + | Q22−Q44 |)
+ (| G21−G32 | + | G32−G43 | + | Q12−Q23 | + | G23−G34 |) / 2
DDup = (| P31−P33 | + | G32−G34 |)
+ (| Q22-Q24 | + | Q42-Q44 | + | G21-G23 | + | G41-G43 |) / 2
However,
DDru: Directional data calculation formula in the upper right direction DDri: Directional data calculation formula in the right direction DDrd: Directional data calculation formula in the lower right direction DDdo: Directional data calculation formula in the lower direction DDld: Lower left direction Directional data calculation formula DDle: Directional data calculation formula in the left direction DDlu: Directional data calculation formula in the diagonally upper left direction DDup: Directional data calculation formula in the upward direction

   Next, in step S2, the directionality detection unit 62B detects the directionality of the pixel of interest and the directionality using the directionality data calculated in step S1.

   FIG. 9 is a flowchart for explaining an operation example of the directionality detection unit 62B (another example of step S2) according to the third embodiment.

  Below, the process procedure of the directionality detection process by the directionality detection part 62B is demonstrated in order according to each step.

   As shown in FIG. 9, first, in step S21B, the maximum values of the eight types of direction data calculated in step S1 are detected.

   Next, in step S22B, whether or not the maximum value of the directionality data detected in step S21B is smaller than the threshold value B stored in a register (not shown), that is, there is no edge near the target pixel and the luminance gradient is gentle. It is determined whether or not it is a flat portion. If it is a flat part (Yes), a signal indicating “no directionality” is output in step S23B, and the directionality detection processing of the target pixel is ended. If it is not a flat portion (No), the process proceeds to step S24B.

   The range of values that can be taken by the threshold B varies depending on the directionality data calculation formula, but as the threshold B is increased, the number of pixels determined to be a flat portion increases, and as the threshold B is decreased, the number of pixels determined to be a flat portion decreases. To do.

    Next, in step S24B, one or more directivities are detected in ascending order of values from the eight types of directivity data calculated in step S1. One or more directions detected at this time are called direction candidates.

For example, if the following judgment formula (Formula 14) is satisfied, the directionality corresponding to the directionality data is detected.
(Formula 14)
DDmin ≧ threshold C × DDi
DDi: Directionality data of target pixel DDmin: Minimum value of directionality data Note that the range of values that can be taken by the threshold C is 0 ≦ threshold C ≦ 1, and the larger the threshold C, the more the number of detected orientations. As the number decreases, the number of detected directions increases.

   Finally, in step S26B, the directionality detected in step S24B is output, and the directionality detection processing for the target pixel ends.

   Next, in step S3 of FIG. 3, the directional color signal calculation unit 63B calculates a color signal for the direction detected by the direction detection unit 62B.

   In the third embodiment, color signals can be calculated for eight directions including an oblique direction by a color signal calculation formula using color correlation. In the third embodiment, as a color signal corresponding to eight directions, a color signal for eight neighboring pixels in contact with the target pixel is calculated instead of the color signal of the target pixel.

   For example, the color signal V42 is calculated as the directional color signal in the upper right direction of the target pixel V33 in FIG.

   Further, a color signal of V34 is calculated as, for example, a downward direction color signal of the target pixel V33 in FIG.

   When calculating the directional color signal, the color signal calculation formula using the correlation related to the color signal difference described in the first embodiment or the color signal ratio correlation described in the second embodiment is used. The color signal for each directional candidate can be calculated using the used color signal calculation formula.

The following (Expression 15) and (Expression 16) are examples of directional color signal calculation expressions for eight directions. (Expression 15) is a directional color signal calculation expression when the target pixel is G33 (see FIG. 12A), and (Expression 16) is when the target pixel is P33 (see FIG. 12B). This is a directional color signal calculation formula.
(Formula 15)
Gru = (G33 + 2 × G42 + G51) / 4
Pru = (P32 + P52) / 2 + (2 × G42−G31−G53) / 4
Qru = (Q41 + Q43) / 2 + (2 × G42−G31−G53) / 4
Gri = (G53 + G33) / 2
Pri = (P32 + P52 + P34 + P54) / 4
+ (G33 + G53-G42-G44) / 2
Qri = Q43
Grd = (G33 + 2 × G44 + G55) / 4
Prd = (P34 + P54) / 2 + (2 × G44−G35−G53) / 4
Qrd = (Q43 + Q45) / 2 + (2 × G44−G35−G53) / 4
Gdo = (G33 + G35) / 2
Pdo = P34
Qdo = (Q23 + Q25 + Q43 + Q45) / 4
+ (G33 + G35-G24-G44) / 2
Gld = (G33 + 2 × G24 + G15) / 4
Pld = (P14 + P34) / 2 + (2 × G24−G33−G15) / 4
Qld = (Q23 + Q25) / 2 + (2 × G24−G33−G15) / 4
Gle = (G33 + G13) / 2
Ple = (P12 + P32 + P14 + P34) / 4
+ (G33 + G13-G22-G24) / 2
Qle = Q23
Glu = (G33 + 2 × G22 + G11) / 4
Plu = (P12 + P32) / 2 + (2 × G22−G31−G13) / 4
Qlu = (Q21 + Q23) / 2 + (2 × G22−G31−G13) / 4
Gup = (G33 + G31) / 2
Pup = P32
Qup = (Q21 + Q23 + Q41 + Q43) / 4
+ (G33 + G31-G22-G42) / 2
Gru: G component in the upper right direction Pru: P component in the upper right direction Qru: Q component in the upper right direction Gri: G component in the right direction Pri: P component in the right direction Qri: Q component in the right direction Grd: G component in the lower right direction Prd: P component in the lower right direction Qrd: Q component in the lower right direction Gdo: G component in the lower direction Pdo: P component in the lower direction Qdo: Q component in the lower direction Gld: Oblique left G component in the downward direction Pld: P component in the diagonally lower left direction Qld: Q component in the diagonally lower left direction Gle: G component in the leftward direction Ple: P component in the leftward direction Qle: Q component in the leftward direction Glu: diagonally upward in the left direction Glu: P component in the diagonally upward left direction Qlu: Q component in the diagonally upward left direction Gup: G component in the upward direction Pup: P component in the upward direction Qup: Q component in the upward direction (Formula 16)
Gru = (G41 + G32 + G52 + G43) / 4
+ (P33 + P51-P31-P53) / 4
Pru = (P33 + P51) / 2
Qru = Q42
Gri = G43
Pri = (P33 + P53) / 2
Qri = (Q42 + Q44) / 2
+ (4 × G43−G32−G52−G34−G54) / 4
Grd = (G45 + G34 + G54 + G43) / 4
+ (P33 + P55-P35-P53) / 4
Prd = (P33 + P55) / 2
Qrd = Q44
Gdo = G34
Pdo = (P33 + P35) / 2
Qdo = (Q24 + Q44) / 2
+ (4 × G34-G23-G25-G43-G45) / 4
Gld = (G25 + G23 + G14 + G34) / 4
+ (P33 + P15-P13-P35) / 4
Pld = (P33 + P15) / 2
Qld = Q24
Gle = G23
Ple = (P13 + P33) / 2
Qle = (Q22 + Q24) / 2
+ (4 × G23−G12−G32−G14−G34) / 4
Glu = (G21 + G12 + G32 + G23) / 4
+ (P33 + P11-P31-P13) / 4
Plu = (P33 + P11) / 2
Qlu = Q22
Gup = G32
Pup = (P33 + P31) / 2
Qup = (Q22 + Q42) / 2
+ (4 × G32−G21−G23−G41−G43) / 4
Next, in step S4 of FIG. 3, the directional color signal calculation unit 63B (step S3) corresponding to each directionality candidate detected by the directionality detection unit 62B (step S2) by the target pixel color signal calculation unit 64B. ), The average (rr, gg, bb) of the color signals is calculated as the color signal of the target pixel.

  Through steps S5, the processes in steps S1 to S4 are repeated until the interpolation process for all pixels is completed.

   According to the third embodiment, by performing directionality detection in eight directions around the target pixel, interpolation processing with few false color signals can be performed in consideration of the end points of the edges.

   As another method of calculating the pixel-of-interest color signal, the color of the average color (rr, gg, bb) of the directional color signal corresponding to the direction candidate calculated as described above and the color component of the pixel of interest There is also a method for calculating the color signal of the pixel of interest using signal correlation.

For example, when the color component of the target pixel is the G component g, the color components Go, Ro, and Bo of the color signal of the target pixel can be calculated using the following (Equation 17).
(Formula 17)
Go = g
Ro = g + rr−gg
Bo = g + bb-gg
According to the method using (Equation 17), when calculating the directional color signal in each direction, the color signal is calculated using the correlation between the color signals, and the average color of the color signals is calculated. The correlation with the color component color of the pixel of interest is used. As described above, by calculating the color signal again using the correlation of the color signal with respect to the color signal obtained by the correlation of the color signal, the generation of the false color signal can be further suppressed.
(Embodiment 4)
In the fourth embodiment, the process (step S1 in FIG. 3) by the directionality data calculation unit 61C shown in FIG. 2 is a case different from the cases of the first to third embodiments, and for a plurality of directions near the target pixel. A case where the intensity of the correlation of the color signal is taken into account when quantifying the directionality of the edge will be described.

  As illustrated in FIGS. 1 and 2, the image processing apparatus 10 </ b> C according to the fourth embodiment includes a solid-state imaging device 1 that can image a subject, and A / D that performs A / D conversion on imaging data from the solid-state imaging device 1. A conversion unit 2 and an image processing unit 3C as an image processing apparatus that performs various signal processing on the digital signal from the A / D conversion unit 2 are provided.

  The image processing unit 3C includes a white balance unit 4, a gamma correction unit 5, and a data interpolation unit 6C. The data interpolation unit 6C includes a directional data calculation unit 61C, a directional detection unit 62, a directional color signal calculation unit 63, a target pixel color signal calculation unit 64, and a control unit 65 that controls these. ing.

  Hereinafter, a direction data calculation method by the direction color data calculation unit 61C (another example of step S1 in FIG. 3) according to the fourth embodiment will be described. Note that the directionality detection unit 62 (step S2 in FIG. 3), the directionality color signal calculation unit 63 (step S3 in FIG. 3), and the target pixel color signal calculation unit 64 (step S4 in FIG. 3) are described in the first embodiment. Since the same processing as in the case of ~ 3 can be performed, the description thereof is omitted here.

   In the first to third embodiments, for example, the directionality data is calculated such that the value decreases as the degree of approximation between the specified edge direction and the actual edge direction near the target pixel increases. On the other hand, in the fourth embodiment, a term representing the intensity of the correlation of the color signal is added to such a directional data calculation formula. For example, a term is added to the directional data calculation formula such that the value decreases when there is a color signal correlation at the edge portion and increases when there is no color signal correlation. That is, when the directionality data calculation unit 61C quantifies the directionality of the edge with respect to a plurality of directions in the vicinity of the target pixel including the target pixel, the correlation strength of the color signal that greatly changes depending on the presence or absence of the edge is obtained. The directionality of the edge is quantified in consideration.

When there is a correlation between the G component and the P component of the color signal at two points a and b in the vicinity of the target pixel,
Ga-Pa = Gb-Pb
It becomes. Therefore,
| Ga-Gb + Pb-Pa |
It can be said that the closer the value is to 0, the more there is a color correlation between the two points a and b.

For example, a correlation strength term such as the following formula (formula 18) is added to the directionality data calculation formula (formula 2) used in the first and second embodiments.
(Formula 18)
When the target pixel is G33 (see FIG. 12A)
DDru '= DDru'
+ (| (G22 + G42-G24-G44) / 2 + (P34-P32) |
+ | (G42 + G44-G22-G24) / 2 + (Q23-Q43) |) / 2 DDho '= DDho
+ | (G22 + G42-G24-G44) / 2 + (P34-P32) |
DDrd ′ = DDrd
+ (| (G22 + G42-G24-G44) / 2 + (P34-P32) |
+ | (G42 + G44-G22-G24) / 2 + (Q23-Q43) |) / 2 DDve '= DDve
+ | (G42 + G44-G22-G24) / 2 + (Q23-Q43) |
When the target pixel is P33 (see FIG. 12B)
DDru '= DDru'
+ | (G32 + G23-G43-G34) / 2 + (Q44-Q22) |
DDho '= DDho'
+ | (G32−G34) + (Q24 + Q44−Q22−Q42) / 2 |
DDrd ′ = DDrd
+ | (G32 + G23-G43-G34) / 2 + (Q24-Q22) |
DDve '= DDve
+ | (G23−G43) + (Q42 + Q44−Q22−Q24) / 2 |
DDru ': Directional data calculation formula in the upper right direction DDho': Directional data calculation formula in the horizontal direction DDrd ': Directional data calculation formula in the lower right direction DDve': Directional data calculation formula in the vertical direction A correlation strength term such as the following formula (formula 19) is added to the directional data calculation formula (formula 13) used in the third embodiment.
(Formula 19)
When the target pixel is G33 (see FIG. 12A)
DDru '= DDru'
+ (| (G31 + G51-G33-G53) / 2 + (Q43-Q41) |
+ | (G31 + G33−G51−G53) / 2 + (P52−P32) |) / 2 DDri ′ = DDri
+ (| (P32 + P52-P34-P54) / 2 + (G44-G42) |)
DDrd ′ = DDrd
+ (| (G33 + G35-G53-G55) / 2 + (P54-P34) |
+ | (G33 + G53-G35-G55) / 2 + (Q45-Q43) |) / 2 DDdo '= DDdo
+ (| (Q23 + Q25-Q43-Q45) / 2 + (G44-G24) |)
DDld '= DDld
+ (| (G13 + G33-G15-G35) / 2 + (Q25-Q23) |
+ | (G13 + G15−G33−G35) / 2 + (P34−P14) |) / 2 DDle ′ = DDle
+ (| (P12 + P32-P14-P34) / 2 + (G24-G22) |)
DDlu '= DDlu
+ (| (G11 + G31-G13-G33) / 2 + (Q23-Q21) |
+ | (G11 + G13−G31−G33) / 2 + (P32−P12) |) / 2 DDup ′ = DDup
+ (| (Q21 + Q23-Q41-Q43) / 2 + (G42-G22) |)
When the target pixel is P33 (see FIG. 12B)
DDru '= DDru'
+ (| (G41 + G32-G52-G43) + (P53-P31) |)
DDri '= DDri
+ (| (G32 + G52-G34-G54) / 2 + (Q44-Q42) |)
DDrd ′ = DDrd
+ (| (G43 + G54-G34-G45) + (P35-P53) |)
DDdo '= DDdo
+ (| (G23 + G25-G43-G45) / 2 + (Q44-Q24) |)
DDld '= DDld
+ (| (G23 + G14-G34-G25) + (P35-P13) |)
DDle '= DDle
+ (| (G12 + G32-G14-G34) / 2 + (Q24-Q22) |)
DDlu '= DDlu
+ (| (G21 + G32-G12-G23) + (P13-P31) |)
DDup '= DDup
+ (| (G21 + G23−G41−G43) / 2 + (Q42−Q22) |)
DDru ′: Directional data calculation formula for the upper right direction DDri ′: Directional data calculation formula for the right direction DDrd ′: Directional data calculation formula for the lower right direction DDdo ′: Directional data calculation formula for the lower direction DDld ′ : Directional data calculation formula for diagonally downward left direction DDle ': Directional data calculation formula for horizontal direction DDlu': Directional data calculation formula for diagonally upward left direction DDup ': Directional data calculation formula for upward direction In the directionality data calculation unit 61C, the directionality data is calculated by using (Equation 18) instead of the above (Equation 2) as an equation for calculating the directionality data. As described above, the processing by the directionality detection unit 62, the directional color signal calculation unit 63, and the target pixel color signal calculation unit 64 is performed.

   Alternatively, in the directionality data calculation unit 61C shown in FIG. 2, the directionality data is calculated using (Equation 19) instead of the above (Equation 3) as the directionality data calculation formula, and using the directionality data, As described in the third embodiment, processing by the directionality detection unit 62, the directional color signal calculation unit 63, and the target pixel color signal calculation unit 64 is performed.

As described above, in the fourth embodiment, the color signal calculation is performed in consideration of the correlation of the color signals by using the characteristics of the color signals in the local region of a general image. However, in general, there is not necessarily a correlation between color signals in the entire image. The fourth embodiment is particularly suitable for performing interpolation using the color signal correlation on the edge portion of the image where the color signal correlation is strong.
(Embodiment 5)
In the fifth embodiment, a personal computer (hereinafter referred to as a personal computer), an input / output device, an image processing program of the present invention, a parameter file storing various data such as threshold values, and a computer readable storing Bayer data are readable. A case will be described in which any one of the first to fourth embodiments is performed using a data processing apparatus including a readable recording medium (storage medium).

  FIG. 10 is a block diagram illustrating a configuration example of the data processing apparatus according to the fifth embodiment.

   As shown in FIG. 10, the data processing device 7 of the fifth embodiment includes a personal computer main body 8 and an input / output device 9.

   The personal computer main body 8 includes a CPU 81 as a control unit for controlling each unit, a memory 82 such as a hard disk, and an input / output interface 83, and the input / output device 9 includes a readable recording medium 91 such as an optical disk. ing. The memory 82 is also a computer-readable readable recording medium in which an image processing program is stored.

  The CPU 81 controls the input / output device 9 via the input / output interface 83, and the image processing program, parameter file, and Bayer data read from the readable recording medium 91 in the input / output device 9 are input via the input / output interface 83. And stored in the memory 82.

  Further, the CPU 81 reads the image processing program, the parameter file, and the Bayer data from the memory 82 in the personal computer main body 8, and according to each command of the image processing program, any one of the first to fourth embodiments described above is performed on the input Bayer data. After image processing including interpolation processing is performed, the input / output device 9 is controlled from the input / output interface 83 of the personal computer body 8, and the image data after image processing is output to the readable recording medium 91 via the input / output interface 83. Is done.

  Therefore, according to the fifth embodiment, the image processing including the interpolation processing according to any of the first to fourth embodiments can be performed on the personal computer. That is, although not shown here, only Bayer data is output from the imaging device, and image processing by any of the image processing units 3 and 3A to 3C is performed on the personal computer main body 8 side, whereby high-speed shooting is performed on the imaging device side. It can be carried out. Further, the personal computer main body 8 side can perform image processing a plurality of times by changing the threshold value, and a desired high-quality image can be easily obtained.

  As described above, according to the first to fourth embodiments, the data interpolation unit quantifies the directionality of edges with respect to a plurality of directions including at least one of the upper right diagonal direction and the lower right diagonal direction near the target pixel. A directionality data calculation unit that calculates data, a directionality detection unit that detects the directionality of an edge near the target pixel using the directionality data for a plurality of directions calculated by the directionality data calculation unit 61, and A directional color signal calculation unit that calculates a color signal corresponding to the directionality detected by the directionality detection unit, and the color of the target pixel using one or more color signals calculated by the directionality color signal calculation unit A pixel-of-interest color signal calculation unit that calculates a signal. This makes it possible to further suppress the generation of false color signals by utilizing the correlation of the color signals with respect to edges in a plurality of directions including an oblique direction.

  In addition, as mentioned above, although this invention has been illustrated using preferable Embodiment 1-5 of this invention, this invention should not be limited and limited to this Embodiment 1-5. It is understood that the scope of the present invention should be construed only by the claims. It is understood that those skilled in the art can implement an equivalent range based on the description of the present invention and the common general technical knowledge from the description of specific preferred embodiments 1 to 5 of the present invention. Patents, patent applications, and documents cited herein should be incorporated by reference in their entirety, as if the contents themselves were specifically described herein. Understood.

  According to the present invention, from a color signal output from a solid-state imaging device in which a plurality of types of color filters are discretely arranged in each pixel unit, all color signals in each pixel are obtained, and color signals of a target pixel and its neighboring pixels are obtained. Image processing apparatus generated by interpolation processing used, single-plate digital camera using the same, photographing apparatus such as video camera and camera-equipped mobile phone apparatus, image processing method using the image processing apparatus, and execution of this on a computer In the field of an image processing program for image processing and a computer-readable readable recording medium storing the image processing program, a color signal output from a solid-state imaging device in which a plurality of types of color filters are discretely arranged in each pixel block For example, all RGB color signals in each pixel are interpolated using the color signals of the target pixel and its neighboring pixels. The color signal calculation that further suppresses the generation of false color signals by using the feature in the local area of the image and performing the interpolation process that also uses the correlation of the color signal with the diagonal edge It can be performed. In addition, by performing color signal calculation by detecting directionality in eight directions with the pixel of interest at the center, interpolation processing with fewer false color signals can be performed. Furthermore, the false color signal can be further reduced by calculating the color signal using the color correlation using the color signal calculated using the characteristics of the color correlation. According to the image processing device, the imaging device, the image processing method, the image processing program, and the readable recording medium of the present invention, it is possible to detect a color signal in which an edge included in a subject is detected with high accuracy and generation of a false color signal is further suppressed. Since it can be calculated, a high-quality image can be obtained.

It is a block diagram which shows the structural example of the image processing apparatus of Embodiment 1-4 of this invention. It is a block diagram which shows the detailed structural example of the data interpolation part of FIG. 3 is a flowchart for explaining an example of an interpolation processing operation by a data interpolation unit in FIG. 2. It is a top view for demonstrating the interpolation designation | designated area | region in the input Bayer data area. It is a schematic diagram for demonstrating four types of designated directions used by Embodiment 1, 2 of this invention. 5 is a flowchart for explaining an example (first and second embodiments) of a directionality detection processing operation by a directionality detection unit in FIG. 2. It is a top view which shows the pixel position of a Bayer arrangement in case the color component of an attention pixel is G, R, or B. FIG. It is a schematic diagram for demonstrating eight types of designated directions used in Embodiment 3 of this invention. 10 is a flowchart for explaining another example (third embodiment) of the directionality detection processing operation by the directionality detection unit of FIG. It is a block diagram which shows the structural example of the data processor of Embodiment 5 of this invention. It is a top view which shows an example of the color filter array of a Bayer arrangement. (A) is a top view which shows a Bayer arrangement | sequence when the color component of an attention pixel is G, (b) is a top view which shows a Bayer arrangement when the color component of an attention pixel is R or B. (A) is a top view which shows an example of the color signal value (luminance data) of a solid-state image sensor, (b) is a top view which shows the example of arrangement | positioning of the color filter of a Bayer array in each pixel.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Solid-state image sensor 2 A / D (analog / digital) conversion part 3, 3A-3C Image processing part (image processing apparatus)
4 White balance unit 5 Gamma correction unit 6, 6A to 6C Data interpolation unit 61, 61B, 61C Directional data calculation unit 62, 62B Directional detection unit 63, 63A, 63B Directional color signal calculation unit 64, 64B Target pixel color Signal calculation unit 65 Control unit 7 Data processing device 8 Personal computer body 81 CPU
82 Memory 83 Input / output interface 9 Input / output device 91 Readable recording medium 10, 10A to 10C Imaging device

Claims (42)

Data interpolation using color signals from a solid-state image sensor in which multiple types of color filters are discretely arranged in each pixel unit, using all the color signals in each pixel using the color signals of the pixel of interest and its neighboring pixels In an image processing apparatus having a data interpolation unit generated by processing,
The data interpolation unit
Directionality data calculating means for calculating directionality data quantifying the directionality of edges with respect to a plurality of directions including an oblique direction in the vicinity of the target pixel;
Directionality detection means for detecting the directionality of an edge in the vicinity of the target pixel using the directionality data for the plurality of directions calculated by the directionality data calculation means;
Directional color signal calculating means for calculating a color signal corresponding to the directionality of one or a plurality of edges detected by the directionality detecting means;
A pixel-of-interest color signal calculation unit that calculates a color signal of the pixel of interest using one or more color signals calculated by the directional color signal calculation unit;
In the data interpolation process, the directional color signal calculation means includes a pixel adjacent to the pixel in the diagonal direction and a pixel further adjacent to the pixel when performing data interpolation in the diagonal direction of the Bayer array. Pay attention to a pixel, calculate a color signal at an intersection between 2 × 2 four pixels that are adjacent to each other and are not related to the center of the pixel and belong to the pixel in the target range, and correlate the color with the calculated color signal An image processing apparatus that performs an interpolation process that also uses the correlation of the color signal to the edge in the diagonal direction by calculating the color signal of the pixel of interest by. Data interpolation using color signals from a solid-state image sensor in which multiple types of color filters are discretely arranged in each pixel unit, using all the color signals in each pixel using the color signals of the pixel of interest and its neighboring pixels In an image processing apparatus having a data interpolation unit generated by processing,
The data interpolation unit
Directionality data calculating means for calculating directionality data quantifying the directionality of edges with respect to a plurality of directions including an oblique direction in the vicinity of the target pixel;
Directionality detection means for detecting the directionality of an edge in the vicinity of the target pixel using the directionality data for the plurality of directions calculated by the directionality data calculation means;
Directional color signal calculating means for calculating a color signal corresponding to the directionality of one or a plurality of edges detected by the directionality detecting means;
A pixel-of-interest color signal calculation unit that calculates a color signal of the pixel of interest using one or more color signals calculated by the directional color signal calculation unit;
In the data interpolation process, the directional color signal calculation means pays attention to the 2 × 2 4 pixels including the target pixel and one pixel in the diagonal direction when performing data interpolation in the diagonal direction of the Bayer array. In addition to calculating the color signal at the intersection of the four pixels without being bound by the center of the pixel, pay attention to the 2 × 2 four pixels including the target pixel and the other pixel in the oblique direction, and not depending on the center of the pixel By calculating the color signal at the intersection of four pixels and calculating the color signal of the pixel of interest using the color correlation with each calculated color signal, the correlation of the color signal with respect to the diagonal edge is also used. Image processing apparatus for performing the interpolation processing.   The image processing apparatus according to claim 1, wherein the plurality of directions including the oblique direction are a plurality of directions including at least one of an upper right oblique direction and an oblique lower right direction in the vicinity of the target pixel.   The directionality data calculation unit calculates the directionality data with respect to four directions including a diagonally right upward direction, a horizontal direction, a diagonally right downward direction, and a vertical direction passing through the target pixel. Image processing apparatus.   The directionality data calculating means has the directionality with respect to eight directions of an upward direction, an oblique upper right direction, a right direction, an oblique lower right direction, a downward direction, an oblique lower left direction, a left direction, and an oblique upper left direction with the target pixel as a center. The image processing apparatus according to claim 1, which calculates data.   When the directionality data calculating means quantifies the directionality of the edge with respect to a plurality of directions near the target pixel, the term indicating the intensity of the correlation of the color signal that changes depending on the presence or absence of the edge is the directionality data. The image processing apparatus according to claim 1, wherein the image processing apparatus is added to the calculation formula.   7. A term is added to the directional data calculation formula such that a value decreases when there is a color signal correlation at an edge portion, and a value increases when there is no color signal correlation at the edge portion. An image processing apparatus according to 1.   The directionality detection means determines whether or not the flat portion has a smooth luminance gradient without an edge near the target pixel depending on whether or not the maximum value of the directionality data for the plurality of directions is smaller than a predetermined threshold value. The image processing apparatus according to claim 1, wherein the image processing apparatus determines and detects “no directionality” in the case of the flat portion, and detects “directionality” in the case of not being the flat portion.   The image processing apparatus according to claim 8, wherein the directionality detection unit detects a direction in which the value of the directionality data is a minimum value as an edge direction in the vicinity of a target pixel when detecting the “with directionality”. . The directionality detecting means detects “no directionality” when there are a plurality of directions in which the value of the directionality data is a minimum value even when the “directionality” is detected. The image processing apparatus according to claim 8.   The image processing device according to claim 1, wherein the directionality detection unit detects one or more directions as directional candidates for edges in the vicinity of the target pixel in order of increasing value from each directionality data for a plurality of directions. The directional color signal calculation means includes
Color difference calculating means for calculating a difference between two different color signals in the direction of the edge at an arbitrary position near the target pixel;
The image processing apparatus according to claim 1, further comprising: a color signal calculation unit that calculates a color signal using a correlation between the color signal differences calculated by the color difference calculation unit. The directional color signal calculation means includes
Color ratio calculating means for calculating a ratio of two different color signals in the direction of the edge at an arbitrary position near the target pixel;
The image processing apparatus according to claim 1, further comprising: a color signal calculation unit that calculates a color signal by using the correlation of the ratio of the color signals calculated by the color ratio calculation unit.   The directional color signal calculation means may include, as an arbitrary position in the vicinity of the target pixel, a position of the target pixel, a center position of the neighboring pixel of the target pixel, and an intersection between adjacent pixels in the vicinity of the target pixel. The image processing apparatus according to claim 12, wherein any one of the above is used.   The directional color signal calculation means calculates from the neighboring pixels of the target pixel other than the detected direction when there is no pixel of the color component to be calculated with respect to the direction detected by the directionality detection means. The image processing apparatus according to claim 1, wherein a color signal corresponding to the directionality of the edge is calculated with reference to pixels of the color component to be calculated.   The image processing apparatus according to claim 15, wherein the directional color signal calculation unit uses an average value of the color components to be calculated in a plurality of neighboring pixels of the target pixel other than the detected direction.   The pixel-of-interest color signal calculation unit calculates the color signal calculated by the direction-color signal calculation unit as a color signal in the pixel of interest for the pixel detected as “directional” by the direction detection unit. The image processing apparatus according to claim 8 or 9.   The pixel-of-interest color signal calculation unit calculates a color signal of the pixel of interest using a bilinear method for a pixel detected as “no directionality” by the directionality detection unit. Image processing device.   3. The target pixel color signal calculating unit includes an average color calculating unit that calculates an average color of each color signal corresponding to a plurality of directions detected by the direction detecting unit as a color signal in the target pixel. An image processing apparatus according to 1.   The target pixel color signal calculation means includes an average color calculation means for calculating an average color of each color signal corresponding to a plurality of directions detected by the directionality detection means, and an average color calculated by the average color calculation means The image processing apparatus according to claim 1, further comprising: a multi-color correlated color signal calculating unit that calculates a color signal at the target pixel using a correlation between the color of the target pixel and the color of the target pixel.   2. A solid-state imaging device capable of imaging a subject, an analog / digital conversion unit for analog / digital conversion of imaging data from the solid-state imaging device, and a digital signal from the analog / digital conversion unit are signal-processed. An image capturing apparatus comprising the image processing apparatus according to claim 20. Data interpolation using color signals from a solid-state image sensor in which multiple types of color filters are discretely arranged in each pixel unit, using all the color signals in each pixel using the color signals of the pixel of interest and its neighboring pixels In an image processing method generated by processing,
A directionality data calculation step for calculating directionality data quantifying the directionality of the edge with respect to a plurality of directions including an oblique direction in the vicinity of the target pixel;
A directionality detecting step for detecting the directionality of the edge in the vicinity of the target pixel using the directionality data for the plurality of directions calculated in the directionality data calculating step;
A directional color signal calculation step for calculating a color signal corresponding to the directionality of one or a plurality of edges detected in the directionality detection step;
A target pixel color signal calculation step of calculating a color signal of the target pixel using one or more color signals calculated in the directional color signal calculation step;
In the data interpolation process, the directional color signal calculation step includes a pixel including a pixel adjacent to the pixel in the diagonal direction and a pixel further adjacent to the pixel when performing data interpolation in the diagonal direction of the Bayer array. Pay attention to a pixel, calculate a color signal at an intersection between 2 × 2 four pixels that are adjacent to each other and are not related to the center of the pixel and belong to the pixel in the target range, and correlate the color with the calculated color signal An image processing method for performing an interpolation process that also uses the correlation of the color signal with respect to the diagonal edge by calculating the color signal of the pixel of interest by Data interpolation using color signals from a solid-state image sensor in which multiple types of color filters are discretely arranged in each pixel unit, using all the color signals in each pixel using the color signals of the pixel of interest and its neighboring pixels In an image processing method generated by processing,
A directionality data calculation step for calculating directionality data quantifying the directionality of the edge with respect to a plurality of directions including an oblique direction in the vicinity of the target pixel;
A directionality detecting step for detecting the directionality of the edge in the vicinity of the target pixel using the directionality data for the plurality of directions calculated in the directionality data calculating step;
A directional color signal calculation step for calculating a color signal corresponding to the directionality of one or a plurality of edges detected in the directionality detection step;
A target pixel color signal calculation step of calculating a color signal of the target pixel using one or more color signals calculated in the directional color signal calculation step;
The directional color signal calculation step pays attention to 2 × 2 4 pixels including the target pixel and one pixel in the diagonal direction when performing data interpolation in the diagonal direction of the Bayer array in the data interpolation processing. In addition to calculating the color signal at the intersection of the four pixels without being bound by the center of the pixel, pay attention to the 2 × 2 four pixels including the target pixel and the other pixel in the oblique direction, and not depending on the center of the pixel By calculating the color signal at the intersection of four pixels and calculating the color signal of the pixel of interest using the color correlation with each calculated color signal, the correlation of the color signal with respect to the diagonal edge is also used. Image processing method for performing the interpolation processing.   The image processing according to claim 22 or 23, wherein the directionality data calculation step calculates the directionality data with respect to four directions including a right upper direction, a horizontal direction, a right lower direction, and a vertical direction passing through the target pixel. Method.   The directionality data calculating step includes the directionality with respect to eight directions of an upward direction, an oblique upper right direction, a right direction, an oblique lower right direction, a downward direction, an oblique lower left direction, a left direction, and an oblique upper left direction with the target pixel as a center. The image processing method according to claim 22 or 23, wherein data is calculated.   In the directionality data calculation step, when quantifying the directionality of the edge with respect to a plurality of directions in the vicinity of the target pixel, a term representing the intensity of the correlation of the color signal that changes depending on the presence or absence of the edge is the directionality data. The image processing method according to any one of claims 22 to 25, which is added to a calculation formula.   27. A term is added to the directional data calculation formula such that a value decreases when there is a color signal correlation at an edge portion, and a value increases when there is no color signal correlation at the edge portion. Image processing method.   In the direction detection step, whether or not the maximum value of the direction data for a plurality of directions is smaller than a predetermined threshold value determines whether or not the pixel is a flat portion where there is no edge near the target pixel and the luminance gradient is gentle. 24. The image processing method according to claim 22, wherein “no directionality” is detected in the case of the flat portion, and “directionality” is detected in the case of not being the flat portion.   29. The image processing method according to claim 28, wherein the direction detection step detects a direction in which a value of the direction data is a minimum value as an edge direction in the vicinity of a target pixel when the “with direction” is detected. .   The directionality detecting step detects “no directionality” when there are a plurality of directions in which the value of the directionality data has a minimum value even when the “directionality” is detected. The image processing method according to claim 28.   The image processing method according to claim 22 or 23, wherein the directionality detection step detects one or more directions as directionality candidates in the edge direction in the vicinity of the target pixel in order of increasing values from the directionality data for a plurality of directions. The directional color signal calculation step includes:
A color difference calculating step of calculating a difference between two different color signals in the direction of the edge at an arbitrary position in the vicinity of the target pixel;
24. The image processing method according to claim 22, further comprising: a color signal calculation step of calculating a color signal using the correlation of the color signal difference calculated in the color difference calculation step. The directional color signal calculation step includes:
A color ratio calculating step of calculating a ratio of two different color signals in the direction of the edge at an arbitrary position in the vicinity of the target pixel;
24. The image processing method according to claim 22, further comprising a color signal calculation step of calculating a color signal by using the correlation of the ratio of the color signals calculated in the color ratio calculation step.   In the directional color signal calculation step, as an arbitrary position in the vicinity of the target pixel, the position of the target pixel, the center position of the vicinity pixel of the target pixel, and an intersection between adjacent pixels in the vicinity of the target pixel The image processing method according to claim 32 or 33, wherein any one of the above is used.   In the directional color signal calculation step, when there is no pixel of the color component to be calculated with respect to the detection direction detected by the directional detection unit, a plurality of pixels of interest other than the detected direction are detected. 36. The image processing method according to claim 22, wherein a color signal corresponding to the directionality of the edge is calculated from neighboring pixels with reference to a pixel of the color component to be calculated.   36. The image processing method according to claim 35, wherein the directional color signal calculation step uses an average value of the color components to be calculated in a plurality of neighboring pixels of the target pixel other than the detected direction.   The pixel-of-interest color signal calculation step calculates a color signal calculated in the directionality detection step as a color signal in the pixel-of-interest for a pixel detected as “directional” in the directionality detection step. 30. The image processing method according to 28 or 29.   31. The image according to claim 28 or 30, wherein the pixel-of-interest color signal calculation step calculates a color signal of the pixel of interest using a bilinear method for a pixel detected as “no directionality” in the directionality detection step. Processing method.   24. The target pixel color signal calculation step includes an average color calculation step of calculating an average color of each color signal corresponding to a plurality of directions detected in the direction detection step as a color signal in the target pixel. An image processing method described in 1.   The pixel-of-interest color signal calculation step includes an average color calculation step for calculating an average color of each color signal corresponding to a plurality of directions detected in the directionality detection step, and an average color calculated in the average color calculation step. 24. The image processing method according to claim 22 or 23, further comprising a multiple color correlated color signal calculation step of calculating a color signal at the target pixel using a correlation with the color of the target pixel.   The image processing program for making a computer perform each process step of the image processing method in any one of Claims 22-40.   42. A computer-readable readable recording medium in which the image processing program according to claim 41 is stored.

Images