JP5151075B2 - Image processing apparatus, image processing method, imaging apparatus, and computer program - Google Patents

Description

  The present invention relates to an image processing device and an image processing method for processing an output signal of a solid-state imaging device having a filter (color filter) of a predetermined color coding, an imaging device using the image processing device or the image processing method, and a computer Regarding the program.

  Cameras have a long history as a means of recording visual information. Recently, in place of silver halide cameras that shoot using film and photosensitive plates, digital cameras that digitize images captured by solid-state imaging devices such as CCD (Charge Coupled Device) and CMOS (Complementary Mental-Oxide Semiconductor) are widely used. Is popular.

  An image sensor using a solid-state imaging device is configured by a mechanism in which each pixel (photodiode) arranged two-dimensionally converts light into electric charges using a photoelectric effect. Usually, a color filter composed of three colors of R (red), green (G), and blue (B) is provided on the light receiving surface to separate incident light into three primary colors that can be seen by human eyes. In each pixel, an RGB image signal that can be seen by human eyes can be obtained by reading the signal charge accumulated corresponding to the amount of incident light that has passed through the color filter.

  Further, the RGB image signal is color space converted into a YUV image signal and used for display output and image recording. YUV expresses a color with two chromaticities consisting of a luminance signal Y, a red color difference U and a blue color difference V, and YUV conversion is a human whose resolution for luminance is high but resolution for color is low. This makes it easy to perform data compression. For example, in the NTSC (National Television Standards Committee) standard, color space conversion from RGB to YUV is performed by the following three formulas.

Y = 0.6G + 0.3R + 0.1B (1)
Cr (R−Y) = R− (0.3R + 0.6G + 0.1B) (2)
Cb (B−Y) = B− (0.3R + 0.6G + 0.1B) (3)

  For example, R (red), G (green), and B (blue) primary color filters are arranged in the same spatial phase with respect to three solid-state imaging devices, thereby obtaining a high-resolution RGB signal and improving image quality. An realized three-plate imaging device is known. However, since the three-plate imaging device needs to use a prism that separates incident light into RGB colors using three solid-state imaging devices, it is difficult to reduce the size and cost of the device. .

  On the other hand, there is also a single-plate image pickup apparatus that is reduced in size and cost by appropriately arranging RGB color filters in units of pixels in one solid-state image pickup apparatus. In this single-plate imaging device, color coding is performed in which R, G, and B color filters are intermittently arranged in units of pixels. A typical example of the color coating is a Bayer array that has been widely used in the past (see FIG. 24).

  When color coding is performed, information on the G and B components is lost in the pixel in which the R color filter is arranged, and information on the R and B components is missing in the pixel in which the G color filter is arranged. In the pixel in which G is arranged, information on the components of G and R is lost. When color space conversion is performed in later signal processing, all RGB signals are required in each pixel. This is because if the luminance signal Y and the color difference signals Cr and Cb are generated from the RGB signals having spatially different phases using the above equations (1) to (2), a false color signal is generated. Therefore, it is necessary to perform color space conversion after restoring the color signal lost in each pixel by color filter array interpolation to create an RGB signal having the same spatial phase (see FIG. 25). Such an interpolation technique is called “demosaic”.

  Since the interpolation accuracy of the RGB signal depends on the color coding, the characteristics of the solid-state imaging device differ depending on the color coding. In the Bayer arrangement shown in FIG. 24, R color filters and G color filters are alternately arranged in odd rows, and G color filters and B color filters are alternately arranged in even rows. Since the Bayer arrangement employs a configuration in which many G color filters are arranged for each RB color filter, G can be interpolated with higher accuracy than RB.

  As is clear from the above equation (1), G is a main component in creating a luminance signal, and the resolution of the luminance signal greatly depends on the resolution of G. Although the resolution of the image signal is proportional to the pixel sampling rate 1 / fs (fs is the pixel sampling frequency), according to the Bayer array, by improving the accuracy of the G color filter array interpolation, the resolution of the RB The characteristic that the resolution of G is high can be obtained. Since humans have a visibility characteristic of high resolution with respect to luminance and low resolution with respect to color, the Bayer array can be said to be color coding that makes good use of human visibility characteristics.

  It is also important how the RGB signal having the same spatial phase can be optimally generated from the RGB signals having different spatial phases by the interpolation technique. If the interpolation processing of the image signal is not optimal, a high-resolution image cannot be obtained, or a false color signal is generated.

  For example, when performing interpolation processing of an RGB image signal color-coded with a Bayer array, eight pixels in the vicinity of the pixel to be interpolated, that is, the upper, lower, right, left, upper right, lower right, upper left, lower left of the target pixel. A change amount of a total of 8 pixels is calculated, a correlation value is calculated by weighting the calculated change amount, an interpolation coefficient is determined based on the calculated correlation value, and an interpolation coefficient is calculated for the interpolation data. Proposals have been made on image processing apparatuses that add after being multiplied (see, for example, Patent Document 1).

  According to this image processing apparatus, by performing an interpolation process based on the above four pixel information based on the interpolation coefficient, it is possible to correlate well even with an edge in a direction not orthogonal to the correlation value calculation direction. Therefore, it is possible to satisfactorily reproduce the oblique edge after the interpolation processing without blurring, and it is possible to satisfactorily interpolate a bent line or a corner portion bent at a right angle.

By the way, in the Bayer array, the limit resolution of the G resolution and the RB resolution determined from the color filter array is the same in the oblique 45 degree direction. Since G is a main component in creating a luminance signal and greatly depends on the luminance resolution, and RB greatly depends on the resolution of the color difference signal, increasing G resolution generates a high-resolution image. Become. In addition, the human eye can recognize up to a high frequency with respect to luminance, but it is difficult to recognize a high frequency with respect to color, so the Bayer array has a balance between color resolution and luminance resolution. It is thought that it does not match the visibility characteristics sufficiently.

  In view of this point, the present applicant arranges the G component, which is the main component in creating the luminance component, so as to surround each of the other RB components, and sets the number of RB pixels to 1 with respect to the Bayer array. A color coding method has been proposed in which the luminance resolution is approximately doubled by slightly increasing the G resolution by increasing G instead of / 2 (see, for example, Japanese Patent Application No. 2005-107037). about). Such color coding matches human visibility characteristics more than Bayer color coding, but requires more advanced interpolation processing to achieve higher resolution than Bayer color coding. .

Japanese Patent Laid-Open No. 11-177994

  An object of the present invention is to provide an excellent image processing device, image processing method, imaging device, and color luminance that can be obtained by performing color filter array interpolation on a color-coded RGB image with higher accuracy. To provide a computer program.

The present invention has been made in consideration of the above problems, and a first aspect of the present invention is an image processing apparatus that processes an image captured by an imaging unit having a color filter for color coding.
Means for inputting an image signal;
Correlation value calculating means for calculating at least a first correlation value and a second correlation value indicating a degree of correlation between pixels for the image signal;
Interpolation pixel specifying means for specifying other pixels to be used when interpolating a desired pixel based on the first correlation value and the second correlation value;
Interpolating means for interpolating the desired pixel from the pixels specified by the interpolation pixel specifying means;
An image processing apparatus comprising:

  The image processing apparatus according to the present invention becomes a main component when calculating a luminance component, for example, for a pixel array in which each pixel is arranged in a square lattice so as to be equally spaced in the horizontal direction and the vertical direction. Color filter array interpolation is performed on an image signal subjected to color coding using color filters arranged so that the color components surround each of the other color components. Alternatively, on the color filter for color coding, each pixel becomes a main component when calculating a luminance component with respect to an oblique pixel array in which each pixel is shifted by ½ of the pixel pitch for each row and each column. The color components are arranged so as to surround each of the other color components.

  Such color coding increases the theoretical limit resolution of G by increasing G instead of halving the number of RB pixels relative to the Bayer array. Accordingly, since the G resolution is improved to near the limit resolution by performing the interpolation processing with high accuracy of G, the luminance resolution can be improved by about twice, although the color resolution is sacrificed slightly.

  When the correlation value calculation means interpolates G in a pixel at a spatial position to be interpolated (that is, G that is the main component of the luminance signal is missing), the correlation value calculation means calculates G between pixels for the surrounding image signal surrounding the pixel. A correlation value indicating the strength of the correlation is calculated in at least two directions.

  The correlation value between pixels can be obtained by filtering the image signal in two orthogonal directions such as the horizontal direction and the vertical direction of the image, and calculating the correlation value in one direction characteristic from the filter output. The filter here is preferably a band-pass filter that removes a DC component, and a high-pass filter such as a differential filter can be used.

  When the band-pass filter output in the horizontal direction is BPF (H) and the band-pass filter output in the vertical direction is BPF (V), the correlation value S (HV) in the horizontal and vertical direction HV is BPF (H) / {BPF (H ) + BPF (V)}. S (HV) takes a value of 0 to 1, but when it approaches 1, it indicates that there is a wave of the image signal in the horizontal direction and the correlation between the pixels is high in the vertical direction. Of these, it is preferable to specify a pixel positioned in the vertical direction as a pixel for interpolation. On the other hand, when S (HV) approaches 0, there is a wave of the image signal in the vertical direction, indicating that the correlation between the pixels is high in the horizontal direction. Therefore, the pixel located in the horizontal direction of the desired pixel is used as a pixel for interpolation. It is preferable to specify.

  The correlation value calculation means can calculate the correlation value S (HV) in the horizontal and vertical directions as the first correlation value. However, when only the correlation value calculated with a single direction characteristic is used, when the correlation value S (HV) is a value near 0.5, the correlation between pixels is high in the direction of +45 degrees with respect to the vertical axis. Or whether the correlation between the pixels is high in the direction of -45 degrees is unclear, and the interpolation pixels cannot be specified.

  Therefore, the correlation value calculating means further obtains the second correlation value in the direction characteristic different from the first correlation value, and compensates for the case where the first correlation value is an intermediate value. Improved the resolution to specify the direction to interpolate pixels.

  The correlation value calculating means arranges another set of orthogonal filters having different directional characteristics from the set of orthogonal filters used when calculating the first correlation value, and based on these filter outputs. To calculate a second correlation value.

  When the correlation value S (HV) in the horizontal / vertical direction HV is calculated as the first correlation value as described above, the band-pass filter output BPF (NH) in the non-horizontal direction NH and the BPF ( The correlation value S (NH / NV) of the non-horizontal non-vertical direction NH / NV calculated based on the band pass filter output BPF (NH) of the non-horizontal NH that is orthogonal to HN) can be used. S (NH / NV) is represented by BPF (NH) / {BPF (NH) + BPF (NV)}.

  The second correlation value is obtained from the outputs of the bandpass filters arranged in the orthogonal NH and NV directions, with the directions rotated by ± 45 degrees with respect to the vertical direction H as non-horizontal directions NH and non-vertical directions NV, respectively. In this case, when one of the first correlation value and the second correlation value is a correlation value in the vicinity of 0.5, the other correlation value can be supplemented, and the resolution for specifying the interpolation direction is maximized. .

  That is, according to the image processing apparatus of the present invention, the interpolation pixel specifying unit determines the direction of correlation in all directions (360 degrees) by comparing a plurality of correlation values obtained by the correlation value calculating unit. The direction to be interpolated can be specified with a finer granularity. Then, the interpolation means may perform the interpolation process on the pixel to be interpolated based on the information on the peripheral pixels of the pixel existing in the specified direction.

According to a second aspect of the present invention, there is provided an image processing apparatus for processing an image captured by an imaging unit having a color filter for color coding.
Means for inputting an image signal;
Correlation value calculating means for calculating at least a first correlation value and a second correlation value indicating a degree of correlation between pixels for the image signal;
Reliability calculation means for calculating the reliability of each correlation value calculated by the correlation value calculation means;
An interpolation means for interpolating a desired pixel by an interpolation process according to the reliability;
An image processing apparatus comprising:

  According to the first aspect of the present invention, two or more correlation values having different directional characteristics are obtained for the surrounding pixels of the pixel to be interpolated, and the correlation direction is determined in all directions by comparing the plurality of correlation values. 360 degrees), the direction to be interpolated can be specified with a finer granularity. However, such a method for specifying the direction to be interpolated assumes that the correlation value obtained in each direction characteristic is reliable.

  Therefore, in the image processing apparatus according to the second aspect of the present invention, the reliability relating to the first correlation value and the second correlation value is calculated, and a desired pixel is calculated using an interpolation processing method according to the reliability. It was configured to interpolate the signal. For example, the reliability can be calculated based on the sum of the correlation values calculated by the correlation value calculation means.

  Here, when the calculated correlation value is reliable, the direction of the correlation is determined by comparing the obtained correlation values with the correlation line, as in the image processing apparatus according to the first aspect of the present invention. By determining over all directions (360 degrees), the direction to be interpolated is determined, and more accurate pixel interpolation is performed. As a result, a high-resolution luminance signal can be obtained based on a highly accurate G signal.

  On the other hand, when the calculated correlation value is unreliable, interpolation is performed using the average value of the information on the peripheral pixels of the pixel to be interpolated. In this case, the accuracy of the interpolated pixels can be suppressed and a luminance signal with high resolution cannot be obtained, but the SN can be improved by the averaging process.

According to a third aspect of the present invention, there is provided a computer program written in a computer-readable format so as to execute processing on an image picked up by an image pickup means having a color filter for color coding on the computer. Whereas
A correlation value calculation procedure for calculating at least a first correlation value and a second correlation value indicating a degree of correlation between pixels for an image signal by the imaging means;
An interpolation pixel specifying procedure for specifying another pixel to be used when interpolating a desired pixel based on the first correlation value and the second correlation value;
An interpolation procedure for interpolating the desired pixel from the pixels identified by performing the interpolation pixel identification procedure;
Is a computer program characterized in that

According to a fourth aspect of the present invention, there is provided a computer program written in a computer-readable format so as to execute processing on an image picked up by an image pickup means having a color filter for color coding on the computer. Whereas
A correlation value calculation procedure for calculating at least a first correlation value and a second correlation value indicating a degree of correlation between pixels for an image signal by the imaging means;
A reliability calculation procedure for calculating a reliability for each correlation value calculated by executing the correlation value calculation procedure;
An interpolation procedure for interpolating a desired pixel by interpolation processing according to the reliability,
Is a computer program characterized in that

  The computer program according to each of the third and fourth aspects of the present invention defines a computer program described in a computer-readable format so as to realize predetermined processing on the computer. In other words, by installing the computer program according to the third and fourth aspects of the present invention in a computer, a cooperative action is exhibited on the computer, and the first and second aspects of the present invention. The same operational effects as those of the image processing apparatus according to the above can be obtained.

  According to the present invention, an excellent image processing apparatus, image processing method, imaging apparatus, and color luminance that are obtained by performing color filter array interpolation on a color-coded RGB image with higher accuracy, and an imaging apparatus, A computer program can be provided.

  According to the present invention, in particular, in a pixel array in which each pixel is arranged in a square lattice so as to be equally spaced in the horizontal direction and the vertical direction, the color component that is the main component when calculating the luminance component is The main component color filter array interpolation for calculating the luminance component can be performed with high accuracy on the image signal subjected to color coding using the color filters arranged so as to surround each of the color components.

  Further, the image processing apparatus according to the present invention can determine the direction of correlation in all directions (360 degrees) with respect to the pixel to be interpolated, and therefore can perform appropriate interpolation processing based on the determined direction. .

  Other objects, features, and advantages of the present invention will become apparent from more detailed description based on embodiments of the present invention described later and the accompanying drawings.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows an example of a configuration of an imaging apparatus using an image processing apparatus or an image processing method according to the present invention.

  Camera module including an imaging apparatus, a solid-state imaging apparatus as an imaging device, an optical system that forms image light of a subject on an imaging surface (light-receiving surface) of the solid-state imaging apparatus, and a signal processing circuit of the solid-state imaging apparatus In addition, a digital still camera and a video camera mounted with the camera module, and an electronic device such as a mobile phone are included.

  In FIG. 1, image light from a subject (not shown) is imaged on an imaging surface of an imaging device 12 by an optical system, for example, an imaging lens 11. As the imaging device 12, a large number of pixels including photoelectric conversion elements are two-dimensionally arranged in a matrix, and a color filter including a color component as a main component and other color components for generating a luminance component is disposed on the surface of the pixel. A solid-state imaging device is used.

  As a solid-state imaging device having a color filter, a charge transfer type solid-state imaging device typified by a CCD (Charge Coupled Device) or an XY typified by a MOS (Metal Oxide Semiconductor: metal oxide semiconductor). Any of an address type solid-state imaging device and the like may be used.

  In the following, in the color filter, for example, green (G) is used as a color component that is a main component in creating a luminance (Y) component, and red (R) and blue (B) are used as other color components. Will be described. However, the gist of the present invention is not limited to the combination of these color components. For example, white, cyan, yellow, and the like are used as the main color components in creating the Y component. For example, magenta, cyan, yellow, etc. can be used.

  In the imaging device 12, only the light of each color component out of the incident image light passes through the color filter and enters each pixel. Light incident on each pixel is photoelectrically converted by a photoelectric conversion element such as a photodiode. Then, it is read out from each pixel as an analog image signal, converted into a digital image signal by the A / D converter 13, and then input to a camera signal processing circuit 14 which is an image processing apparatus according to the present invention.

  The camera signal processing circuit 14 includes an optical system correction circuit 21, a WB (white balance) circuit 22, an interpolation processing circuit 23, a gamma correction circuit 24, a Y (luminance) signal processing circuit 25, a C (chroma) signal processing circuit 26, a band A limiting LPF (low-pass filter) 27, a thinning processing circuit 28, and the like are included.

  The optical system correction circuit 21 corrects a digital clamp that matches a black level to a digital image signal input to the camera signal processing circuit 14, a defect correction that corrects a defect of the imaging device 12, and a peripheral light amount drop of the imaging lens 11. The imaging device 12 and the optical system are corrected, such as shading correction.

  The WB circuit 22 performs a process of adjusting white balance on the image signal that has passed through the optical system correction circuit 21 so that the white subject has the same RGB. The interpolation processing circuit 23 creates pixels having different spatial phases by interpolation, that is, creates three planes (RGB signals at the same spatial position) from RGB signals that are spatially out of phase. Specific interpolation processing in the interpolation processing circuit 23 is a feature of the present invention, and details thereof will be described later.

  The gamma correction circuit 24 applies gamma correction to the RGB signals at the same spatial position, and then supplies them to the Y signal processing circuit 25 and the C signal processing circuit 26. The gamma correction is an R output from the WB circuit 22 so that the photoelectric conversion characteristics of the entire system including the imaging device 12 and the video playback means in the subsequent stage are set to 1 in order to correctly represent the color gradation of the subject. , G and B color signals are each multiplied by a predetermined gain.

  The Y signal processing circuit 25 generates a luminance (Y) signal from the above equation (1), and the C signal processing circuit 26 calculates the color difference signals Cr (R−Y) and Cb (B) from the above equations (2) and (3). -Y).

  The band limiting LPF 27 is a filter whose cut-off frequency is 1/8 of the sampling frequency fs, and reduces the pass band from (1/2) fs to (1/8) fs for the color difference signals Cr and Cb. This is an output in accordance with the TV signal format, and when output without band limitation, a frequency signal of 1/8 fs or more is output as a false color signal. The thinning processing circuit 28 thins out the sampling of the color difference signals Cr and Cb.

  Here, the point is that the color (C) signal (color difference signals Cr and Cb) requires only a quarter band or less of the luminance (Y) signal. The reason is that, as a characteristic of the human eye, it is possible to recognize up to a high frequency with respect to luminance, but it is difficult to recognize a high frequency with respect to color (as described above). The format is also so arranged.

  Actually, when the difference between Y: Cr: Cb = 4: 4: 4 (the output bands of Y and C are the same) and Y: Cr: Cb = 4: 1: 1 is observed in the output image It is difficult to identify the difference between special subjects, for example, normal subjects other than red and blue point light sources. That is, as defined in the TV signal format, it can be seen that a sufficient resolution can be obtained when the C signal has a quarter band of the Y signal.

  Here, the interpolation processing in the interpolation processing circuit 23 which is a characteristic part of the present invention will be described.

  As described above, the interpolation processing generates a signal of a color component lacking in a pixel because the color filters are arranged intermittently by interpolation using signals of surrounding pixels, that is, pixels having different spatial phases. Color filter array interpolation, also called demosaic. This interpolation processing is very important for obtaining an image with high resolution. This is because if interpolation cannot be performed well by interpolation processing, a false signal is generated, resulting in a decrease in resolution and occurrence of color false. Further, if the G component, which is the main component of the luminance signal, cannot be interpolated with high accuracy, the resolution of the luminance signal is reduced.

  Conventionally, interpolation processing using correlation processing has been performed as interpolation processing for achieving high resolution. The “correlation process” here refers to a process of performing interpolation using pixel information in a direction in which the correlation is high for the pixel to be interpolated.

  For example, let us consider pixel interpolation processing using an input image made up of a resolution chart as shown in FIG. 3 as an example. The resolution chart is a chart in which the central portion is a low-frequency signal and becomes a high-frequency signal as the distance from the center increases. The resolution chart has various directions even with the signal of the same frequency. By inputting the resolution chart signal to the signal processing circuit, it is possible to analyze what processing is suitable for the various signals. . In the same figure, when the subject is a horizontal line such as point A on the vertical axis, the image signal is wavy in the vertical direction, but the correlation between the horizontal image signals is high. Interpolate using. In addition, when the subject is a vertical line such as point C on the horizontal axis, undulation is observed in the horizontal direction, while the correlation between the vertical image signals is high, so interpolation is performed using vertical pixels. To do. By performing interpolation processing using surrounding images in a direction with high correlation in this way, higher resolution can be realized.

  For example, as shown in FIG. 2, the interpolation processing using the correlation processing is performed in a color coding such as a Bayer arrangement in which G is arranged in a checkered pattern. This is effective for processing when generating the G signal of the position. Of course, not only the G signal but also the remaining R and B signals are usually subjected to interpolation processing. In this specification, the G signal interpolation processing for realizing a high resolution with respect to the luminance signal is described. However, the interpolation processing similar to G is performed using R, B, or other colors such as cyan and yellow. It is also possible to apply to.

  Interpolation processing is processing for generating a signal of a pixel to be interpolated by interpolation using signals of surrounding pixels having different spatial phases. Therefore, the procedure of the interpolation processing also depends on the layout of surrounding pixels, that is, color coding. First, the interpolation process for the Bayer array will be described. As can be seen from FIG. 2, G in the Bayer array is arranged in a checkered pattern, and interpolation processing is performed on a missing portion of the G signal (hereinafter also referred to as “interpolated pixel”).

By filtering using a band pass filter in the horizontal and vertical directions around the spatial position (interpolated pixel) X, how much amplitude of an image signal is present during observation from the horizontal and vertical directions (that is, horizontal) And the waviness of the image signal in each vertical direction). As shown in FIG. 4, the band-pass filter is a filter that has a peak at ¼ fs (where fs is a sampling frequency) and outputs a value for a signal up to the vicinity of the limit resolution of ½ fs. When the output of the bandpass filter is large, a signal having a large amplitude exists in that direction. Conversely, when the output of the bandpass filter is small, the signal fluctuation is small in that direction, that is, there is a low frequency signal. For example, a high-pass filter such as a differential filter can be used, but any filter having a band limiting characteristic that removes a DC component may be used.

  Here, assuming that the filter output of the bandpass filter arranged in the horizontal direction around the spatial position (interpolated pixel) X is Bpf_H, and the filter output of the bandpass filter arranged in the vertical direction is Bpf_V, these Each filter output is expressed as follows.

Bpf_H = − (G3 + G8) +2 (G4 + G9) − (G5 + G10)
Bpf_V = − (G1 + G2) +2 (G6 + G7) − (G11 + G12)

Let us observe the outputs Bpf_H and Bpf_V of these two bandpass filters at points A to C in FIG. At point A, there is no amplitude of the image signal in the horizontal direction, so Bpf_H = 0. At point B, there is an amplitude of the image signal in the oblique 45 degree direction, so Bpf_H = Bpf_V, and at point C, the vertical direction. Since there is no amplitude of the image signal, Bpf_V = 0.

  Subsequently, by using the outputs Bpf_H and Bpf_V of the band pass filter, the correlation values S_H and S_V in the horizontal and vertical directions are calculated from the following equations.

S_H = Bpf_V / (Bpf_H + Bpf_V)
S_V = 1−S_H

  These horizontal and vertical correlation values S_H and S_V represent the strength of correlation of image signals between adjacent pixels in the horizontal and vertical directions. Here, “correlation” refers to the fluctuation ratio of the signal. As shown in the above equation, the correlation values S_H and S_V are represented by the ratios of the filter outputs Bpf_H and Bpf_V of the bandpass filters positioned in the horizontal and vertical directions, respectively. The correlation is low when the signal fluctuation ratio is large, and the correlation is high when the signal fluctuation ratio is small.

  In the example shown in FIG. 3, at point A, S_H = 1 and S_V = 0, and it can be regarded as a linear component (no fluctuation or flat) in the horizontal direction. Since it is small, the correlation is high in the horizontal direction, and there is no correlation in the vertical direction. The interpolation accuracy is improved by interpolating with pixels in the direction where the correlation value is large, that is, the correlation is high. If accurate interpolation is realized for the G signal, the resolution of the luminance signal having G as a main component also becomes high.

  At point B, S_H = S_V = 0.5, indicating that the horizontal and vertical correlations are the same, that is, the same image changes in both horizontal and vertical directions. At point C, S_H = 0 and S_V = 1, indicating that the vertical correlation is high.

  Subsequently, the interpolation value X of the interpolation pixel X is calculated using the G signals of the upper and lower and left and right surrounding pixels according to the following equation.

    X = {(G6 + G7) × S_H + (G4 + G9) × S_V} / 2

  As can be seen from the above equation, interpolation processing is performed on the interpolation pixel X by applying a greater weight to the direction of high correlation. For example, at point A, X = (G6 + G7) / 2, and interpolation is performed using pixels in the horizontal direction. At point B, X = (G6 + G7 + G4 + G9) / 4, and interpolation is performed with equal weighting using horizontal and vertical pixels. At point C, X = (G4 + G9) / 2, and interpolation is performed using pixels in the vertical direction.

  Thus, high resolution can be realized by interpolating while weighting (weighting) the component of the pixel in the direction in which the change in the amplitude of the image signal is small, that is, the direction in which the correlation is high, using correlation processing. Please understand that you can.

  FIG. 5 shows the limit resolutions of the G resolution and the RB resolution in the Bayer array. As can be seen from the figure, the resolution of G is 1/2 fs in the horizontal and vertical directions, and (1 / 2√2) fs in the 45-degree direction. In particular, in the 45-degree direction, the G resolution and RB The resolution is the same.

  As can be seen from the above-described equations (1) to (3) for performing color space conversion, G is a main component in producing a luminance signal and greatly depends on the luminance resolution, and RB is largely dependent on the resolution of the color difference signal. Dependent. Therefore, increasing the resolution of G is a point for generating a high-resolution image. In addition, the human eye can recognize up to high frequencies in terms of brightness, but it is difficult to recognize high frequencies in terms of color, so the Bayer array has a balance between color resolution and luminance resolution. It does not match the visual sensitivity characteristics.

  In view of this point, in Japanese Patent Application No. 2005-107037 already assigned to the present applicant, the G component which is the main component in creating the luminance component is arranged so as to surround each of the RB components. Proposed color coding (described above). According to this color coding, by increasing G instead of halving the number of RB pixels with respect to the Bayer array, the color resolution can be slightly sacrificed, but the luminance resolution can be improved approximately twice. .

  The present inventors think that color coding in which the RB component is surrounded by the G component in this way matches human visual sensitivity characteristics more than Bayer color coding. In particular, in a signal processing system compliant with an output format of Y: Cr: Cb = 4: 1: 1, for example, a signal processing system such as a video camera, it can be said that color coding in the above band is more desirable. However, in order to obtain a luminance signal having G as a main component at a high resolution, a more advanced interpolation process is required than in the case of Bayer color coding.

  In the following, two color coding examples will be given as color coding in which, for example, G, which is a main component in creating a luminance component, is arranged so as to surround each of R, B, for example. Each of these two color coding examples Interpolation processing for will be described as Example 1 and Example 2.

[Example 1]
FIG. 6 shows an example of color coding to be subjected to interpolation processing in the first embodiment of the present invention. In the color coding example shown in the figure, the pixels are arranged in a square lattice so that the pixels are equally spaced (pixel pitch) d in the horizontal direction (row direction along the pixel row) and the vertical direction (column direction along the pixel column). For the arranged pixel arrangement, the first row is arranged by repeating RGBG in units of 4 pixels in the horizontal direction, only the G is arranged in the second row, and the fourth row is united by 4 pixels in the horizontal direction And the fourth row is arranged with only G, and thereafter, this four rows are repeated as a unit. As can be seen from the figure, the color component (G in this example) which is the main component in creating the luminance (Y) component and the other color components (R and B in this example) are G and R and B In addition, R and B are arranged at an interval of 4d with respect to the horizontal and vertical directions.

  When the sampling rate between pixels is d corresponding to the pixel pitch and the sampling rate is considered in the horizontal and vertical directions, the horizontal and vertical sampling rates of R and B are ½ of G. In this way, they are arranged every other column (odd column in the first embodiment) and every other row (odd row in the second embodiment described later). That is, the sampling rate of G is d, the sampling rate of R and B is 2d, and there is a difference in resolution between G and R and B in the horizontal and vertical directions twice. Further, when considering the sampling rate in a 45 ° oblique direction, the G sampling rate is d / 2√2, and the R and B sampling rates are 2d / √2.

  Here, let us consider the spatial frequency characteristics. Since the sampling rate of G is d in the horizontal and vertical directions, the G signal can be captured up to a frequency of (1/2) fs based on the sampling theorem. Further, since the G sampling rate is d / 2√2 in the 45 ° direction, the G signal can be captured up to (1 / √2) fs based on the sampling theorem.

  Similarly, let us consider the spatial frequency characteristics of R and B. However, since R and B have the same pixel array interval and can be considered similarly, only R will be described below. Since the sampling rate of R is 2d in the horizontal and vertical directions, it is possible to capture the R signal up to a frequency of 1/4 fs based on the sampling theorem. Further, in the oblique 45 degree direction, since the sampling rate of R is d / 2√2, a signal can be captured up to a frequency of (1 / 4√2) fs based on the sampling theorem.

  FIG. 7 shows the spatial frequency characteristics in the color coding example shown in FIG. G can capture a signal up to a frequency of (1/2) fs in the horizontal and vertical directions, and can capture a signal up to (1 / √2) fs in a 45-degree oblique direction. R and B can capture signals up to a frequency of (1/4) fs in the horizontal and vertical directions, and can capture signals up to a frequency of (1 / 4√2) fs in a 45-degree oblique direction. . That is, as apparent from the comparison between FIG. 5 and FIG. 7, the G-limited resolution is greatly improved with respect to the Bayer arrangement by using the color coding example shown in FIG. The resolution of the luminance signal is approximately doubled.

  Next, the correction process for the color coding example shown in FIG. 6 will be specifically described below. FIG. 8 shows a pixel array in which only G in the color coding example shown in FIG. 6 is extracted.

  As shown in the figure, there are G signals in a total of eight surrounding pixels in the horizontal, vertical, and diagonal directions surrounding the interpolation pixel X, that is, the pixels G4, G5, G6, G8, G9, G11, G12, and G13. Is used to interpolate the G signal of the interpolated pixel X. In the Bayer color coding, G pixels exist only in four horizontal and vertical directions. On the other hand, in the color coding example shown in FIG. 6, there are 8 pixels in the horizontal, vertical, and diagonal directions, and when the interpolation pixel X is observed, the G pixel is arranged like a grid. I understand that. This is very important for realizing high resolution.

  In the first embodiment, the interpolation processing circuit 23 obtains the correlation between the interpolated pixel X and the surrounding pixels not only in the horizontal and vertical directions but also in the diagonal direction, and based on the correlation between the horizontal and vertical directions and the diagonal direction. Then, interpolation processing is performed while judging which surrounding pixels are actually used for interpolation.

  FIG. 10 shows the procedure of the interpolation processing performed in the interpolation processing circuit 23 in the form of a flowchart. In the following, as shown in FIG. 9, the horizontal direction is H direction, the vertical direction is V direction, and the axial direction rotated right by 45 degrees with respect to the H direction is rotated left by 45 degrees with respect to the NH direction and H direction. The axial direction is described as the NV direction. Hereinafter, the interpolation process will be described in detail with reference to FIG.

  8 is set as an interpolation pixel (a pixel to be interpolated) (step S11). First, a correlation value in the HV direction is calculated for each surrounding pixel of the interpolation pixel X (step S12). Specifically, the correlation value in the HV direction is calculated by applying filtering by a bandpass filter around the pixel G4 on the upper left of the interpolation pixel X in the HV direction. FIG. 11 shows the frequency characteristics (filter characteristics) of the bandpass filter in the HV direction.

  Here, if the horizontal output of the bandpass filter is Bpf_H_G4 and the vertical output is Bpf_V_G4, a filtering result represented by the following equation is obtained.

Bpf_H_G4 = -G3 + 2G4-G5
Bpf_V_G4 = -G1 + 2G4-G8

  Subsequently, a correlation value S_H_G4 in the H direction with respect to the pixel G4 is calculated according to the following equation. The correlation value S_H_G4 in the H direction represents the ratio of the horizontal and vertical filter outputs Bpf_H and Bpf_V of bandpass filters having the same filter characteristics.

    S_H_G4 = Bpf_V / (Bpf_H + Bpf_V)

  The correlation value S_V_G4 in the V direction is S_V_G4 = 1−S_H_G4, and even if the correlation value S_H_G4 in the H direction is calculated, it is not necessary to calculate in particular. That is, at this time, the correlation values S_H_G4 and S_V_G4 in the HV direction centering on the pixel G4 are calculated. For example, when the correlation value S_H_G4 in the H direction is 1.0 and the correlation value S_V_G4 in the V direction is 0.0, a good interpolation result can be obtained by interpolating the pixel X with the correlation value S_V_G4 in the V direction.

  Similarly to the calculation of the correlation value S_H_G4 in the H direction with the pixel G4 at the upper left of the interpolation pixel X as the center, the pixel G6 at the upper right, the pixel G11 at the lower left, and the pixel G13 at the lower right are calculated. The correlation values S_H_G6, S_H_G11, and S_H_G13 in the respective H directions at the center are also calculated.

  Through the processing up to this point, that is, the processing in step S12, the correlation values S_H_G4, S_H_G6, S_H_G11, and S_H_G13 in the HV directions around the four pixels G4, G6, G11, and G13 surrounding the interpolation pixel X are calculated. (See FIG. 26).

  Subsequently, only two correlation values suitable for substitution for the interpolation pixel X are selected from the four calculated correlation values S_H_G4, S_H_G6, S_H_G11, and S_H_G13. Specifically, of the four correlation values, the two having the highest correlation, that is, the reliability of the calculated correlation value is selected as the correlation value of the interpolation pixel X (step S13).

  As can be seen from FIG. 26, each correlation value correlation value S_H_G4, S_H_G6, S_H_G11, S_H_G13 calculated in step S12 is a correlation value in the HV direction centering on the surrounding four pixels G4, G6, G11, and G13, respectively. Thus, it is not the correlation value S_H in the HV direction around the desired interpolation pixel X. When there is a mounting convenience such as a bandpass filter that cannot be formed around the interpolation pixel X, the correlation value at the interpolation pixel X cannot be directly calculated. Therefore, based on the idea that the correlation values in the adjacent pixels are substantially equal, the HV direction correlation value S_H of the interpolated pixel X is substituted by the highly reliable correlation value in the HV direction obtained from the surrounding pixels. This is the purpose of step S13.

  The reliability for selecting the correlation value can be expressed by using the output value of the bandpass filter calculated in the process of calculating the four correlation values. For example, the correlation reliability value Bpf_Max of the pixel G4 is calculated from the following equation.

    Bpf_Max = | Bpf_H_G4 | + | Bpf_V_G4 |

  The correlation reliability value Bpf_Max is similarly calculated for the other three surrounding pixels G6, G11, and G13. The large output of the bandpass filter means that a signal with a large amplitude exists around the pixel, and the signal is not noise but an image. Conversely, when the output of the bandpass filter is small, the signal is buried in noise and the reliability of the correlation is low, and it is difficult to trust the correlation value. That is, when the correlation reliability value Bpf_Max consisting of the sum of the absolute values of the output of the bandpass filter in two directions orthogonal to each other in the horizontal and vertical directions is large, the reliability of the correlation value calculated from the output of these bandpass filters is high. Can be estimated.

  Then, the correlation confidence values Bpf_Max obtained by the four surrounding pixels are compared in size, and the two surrounding pixels are selected from the larger one, and the correlation value in the HV direction calculated with these two surrounding pixels as the center is obtained. The value is selected as a substitute for the correlation value in the HV direction centered on the interpolation pixel X. FIG. 27 shows a state where S_H_G4 and S_H_G13 are selected as reliable correlation values.

  In calculating the correlation reliability, the correlation reliability value Bpf_Max, that is, the total value of the filter outputs, may be selected as a result of | Bpf_H−Bpf_V |, that is, a filter output having a large difference value. This is because the difference between the output Bpf_H in the H direction of the bandpass filter and the output Bpf_V in the V direction is large, that is, it has a strong correlation in the horizontal and vertical directions. There is a purpose to do.

  Subsequently, the correlation values of the adopted upper two places are averaged to obtain one correlation value (step S14). The averaged correlation value is treated as a correlation value in the HV direction centering on the interpolation pixel X in the following processing steps. For example, if the correlation values at two locations G4 and G6 are selected as having high reliability, the correlation values in the HV direction centering on these two points are averaged, and the correlation in the HV direction for the interpolation pixel X is calculated. It is regarded as a value (see FIG. 28).

  At this time, it is possible to select one horizontal and vertical correlation value from the four correlation values by selecting the correlation value of one surrounding pixel that maximizes the correlation confidence value Bpf_Max. By adopting the average value of the correlation values at the top two locations, good results are obtained. In addition, a method is also conceivable in which the top three correlation values are adopted from the four correlation values and the average value thereof is taken.

  When a band pass filter centered on the interpolation pixel X is assembled, the correlation value at the interpolation pixel X can be directly calculated instead of the average of the correlation values of the surrounding pixels with high reliability, and steps S12 to S14. The processing in is simplified.

  In parallel with the process of calculating the correlation value in the HV direction of the interpolation pixel X in steps S12 to S14, the process of calculating the correlation value in the NH and NV directions of the interpolation pixel X in steps S15 to S17 is executed. Here, the NH direction is an axial direction rotated right by 45 degrees with respect to the H direction, and the NV direction is an axial direction rotated left by 45 degrees with respect to the H direction (described above).

  First, a correlation value in the NH and NV directions is calculated for each surrounding pixel of the interpolation pixel X (step S15). Specifically, correlation values in the NH and NV directions are calculated by applying filtering by a bandpass filter in the oblique direction with the pixel G5 above the interpolation pixel X as the center. FIG. 12 shows frequency characteristics (filter characteristics) of band pulse filters in the NH and NV directions.

  Here, assuming that the output in the NH direction of the bandpass filter is Bpf_NH_G5 and the output in the NV direction is Bpf_NV_G5, a filtering result represented by the following expression is obtained.

Bpf_NH_G5 = −G1 + 2G5-G9
Bpf_NV_G5 = −G2 + 2G5-G8

  Subsequently, a correlation value S_NH_G5 in the NH direction for the pixel G5 is calculated from the following equation. The correlation value S_NH_G5 in the NH direction represents the ratio of the filter outputs Bpf_NH and Bpf_NV in the NH and NV directions of bandpass filters having the same filter characteristics.

    S_NH_G5 = Bpf_NV_G5 / (Bpf_NH_G5 + Bpf_NV_G5)

The correlation value S_NV_G5 in the NV direction is S_NV_G5 = 1−S_NH_G5, and even if the correlation value S_NH_G5 in the NH direction is easily calculated, it is not particularly necessary to calculate here. That is, at this time, the correlation values S_NH_G5 and S_NV_G5 in the NH and NV directions with the pixel G5 as the center are calculated. For example, when the NV direction correlation value S_ N V_G5 by the correlation value S_NH_G5 in the NH direction is 1.0 is 0.0, when interpolating the pixel X with the correlation value S_NV_G5 the NV direction, a good interpolation result.

  Similarly to the calculation of the correlation value S_NH_G5 in the NH direction centering on the pixel G5 above the interpolation pixel X, each NH centering on each of the left pixel G8, the right pixel G9, and the lower pixel G12. The direction correlation values S_NH_G8, S_NH_G9, and S_NH_G12 are similarly calculated.

  Through the processing up to this point, that is, the processing in step S15, the correlation value S_NH_G5 in the NH direction around each of the four pixels G5, G8, G9, and G12 located above, left, right, and bottom of the interpolation pixel X, S_NH_G8, S_NH_G9, and S_NH_G12 are respectively calculated (see FIG. 29).

  Subsequently, only two correlation values that substitute for the interpolation pixel X are selected from the four calculated correlation values S_NH_G5, S_NH_G8, S_NH_G9, and S_NH_G12. Specifically, two of the four correlation values having the highest reliability are employed as the correlation value of the interpolation pixel X (step S16).

As can be seen from FIG. 29, each of the correlation values correlation values S_NH_G5 calculated in step S15, S_ N H_G8, S_ N H_G9, S_ N H_G12 is, HV centered around four pixels G5, G8, G9, G12, respectively The correlation value of the direction, not the correlation value centered on the desired interpolation pixel X. When there is a mounting convenience such as a bandpass filter that cannot be formed around the interpolation pixel X, the correlation value at the interpolation pixel X cannot be directly calculated. Therefore, based on the idea that the correlation values in adjacent pixels are substantially equal, the correlation value in the NH direction of the interpolated pixel X is substituted with the highly reliable correlation value in the NH direction obtained from the surrounding pixels. This is the purpose of step S16.

  The reliability for selecting the correlation value is expressed by using the output value of the band pass filter calculated in the process of calculating the four correlation values. For example, the correlation reliability value Bpf_Max of the pixel G5 is calculated from the following equation. Similarly, the correlation reliability value Bpf_Max is calculated for the other three surrounding pixels G8, G9, and G12. When the correlation confidence value Bpf_Max consisting of the sum of the absolute values of the output of the bandpass filters in the two directions orthogonal to each other of NH and NV is large, it is estimated that the reliability of the correlation value calculated from the outputs of these bandpass filters is high. (Same as above)

    Bpf_Max = | Bpf_NH_G5 | + | Bpf_NV_G5 |

  Subsequently, the adopted correlation values at the top two places are averaged to obtain one correlation value (step S17). FIG. 30 shows a state where S_NH_G5 and S_NH_G9 are selected as reliable correlation values.

  The averaged correlation value is treated as a correlation value in the NH and NV directions centering on the interpolation pixel X in the following processing steps. For example, if the correlation values at two locations G5 and G9 are selected as having high reliability, the correlation values in the HV direction centering on these two points are averaged, and NH and It is regarded as a correlation value in the NV direction (see FIG. 31).

  When a band pass filter centered on the interpolation pixel X is assembled, the correlation value at the interpolation pixel X can be directly calculated instead of the average of the correlation values of surrounding pixels with high reliability, and steps S15 to S17. Is simplified (same as above).

  Through the processing so far, the correlation value S_H in the H direction and the correlation value S_NH in the NH direction for the interpolation pixel X are obtained (see FIGS. 28 and 31).

  Subsequently, based on the correlation values S_H and S_NH in the H direction and NH direction for the interpolation pixel X, which direction the surrounding pixel of the interpolation pixel X has a strong correlation, that is, the directionality of the correlation is obtained (step S18). ).

  Here, taking the input image composed of the resolution chart shown in FIG. 13 as an example, the correlation values S_H and S_NH calculated for the interpolation pixel X in the H direction and the NH direction, respectively, and the correlation between the interpolation pixel X and surrounding pixels surrounding it. Consider the degree. The resolution chart is a chart in which the central portion is a low-frequency signal and becomes a high-frequency signal as the distance from the center increases. The resolution chart has various directions even with the signal of the same frequency. By inputting the resolution chart signal to the signal processing circuit, it is possible to analyze what processing is suitable for the various signals. .

  For example, when the signals at points (a) to (e) in FIG. 13 are input as the interpolated pixel X and the above-described steps S12 to S17 are executed, the two correlation values S_H and S_NH are obtained at the point (a). S_H = 1.0 (S_V = 0.0), S_NH = 0.5 (S_NV = 0.5). From this, it is understood that the interpolation pixel X has a strong correlation with surrounding pixels in the H direction in the HV direction. It can also be seen that the NH and NV directions have the same correlation in the NH direction and the NV direction, that is, the interpolated pixel X has no correlation with surrounding pixels in the oblique direction.

  At point (b), S_H = 0.5 (S_V = 0.5) and S_NH = 1.0 (S_NV = 0.0). That is, the interpolation pixel X has no correlation with surrounding pixels in the HV direction, but has a strong correlation with surrounding pixels in the NH direction in the NH and NV directions.

  At point (c), S_H = 0.0 (S_V = 1.0) and S_NH = 0.5 (S_NV = 0.5). That is, the interpolation pixel X has a strong correlation outside the surrounding image in the V direction in the HV direction. It can also be seen that the interpolation pixel X has no correlation with surrounding pixels in the NH and NV directions.

  At point (d), S_H = 0.5 (S_V = 0.5) and S_NH = 0.0 (S_NV = 1.0). That is, the interpolation pixel X has no correlation with surrounding pixels in the HV direction, but has a strong correlation with surrounding pixels in the NV direction in the NH and NV directions.

  At point (e), S_H = 1.0 (S_V = 0.0) and S_NH = 0.5 (S_NV = 0.5). That is, as with point (a), the interpolation pixel X has a strong correlation with surrounding pixels in the H direction in the HV direction, but has no correlation with surrounding pixels in the NH and NV directions.

  Further, FIG. 14 shows a frequency representing the relationship between the spatial phase of the interpolation pixel and the correlation values of the surrounding pixels in the H and V directions and the NH and NV directions when the resolution chart shown in FIG. 13 is used as an input image. A chart is shown. In the following, the straight line in the figure showing the relationship of the correlation value between the interpolation pixel and the surrounding pixels in the HV direction and the NH and NV directions is also referred to as “correlation line”. This correlation line can be obtained by calculating at least two patterns of correlation values in different directions and plotting the correlation values of the two minimum patterns against straight lines of various angles. In the correlation diagram of FIG. 14, the one-dot chain line (A) corresponds to the correlation value S_H in the HV direction, and the two-dot chain line (B) corresponds to the correlation value S_NH in the NH and NV directions. The correlation lines (A) and (B) are out of phase by 45 degrees.

  When only the correlation value S_H in the HV direction is used, when S_H is close to 0 or 1, it can be specified that the interpolated pixel X has a strong correlation in the H direction or the V direction, but S_H is a value around 0.5. When the interpolation pixel X is taken, there is a possibility that the interpolation pixel X exists at the points (a) and (c) on the resolution chart, and the direction in which the interpolation pixel X has a strong correlation can be obtained only by referring to the correlation line (A). It cannot be specified. On the other hand, two orthogonal directions rotated by 45 degrees with respect to the horizontal and vertical axes, the NH direction rotated right by 45 degrees with respect to the H direction, and the NV direction rotated left by 45 degrees with respect to the H direction. By referring to the correlation line (B) in which the correlation value S_NH calculated from the output of the bandpass filter is plotted, even when S_H takes a value in the vicinity of 0.5, any of the interpolation pixels X is diagonal. Whether the direction has a strong correlation can be identified with high resolution. That is, by comparing the correlation lines of S_H and S_NH, it is possible to determine the direction in which the interpolation pixel X has a strong correlation with the surrounding pixels in all directions (360 degrees). In this embodiment, as an example, two orthogonal bandpass filters rotated by 45 degrees as a predetermined angle are used. However, an angle formed by the HV direction and the NH / NV direction such as 40 degrees to 50 degrees. Even if the angle is different, any effect can be used as long as it has the same effect as described above.

  When the resolution chart shown in FIG. 13 is used as an input image, in step S18, a one-dot chain line (A) and a two-dot chain line (B) in the spatial phase where the interpolation pixel X is located on the frequency chart shown in FIG. By comparing the correlation values indicated by (1), it is possible to obtain the degree of correlation between the interpolation pixel and the surrounding pixels in the HV direction and the NH and NV directions, that is, the directionality with a strong correlation.

  When the correlation direction of the interpolated pixel X is determined in step S18, it is subsequently determined whether or not the correlation values S_H and S_NH obtained in steps S12 to S14 and steps S15 to S17 are reliable (step S18). S19).

  As described above, in steps S12 to S14 and steps S15 to S17, the correlation value in each direction around the interpolation pixel X is not directly calculated, but is calculated around each surrounding pixel surrounding the interpolation pixel X. Of the correlation values in the H direction and the NH direction, those having high reliability are averaged and used as the correlation value of the interpolation pixel X. In such cases, the correlation reliability check is particularly important.

  Here, a specific method for checking the reliability of correlation will be described. The frequency chart when the resolution chart shown in FIG. 13 is used as an input image, that is, the relationship between the correlation values of the interpolation pixels and the surrounding pixels in the HV direction and the NH and NV directions is as shown in FIG. Two orthogonal band pass filters are arranged in the horizontal and vertical directions and in a direction rotated by 45 degrees with respect to the horizontal and vertical directions, respectively, and the correlation value S_H in the H direction and the NH direction from the output of each set of band pass filters The correlation value S_NH can be calculated. The correlation lines (A) and (B) of S_H and S_NH are out of phase by 45 degrees.

  Also, FIG. 32 shows a correlation line composed of an absolute value obtained by subtracting 0.5 from S_H (correlation line (A) in FIG. 14) and 0.5 from S_NH (correlation line (B) in FIG. 14). Each correlation line consisting of absolute values obtained by subtracting is plotted. As can be seen from the figure, if the correlation values S_H and S_NH are ideal ones on the correlation line shown in FIG. 14, that is, have high reliability, the absolute value obtained by subtracting 0.5 from S_H and S_NH The sum of absolute values obtained by subtracting 0.5 is a value close to 0.5. Therefore, the correlation values S_H and S_NH obtained in steps S12 to S14 and steps S15 to S17 are substituted into a conditional expression consisting of the following inequalities, and the reliability of the correlation values S_H and S_NH is determined depending on whether the condition is satisfied. be able to. However, TH1 and TH2 are values close to 0.5 that satisfy the relationship TH1 <TH2 (when TH1 and TH2 are close to 0.5, the determination condition becomes more severe).

    TH1 <| S_H-0.5 | + | S_NH-0.5 | <TH2

  If it is determined in step S19 that the correlation is reliable, the pixel that interpolates the interpolated pixel X is extracted from the surrounding pixels in the direction determined to have a strong correlation in step S18, and pixel interpolation is performed (step S19). S20).

  FIG. 15 shows the relationship between the correlation direction of the interpolation pixel X and the surrounding pixels used for interpolation. For example, if it is determined in step S18 that the correlation direction of the interpolation pixel X is at point (a) (that is, 90 degrees on the frequency chart), the correlation in the H direction is strong, so the interpolation pixel X Are interpolated by X = (G8 + G9) / 2. Further, when the directionality of the correlation of the interpolation pixel X is the point (b) (that is, 45 degrees on the frequency chart), since the correlation in the NH direction is strong, the G signal of the interpolation pixel X is expressed as X = (G4 + G13). ) / 2. The same applies when the direction of the correlation of the interpolation pixel X takes the points (c), (d), and (e).

  When the correlation direction of the interpolation pixel X is the point (f) where the correlation value between the points (c) and (b) is calculated, S_H≈0.25 and S_NH≈0. Since 75 is indicated, interpolation is performed using X = (G1 + G16) / 2 using pixels in that direction.

  Further, when there is no surrounding pixel that just rides on the straight line indicating the correlation direction of the interpolation pixel X, such as when the correlation direction of the interpolation pixel X is intermediate between the points (b) and (f), (B) Linear interpolation based on the relationship between the correlation value and the interpolation value, such as interpolation is performed by weighting the interpolation value at the point and the interpolation value at the point (f). For example, there is no surrounding pixel that just rides in the direction corresponding to point P in FIG. 15, but linear interpolation with two interpolation values (G6 + G11) / 2 and (G7 + G10) / 2 applied in the directionality adjacent to that direction. The correlation value of the interpolation pixel X can be calculated by the following equation. However, a is a weighting coefficient.

    X = {(G6 + G11) / 2} × a + {(G7 + G10) / 2} × (1-a)

  As described above, in the first embodiment, the directionality of the correlation of the interpolation pixel X is not obtained only from the correlation value S_H in the HV direction as in the prior art, but is orthogonally rotated by 45 degrees with respect to the horizontal and vertical axes. The correlation value S_NH calculated from the outputs of the two bandpass filters is also used. When the correlation value S_H in the HV direction is exactly 0.5, it becomes impossible to specify the directionality of the correlation only with S_H. On the other hand, since the correlation line of the correlation value S_NH in the NH and NV directions is out of phase by 45 degrees with respect to S_H (see FIG. 14), by further referring to S_NH, the interpolation pixel X It is possible to specify with high resolution which direction has a strong correlation. Therefore, when color coding is performed using the color filter shown in FIG. 6, the G signal can be interpolated with higher accuracy than the conventional Bayer arrangement, and as a result, a high-resolution luminance signal can be obtained. it can.

  On the other hand, when sufficient reliability is not obtained with respect to the correlation values S_H and S_NH obtained in steps S12 to S14 and steps S15 to S17 (No in step S19), the average value of the surrounding pixels is used for S. / N-oriented pixel interpolation is performed (step S21).

  If the two correlation values S_H and S_NH do not ride on the two correlation lines (A) and (B) shown in FIG. 14, it can be considered that there is no correlation. For example, when S_H = 1.0 and S_NH≈0.5, it can be said that there is a correlation. When S_H = 1.0 and S_NH≈1.0, any correlation line (A) and (B ), The reliability of the correlation is low. In the latter case, there is a low probability that correct interpolation is possible even if interpolation is performed using pixels in any direction, and there is a high possibility that a false color signal is generated by wrong interpolation.

  As described above, when the correlation values S_H and S_NH calculated in steps S12 to S17 do not ride on the two correlation lines (A) and (B) shown in FIG. 14, it is found that the reliability of the correlation value is low. In this case, the S / N-oriented interpolation processing is applied instead of performing resolution-oriented interpolation using surrounding pixels specified based on the directionality of correlation of the interpolation pixel X. For example, by interpolating the G signal of the interpolation pixel X using X = (G5 + G8 + G9 + G12) / 4 using four surrounding pixels, the performance of the imaging apparatus can be improved.

  In the Bayer array interpolation described above, only the correlation S_H in the HV direction can be obtained based on the outputs of a pair of orthogonal bandpass filters arranged in the HV direction. For this reason, the reliability of correlation cannot be evaluated. For example, the same interpolation as in the HV direction is performed for the NH and NV directions.

  On the other hand, in the first embodiment, color coding (see FIG. 6) is performed so that the RB component is surrounded by the G component (see FIG. 6). Therefore, the arrangement is performed in the NH and NV directions rotated by 45 degrees with respect to the HV direction. The correlation in the NH direction can be further calculated based on the outputs of the pair of orthogonal bandpass filters. Therefore, the direction in which the interpolation pixel X has a strong correlation with the surrounding pixels can be determined in all directions (360 degrees), and the interpolation processing can cope with the diagonal interpolation.

  Further, as apparent from the comparison of the limit resolutions of the image signals shown in FIGS. 5 and 7, according to the color coding in the first embodiment, the Bayer arrangement is used in the oblique 45 degree direction with respect to the resolution of G. In comparison, double the resolution can be obtained.

  Further, the reliability of the correlation is evaluated in step S19. When the correlation is reliable, interpolation with emphasis on resolution is performed from the direction of the correlation. When correlation is unreliable, interpolation with emphasis on S / N is performed. Since the processing method is adaptively switched according to the reliability, high-precision interpolation processing can be realized.

  For example, an artificial resolution chart as shown in FIG. 13 basically outputs a high correlation reliability in the entire region. On the other hand, in a subject in which a complicated curve such as a jari road in a general image or a bush of a tree is complicated, a situation in which the reliability of correlation is low may occur. In this case, it is only necessary to adaptively switch to interpolation processing that places importance on S / N.

  A dedicated hardware device that executes a series of interpolation processes according to the first embodiment can be designed and manufactured. For example, the same interpolation process is realized by a software process of executing a predetermined computer program on a computer. It is also possible.

  FIG. 16 shows a configuration example of an interpolation processing circuit 23A having a hardware configuration for executing the interpolation processing according to the first embodiment.

  The G4HV direction correlation value calculation circuit 31 calculates a correlation value in the HV direction by performing a filtering process on the HV direction centering on the pixel G4 on the upper left of the interpolation pixel X. For example, the G4HV direction correlation value calculation circuit 31 can be configured by a bandpass filter having the frequency characteristics shown in FIG. Specifically, the G4HV direction correlation value calculation circuit 31 calculates the correlation value S_H_G4 in the H direction for the pixel G4 from the following arithmetic expression.

  S_H_G4 = Bpf_V / (Bpf_H + Bpf_V)

  Then, the G4HV direction correlation value calculation circuit 31 further calculates the correlation value S_V_G4 in the V direction from the following arithmetic expression.

    S_V_G4 = 1−S_H_G4

  As for the other surrounding pixels G6, G11, and G13 surrounding the interpolation pixel X, the HV direction correlation value calculation circuits 32, 33, and 34 are diagonally right above the interpolation pixel X in the same manner as the G4HV direction correlation value calculation circuit 31. H-direction correlation values S_H_G6, S_H_G11, S_H_G13 and V-direction correlation values S_V_G6, S_V_G11, and S_V_G13 are calculated respectively for the pixel G6, the diagonally lower left pixel G11, and the diagonally lower right pixel G13. Each of the correlation value calculation circuits 31 to 35 realizes a process corresponding to step S12 of the flowchart shown in FIG.

  The selection circuit 35 selects a correlation value to be applied to the interpolation pixel X from the four for each of the H direction and the V direction. Specifically, the selection circuit 35 compares the output values of the bandpass filters calculated in the process of calculating the correlation values in the correlation value calculation circuits 31 to 34, respectively, and compares the correlation confidence values among the four correlation values. Two having the largest Bpf_Max, that is, the most reliable, are adopted as the correlation values of the interpolation pixel X. The selection circuit 35 realizes a process corresponding to step S13 in the flowchart shown in FIG.

  The average value calculation circuit 36 calculates the average value of the correlation values at the top two locations selected by the selection circuit 35 and outputs the correlation values as one correlation value S_H and S_V from each HV direction. The average value calculation circuit 36 realizes a process corresponding to step S14 in the flowchart shown in FIG.

  The G5NH and NV direction correlation value calculation circuit 37 calculates a correlation value in the NH direction and the NV direction by performing filtering processing on the orthogonal NH direction and NV direction around the pixel G5 above the interpolation pixel X. To do. For example, the G5NH and NV direction correlation value calculation circuit 37 can be configured by a bandpass filter having frequency characteristics shown in FIG. Specifically, the G5NH and NV direction correlation value calculation circuit 37 calculates a correlation value S_NH_G5 in the NH direction for the pixel G5 from the following arithmetic expression.

    S_NH_G5 = Bpf_NV_G5 / (Bpf_NH_G5 + Bpf_NV_G5)

  Then, the G5NH and NV direction correlation value calculation circuit 37 further calculates a correlation value S_NV_G5 in the NV direction from the following arithmetic expression.

    S_NV_G5 = 1-S_NH_G5

  Regarding the other surrounding pixels G8, G9, and G12 surrounding the interpolation pixel X, the NH and NV direction correlation value calculation circuits 38, 39, and 40 are similar to the G5NH and NV direction correlation value calculation circuit 37 in the same manner as the interpolation pixel X. NH-direction correlation values S_NH_G8, S_NH_G9, S_NH_G12 and NV-direction correlation values S_NV_G8, S_NV_G9, and S_NV_G12, which are centered on the left pixel G8, the right pixel G9, and the lower pixel G12, are calculated. The correlation value calculation circuits 37 to 40 realize processing corresponding to step S15 in the flowchart shown in FIG.

  The selection circuit 41 selects a correlation value to be applied to the interpolation pixel X from the four for each of the NH direction and the NV direction. Specifically, the selection circuit 41 compares the output values of the bandpass filters calculated in the process of calculating the four correlation values, and has the largest correlation reliability value Bpf_Max among the four correlation values, that is, the highest reliability. Are used as the correlation values of the interpolated pixel X. The selection circuit 41 realizes a process corresponding to step S16 in the flowchart shown in FIG.

  The average value calculation circuit 42 calculates the average value of the correlation values at the top two locations selected by the selection circuit 41, and outputs it as one correlation value S_NH, S_NV from each of the NH direction and the NV direction. The average value calculation circuit 42 realizes a process corresponding to step S17 in the flowchart shown in FIG.

  The comparison circuit 43 calculates the directionality of the correlation of the interpolation pixel X, that is, in which direction the correlation of the interpolation pixel X with the surrounding pixels is strong. Specifically, the comparison circuit 43 includes one correlation value S_H and S_V in each HV direction calculated by the average value calculation circuit 36, and each correlation value in the NH direction and NV direction calculated by the average value calculation circuit 42. The directionality of the correlation is specified by comparing S_NH and S_NV with the correlation diagram shown in FIG. The comparison circuit 43 realizes a process corresponding to step S18 in the flowchart shown in FIG.

  The determination circuit 44 determines whether or not the calculation result of the comparison circuit 43, that is, the directionality having a strong correlation has the reliability of the correlation. Specifically, in the case where the two correlation values S_H and S_NH calculated by the averaging calculation circuits 36 and 42 respectively ride on the two correlation lines (A) and (B) shown in FIG. Although it is considered that there is a correlation, if these two correlation values S_H and S_NH do not ride on the two correlation lines (A) and (B), it is considered that there is no correlation. The determination result by the determination circuit 44 is supplied to the interpolation circuit 45. The determination circuit 44 implements a process corresponding to step S19 in the flowchart shown in FIG.

  The interpolation circuit 45 includes a first interpolation circuit 451 that performs interpolation processing with an emphasis on resolution on the interpolation pixel X, and a second interpolation circuit 452 that performs interpolation processing with an emphasis on S / N, and is supplied from the determination circuit. The interpolation processing is left to one of the interpolation circuits 451 or 452 adaptively according to the reliability of the correlation.

  The first interpolation circuit 451 interpolates using pixels in a direction having a correlation according to the determination result that the reliability of the correlation is high from the determination circuit 44. Interpolation processing performed on pixels in a direction having a correlation is interpolation processing with emphasis on resolution. The first interpolation circuit 451 implements a process corresponding to step S20 in the flowchart shown in FIG.

  On the other hand, the second interpolation circuit 452 performs interpolation using the average value of the peripheral pixels of the interpolation pixel X according to the determination result that the reliability of the correlation is low from the determination circuit 44. For example, the interpolated pixel X is interpolated according to the following equation using the image signals of the four neighboring pixels surrounding the interpolated pixel X. As described above, the interpolation process performed using the average value of the peripheral pixels of the interpolation pixel X is an S / N-oriented interpolation process. The second interpolation circuit 452 implements a process corresponding to step S21 in the flowchart shown in FIG.

    X = (G5 + G8 + G9 + G12) / 4

  Although it has already been described that the HV direction correlation value calculation circuits 31 to 34 and the NH and NV direction correlation value calculation circuits 37 to 40 are configured by bandpass filters, the present invention is not limited to bandpass filters. These correlation value calculation circuits can be configured using, for example, a high-pass filter such as a differential filter, or can be configured as a high-pass filter by combining a low-pass filter and an inverter that inverts the output of the low-pass filter. is there.

  In the color coding shown in FIG. 6, G, which is a main component in creating a luminance component, is eight surrounding pixels positioned in the horizontal, vertical, and diagonal directions with respect to the interpolation pixel X, that is, pixels G4 and G5. , G6, G8, G9, G11, G12, and G13, and achieves higher luminance resolution than the conventional Bayer array. As described above, according to the interpolation processing according to the first embodiment, a plurality of correlation values are obtained and compared with a correlation diagram for a solid-state imaging device having such a color coating filter. Thus, the correlation can be determined for all directions (360 degrees). That is, the G signal can be interpolated with higher accuracy with respect to the color coating shown in FIG. 6, and a luminance signal having a higher resolution than that of the conventional Bayer array can be obtained.

  In particular, in the first embodiment, as described with reference to FIG. 32, the interpolation processing method is adaptively performed depending on whether the correlation reliability is high or low by comparing a plurality of correlation values with a correlation diagram. It is designed to switch. That is, when it is determined that the correlation reliability is high, interpolation processing is performed with emphasis on resolution using the pixel information in the determined direction, and when it is determined that the correlation reliability is low, the correction pixel By performing interpolation with an emphasis on S / N using values obtained by averaging peripheral pixel information, high-performance interpolation processing with higher resolution and resistance to S / N can be realized.

[Example 2]
FIG. 17 shows an example of color coding to be subjected to interpolation processing according to the second embodiment of the present invention. In the color coding shown in the figure, each pixel pitch in the horizontal and vertical directions is set to √2d, and each pixel is shifted by 1/2 of the pixel pitch √2d for each row and each column (in odd and even rows). In contrast to the so-called diagonal pixel arrangement, G and R are alternately arranged in the first row, with the pixel pitch shifted by 1/2 of the pixel pitch in the horizontal direction and by odd and even columns by 1/2 of the pixel pitch in the vertical direction. The second line is a G line in which only G is arranged, the third line is a GB line in which B and G are alternately arranged, the fourth line is a G line in which only G is arranged, and so on. These four rows are repeated as a unit.

  In the color coding example shown in FIG. 17, the color component (G in this example) that is the main component in creating the luminance (Y) component and the other color components (R and B in this example) are G and R And B are arranged so as to surround the periphery, and R and B are arranged at intervals of 2√2d with respect to the horizontal and vertical directions. This color coding is color coding itself in which the color arrangement in the color coding example shown in FIG. 6 is inclined by 45 degrees.

  When the sampling rate in the color coding example shown in FIG. 17 is considered in the horizontal and vertical directions, the sampling rate of G is d / √2 and the sampling rate of R and B is 2√2d. That is, R and B are arranged every other column (odd columns in this embodiment) and every other row (odd rows in this embodiment) so that the sampling rates in the horizontal and vertical directions are 1/4 with respect to G. Has been placed. Therefore, there is a fourfold resolution difference between G and R and B in the horizontal and vertical directions. Further, when considering the sampling rate in a 45 ° oblique direction, the G sampling rate is d, and the R and B sampling rates are 2d.

  Here, let us consider the spatial frequency characteristics. In the horizontal and vertical directions, since the sampling rate of G is d / √2, the G signal can be captured up to a frequency of (1 / √2) fs based on the sampling theorem. Since the sampling rate of G is d in the oblique 45 degree direction, the G signal can be captured up to a frequency of (1/4) fs based on the sampling theorem.

  Similarly, consider R and B. However, since R and B have the same pixel array interval, only R will be described here. Regarding the spatial frequency characteristics of R, since the sampling rate of R is 2√2d in the horizontal and vertical directions, it is possible to capture the R signal up to a frequency of (1 / 4√2) fs based on the sampling theorem. . Further, in the direction of 45 degrees obliquely, since the sampling rate of R is 2d, the R signal can be captured up to a frequency of (1/2) fs based on the sampling theorem.

  Incidentally, the solid-state imaging device having an oblique pixel arrangement can obtain a high resolution because the pixel pitch is narrower than that in the case of a square lattice pixel arrangement. In addition, in the case of the same resolution as that of the square lattice pixel arrangement, it is possible to arrange the pixels with a pixel pitch wider than the pixel pitch of the square lattice pixel arrangement. As a result, S / N can be improved.

  For details of the color coding shown in FIG. 17, for example, refer to paragraphs 0055 to 0072 of Japanese Patent Application No. 2005-107037 already assigned to the present applicant.

  The second embodiment is characterized by an interpolation process for the color coding example shown in FIG. Hereinafter, the interpolation process will be specifically described.

  In FIG. 18, the color coding example shown in FIG. 17 is re-expressed in a square lattice. It can be seen that the arrangement of G in the color coding example shown in FIG. 6 and the arrangement of G in FIG. 18 are in a relationship rotated by 45 degrees. Accordingly, with respect to the color coding example shown in FIG. 17, the G pixel at the spatial position of the R and B pixels can be obtained with high accuracy in the positional relationship obtained by rotating the interpolation processing according to the first embodiment described above by 45 degrees. You will understand that you can interpolate.

  FIG. 19 shows a configuration example of an interpolation processing circuit 23B that executes the interpolation processing according to the second embodiment. As shown in the figure, the interpolation processing circuit 23B according to the present embodiment has a two-stage configuration including a pre-stage interpolation processing circuit 231 and a post-stage interpolation processing circuit 232.

  The pre-stage interpolation processing circuit 231 basically performs interpolation processing in the same procedure as in the first embodiment, and thereby interpolates G pixels at the spatial positions of R and B pixels in the G pixel array in FIG. it can. FIG. 20 shows the result of the interpolation processing by the pre-stage interpolation processing circuit 231, that is, the result of interpolating G pixels at the spatial positions of R and B pixels in the G pixel array of FIG. 18. By this interpolation processing, the G array in FIG. 20 is interpolated with high accuracy. Here, when attention is paid to the arrangement of G in FIG. 20, it can be seen that G is arranged in a checkered pattern.

  The post-stage interpolation processing circuit 232 basically performs an interpolation process in the same procedure as that for the Bayer array described above, and thereby performs an interpolation process on the G pixel array in FIG. By this Bayer array interpolation processing, G is generated for all pixels from G arranged in a checkered pattern. FIG. 21 shows the result of interpolation processing by the post-stage interpolation processing circuit 232, that is, the result of interpolation from G arranged in a checkered pattern to all pixels G.

  As described above, with respect to the color coding example shown in FIG. 17, the interpolation processing by the pre-stage interpolation processing circuit 231, that is, the interpolation processing according to the first embodiment, and the interpolation processing by the post-stage interpolation processing circuit 232, that is, the Bayer array interpolation processing. By performing interpolation processing in combination, it is possible to obtain G resolution up to the limit resolution.

  FIG. 22 shows the spatial resolution characteristics in each of the Bayer array, the color coding shown in FIG. 6, and the color coding shown in FIG. Comparing the color coding resolution shown in FIG. 6 with the color coding resolution shown in FIG. 17, the color coding resolution shown in FIG. 17 is more advantageous in the horizontal and vertical directions. In general, when a subject is photographed with a camera, an image including a linear component is often distributed in the horizontal and vertical directions. From this point of view, it can be said that the color coding shown in FIG. 17 is more advantageous than the color coding shown in FIG.

  Further, considering that the image signal needs to be output in a square lattice pattern as the output of the camera signal processing circuit 14 (see FIG. 1), the color coding shown in FIG. 17 outputs twice the number of pixels. There is an advantage that it can be output as a size. Of course, it can be understood from FIG. 22 that the resolution is significantly higher than that of the Bayer array.

  In the camera signal processing circuit 14, by using the interpolation processing circuit 23B according to the second embodiment as the interpolation processing circuit 23 as it is, the above-described operation and effect can be obtained. Further, it is possible to adopt a configuration such as an interpolation processing circuit 23B ′ obtained by modifying the interpolation processing circuit 23B as shown in FIG.

  The interpolation processing circuit 23B ′ shown in FIG. 23 is provided with a changeover switch 233 on the input side of the post-stage interpolation processing circuit 232, and the output signal of the pre-stage interpolation processing circuit 231 and the output signal of the WB circuit 22 (see FIG. 1). One of the signals (input signal of the pre-stage interpolation processing circuit 231) can be selectively input to the post-stage interpolation processing circuit 232. By adopting such a configuration, one camera system can correspond to the two types of solid-state image pickup devices of color coding and Bayer arrangement shown in FIG.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the gist of the present invention.

  In the present invention, in particular, for a pixel array in which each pixel is arranged in a square lattice so as to be equally spaced in the horizontal direction and the vertical direction, the color component that is the main component when calculating the luminance component is For image signals that have been color-coded using color filters that are arranged so as to surround each of the color components of the color components, the present invention can be suitably applied to the color filter array interpolation processing of the main components for calculating the luminance components. However, the gist of the present invention is not limited to a specific color coding.

  Further, in this specification, two color coding examples shown in FIG. 8 and FIG. 17 are used as the color coding surrounding G and R which are the main components in creating the luminance component. Although the example has been mainly described, the color coding to which the present invention can be applied is not limited to these two color coding examples. For example, with respect to a square lattice pixel arrangement, the first line is arranged by repeating RGGG in units of 4 pixels in the horizontal direction, the second line is arranged with only G, and the third line is 4 pixels in the horizontal direction. In the same way, the interpolation processing according to the present invention is applied even to color coding that is arranged by repeating GGBG with 4 as the unit, only G is arranged in the 4th row, and the 4 rows are repeated as a unit thereafter. can do.

  In this specification, the G signal interpolation processing for realizing a high resolution with respect to the luminance signal has been described. However, the same interpolation processing as G is performed for R, B, or other colors such as cyan and yellow. It is also possible to apply to.

  In short, the present invention has been disclosed in the form of exemplification, and the description of the present specification should not be interpreted in a limited manner. In order to determine the gist of the present invention, the description of the claims should be considered.

FIG. 1 is a block diagram illustrating an example of a configuration of an imaging apparatus to which an image processing apparatus or an image processing method according to the present invention is applied. FIG. 2 is a diagram showing a pixel array in which only G is extracted in the Bayer array. FIG. 3 is a diagram showing a resolution chart. FIG. 4 is a diagram illustrating the frequency characteristics of the bandpass filter. FIG. 5 is a diagram showing limit resolutions of G resolution and RB resolution in the Bayer array. FIG. 6 is a diagram illustrating an example of color coding to be subjected to interpolation processing according to the first embodiment of the present invention. FIG. 7 is a diagram illustrating limit resolutions of G resolution and RB resolution in a color coding example. FIG. 8 is a diagram showing a pixel array in which only G is extracted in the color coding example. FIG. 9 is a diagram illustrating a relationship among the H direction, the V direction, the NH direction, and the NV direction. FIG. 10 is a flowchart showing the procedure of the interpolation processing according to the first embodiment of the present invention. FIG. 11 is a diagram illustrating frequency characteristics of the bandpass filter in the HV direction. FIG. 12 is a diagram illustrating frequency characteristics of band pulse filters in the NH and NV directions. FIG. 13 shows a resolution chart. FIG. 14 is a correlation diagram showing the relationship between each point in the frequency chart and its correlation value. FIG. 15 is a diagram showing the relationship between the correlation value and the interpolation value. FIG. 16 is a block diagram illustrating an example of a configuration of an interpolation processing circuit that executes the interpolation processing according to the first embodiment. FIG. 17 is a diagram illustrating an example of color coding to be subjected to interpolation processing according to the second embodiment of the present invention. FIG. 18 is a diagram in which a color coding example is expressed by a square lattice. FIG. 19 is a block diagram illustrating an example of a configuration of an interpolation processing circuit that executes the interpolation processing according to the second embodiment. FIG. 20 is a diagram illustrating a result of the interpolation processing by the pre-stage interpolation processing circuit. FIG. 21 is a diagram illustrating a result of the interpolation processing by the post-stage interpolation processing circuit. FIG. 22 is a diagram showing spatial resolution characteristics in the Bayer array, color coding 1, and color coding 2. FIG. 23 is a block diagram illustrating a modification of the interpolation processing circuit that executes the interpolation processing according to the second embodiment. FIG. 24 is a diagram illustrating color coding of the Bayer array. FIG. 25 is a diagram illustrating a state in which demosaic processing is performed on an image on which color coding has been performed. FIG. 26 is a diagram showing how the correlation values in the HV direction are calculated with the four surrounding pixels of the interpolation pixel X as the centers. FIG. 27 is a diagram illustrating a state in which S_H_G4 and S_H_G13 are selected as reliable correlation values. FIG. 28 is a diagram showing how the correlation value in the HV direction centering on the interpolation pixel X is calculated. FIG. 29 is a diagram illustrating how the correlation values in the NH and NV directions with the four surrounding pixels of the interpolation pixel X as the center are calculated. FIG. 30 is a diagram illustrating a state in which S_NH_G5 and S_NH_G9 are selected as reliable correlation values. FIG. 31 is a diagram showing how the correlation values in the NH and NV directions with the interpolation pixel X center are calculated. FIG. 32 shows a correlation line composed of absolute values obtained by subtracting 0.5 from the complement S_H (correlation line (A) in FIG. 14) and 0.5 from S_NH (correlation line (B) in FIG. 14). It is the correlation diagram which plotted the correlation line which consists of measured absolute value, respectively.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Imaging lens 12 ... Imaging device 14 ... Camera signal processing circuit 21 ... Optical system correction circuit 22 ... WB (white balance) circuit 23, 23A, 23B, 23B '... Interpolation processing circuit 24 ... Gamma correction circuit 25 ... Y (luminance) ) Signal processing circuit 26 ... C (chroma) signal processing circuit 27 ... Band-limited LPF (low-pass filter)
28 ... Decimation processing circuit 31-34, 37-40 ... Correlation value calculation circuit 35, 41 ... Selection circuit 36, 42 ... Average value calculation circuit 43 ... Comparison circuit 44 ... Determination circuit 45 ... Interpolation circuit

Claims (12)

In an image processing apparatus that processes an image captured by an imaging unit having a color filter for color coding in which a G component is arranged so as to surround each of the other RB components,
Means for inputting an image signal;
Among the peripheral pixels surrounding the interpolation pixel G signal is missing, the correlation of the first of the surrounding pixels, the G signal of the pixel of the first orthogonal direction G signal adjacent to the first orthogonal direction are present A first correlation value calculating means for obtaining the first correlation value indicating a degree, and averaging a predetermined number from a higher correlation value to obtain the first correlation value of the interpolation pixel;
The second correlation value indicating a degree of correlation with a pixel in a second orthogonal direction different from the first orthogonal direction for a second peripheral pixel other than the first peripheral pixel among the respective peripheral pixels. A second correlation value calculating unit that averages a predetermined number from the higher correlation value and sets the second correlation value of the interpolated pixel;
Interpolation pixel specifying means for specifying an interpolation pixel to be used when interpolating the interpolation pixel based on the first correlation value and the second correlation value;
Interpolating means for interpolating the G signal of the interpolation pixel using the G signal existing in the interpolation pixel specified by the interpolation pixel specifying means;
An image processing apparatus comprising: The correlation value calculation means calculates the first correlation value for each of the surrounding pixels based on the outputs of two orthogonal filters, and has different directional characteristics from the calculation of the first correlation value. Calculating the second correlation value based on the outputs of two orthogonal filters having
The image processing apparatus according to claim 1. The correlation value calculating means uses a filter whose direction characteristics are shifted by a predetermined angle from the filter used for calculating the first correlation value in the calculation of the second correlation value.
The image processing apparatus according to claim 2. The interpolating pixel specifying means compares the correlation between the first correlation value and the second correlation value, and based on the correlation of 360 degrees around the desired pixel to be interpolated, Identify other pixels to use when interpolating the pixels,
The image processing apparatus according to claim 2. On the color filter for color coding, the G component surrounds each of the RB components with respect to a pixel array in which the pixels are arranged in a square lattice pattern at equal intervals in the horizontal direction and the vertical direction. Arranged so that,
The image processing apparatus according to claim 1. On the color filter for color coding, the G component surrounds each of the RB components with respect to an oblique pixel array in which each pixel is shifted by ½ of the pixel pitch for each row and for each column. Located in the
The image processing apparatus according to claim 1. The interpolation pixel specifying means obtains the directionality of the correlation of the interpolation pixel based on the first correlation value and the second correlation value, and uses surrounding pixels in a direction with strong correlation as interpolation pixels.
The image processing apparatus according to claim 1. A reliability calculation means for calculating reliability related to the first correlation value and the second correlation value;
The interpolation pixel specifying means specifies an interpolation pixel according to the reliability.
The image processing apparatus according to claim 1. The interpolation pixel specifying means determines the reliability based on a sum of the first correlation value and the second correlation value;
The image processing apparatus according to claim 8. When the reliability falls within a predetermined range, the interpolation pixel specifying means sets the surrounding pixels in the direction of strong correlation as interpolation pixels.
The image processing apparatus according to claim 8. When the reliability is outside a predetermined range, the interpolation pixel specifying unit does not specify an interpolation pixel, and the interpolation unit performs interpolation using an average value of information on peripheral pixels of the interpolation pixel.
The image processing apparatus according to claim 8. In an image processing method for processing an image picked up by an image pickup means having a color coding color filter for color coding arranged so that a G component surrounds each of the other RB components,
Among the peripheral pixels surrounding the interpolation pixel G signal is missing, the correlation of the first of the surrounding pixels, the G signal of the pixel of the first orthogonal direction G signal adjacent to the first orthogonal direction are present A first correlation value calculating step of obtaining the first correlation value indicating a degree and averaging the predetermined number from the higher correlation value to obtain the first correlation value of the interpolation pixel;
The second correlation value indicating a degree of correlation with a pixel in a second orthogonal direction different from the first orthogonal direction for a second peripheral pixel other than the first peripheral pixel among the respective peripheral pixels. A second correlation value calculating step that averages a predetermined number from the higher correlation value to obtain the second correlation value of the interpolated pixel;
An interpolation pixel specifying step for specifying an interpolation pixel to be used when interpolating the interpolation pixel based on the first correlation value and the second correlation value;
An interpolation step of interpolating the G signal of the interpolation pixel using the G signal existing in the interpolation pixel specified in the interpolation pixel specifying step;
An image processing method comprising: