JP2006033333A - Method and apparatus of processing pixel signal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suppress the generation of a false color in an interpolation result even in a region having no similar relation in the state of the change of different color component values from one another in the interpolation of the insufficient color component value of the pixels arranged in a two-dimensional flat surface, and to interpolate always by an optimum interpolating method irrespective of the way of the change of the color component value in the region near the pixel to be interpolated. <P>SOLUTION: A method of processing the pixel signal includes the steps of judging whether the pixel having the color component value of an insufficient color exists in a direction having a strong correlation with the pixel to be interpolated as a center (ST10, 11, 12A, and 12B), executing an interpolating process by a first interpolating method (ST13A, ST13B) when an "existence" is judged, and executing the interpolating process by a second interpolating method (ST14A, ST14B) when "no existence" is judged. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a pixel signal processing method and apparatus, and more particularly to a pixel signal processing unit that is arranged on a two-dimensional plane and does not have at least one color component value among a plurality of color component values. The present invention relates to a pixel signal processing method and apparatus for obtaining a color image by generating non-color component values by interpolation.

  Such pixel signal processing is, for example, a plurality of color component values each generating a plurality of color component values, for example, any one of the three primary colors of red (R), green (G), and blue (B). A type of photoelectric conversion element is used as a part of a color imaging device further including, for example, a Bayer-type imaging element on a two-dimensional plane, and is missing at each pixel position among pixel signals output from the imaging element It is used to interpolate the color component value (insufficient color component value).

  In a conventional imaging device having an imaging device in which red, green, and blue color filters are arranged in a Bayer shape, G, B or B, R or R, and G color component values are insufficient for each pixel. For example, as shown in Patent Document 1 below, the pixel signal of each pixel is replaced with an average value based on the local pixel signal distribution for each color in order to enhance the resolution, and this is assumed. An interpolation method based on the linear similarity ratio between known color geometric figures and missing color geometric figures is used.

JP 2001-197512 A (paragraphs 0048 to 0049, FIG. 8)

  In this conventional method, it is assumed that there is a similar relationship in the state of change of each color component value (for example, R, G, B component values in a Bayer array) in a region near the interpolation target pixel. ing. For this reason, there is a problem that a false color is generated in an interpolation result in an area where the color component value changes are not similar to each other (for example, a boundary between one color and another color). There has been a problem that the interpolation method may not be appropriate depending on how the plurality of color component values change in a nearby region.

  The present invention can suppress the occurrence of a false color in an interpolation result even in an area where there is no similar relationship in the state of change in color component values in an area near the interpolation target pixel. It is an object of the present invention to provide a pixel signal processing method and apparatus that can always perform interpolation by an optimal interpolation method regardless of how color component values change in an area in the vicinity of a target pixel.

The present invention
Each color component value of a plurality of pixels arranged on a two-dimensional plane and having at least one color component value among first to Nth (N is an integer of 2 or more) different color component values. The Kth color (K is any number from 1 to N, but different from J) of the interpolation target pixel having the Jth color component value (J is any number from 1 to N). In a pixel signal processing method for generating component values by interpolation,
An image feature determination step for determining whether or not there is a pixel having the Kth color component value in a direction having a strong correlation around the interpolation target pixel;
When it is determined that “exists” in the image feature determination step, interpolation processing is executed by the first interpolation method. When it is determined that “does not exist” in the image feature determination step, the first An interpolation step for performing an interpolation process with a second interpolation method different from the one interpolation method,
The interpolation by the first and second interpolation methods is both the J-th color component value of the interpolation processing target pixel and the pixels around the interpolation processing target pixel, and the pixels around the interpolation processing target pixel. The pixel signal processing method is characterized in that the K-th color component value of the interpolation target pixel is interpolated based on the K-th color component value of the pixel.

  According to the present invention, generation of false colors can be suppressed, and optimal interpolation can be performed in accordance with how the color component value of a pixel changes in an area near the interpolation target pixel.

  Embodiments of the present invention will be described below with reference to the drawings. The embodiment described below is suitable for use as a part of a color imaging apparatus such as a digital still camera, but the present invention is not limited to this.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a configuration of an imaging apparatus provided with a pixel signal processing apparatus according to Embodiment 1 of the present invention. FIG. 2 shows a configuration of the pixel signal processing circuit 4 of FIG. As shown in FIG. 2, the pixel signal processing circuit 4 includes a frame memory 10 and a processor 11.
The light incident from the lens 1 forms an image on the imaging surface of a two-dimensional image sensor, for example, the solid-state image sensor 2. The solid-state imaging device 2 includes a plurality of photoelectric conversion elements arranged two-dimensionally. The plurality of photoelectric conversion elements are arranged in a red (Bayer) type, for example, as shown in FIG. Covered with a color filter having spectral sensitivity characteristics corresponding to the three primary colors of R), green (G), and blue (B), and from each photoelectric conversion element, an analog signal of a color component corresponding to the color of the color filter Is output.

In FIG. 3, horizontal and vertical represent the horizontal direction (H) and vertical direction (V) of the imaging surface, respectively. The photoelectric conversion element constitutes a pixel, and the position occupied by each photoelectric conversion element on the imaging surface corresponds to the pixel position. Since each pixel is two-dimensionally arranged on the image pickup surface of the image pickup device, their position can be expressed by a coordinate value on the HV coordinate plane (or HV plane).
In the following description, pixels corresponding to photoelectric conversion elements covered with R color filters are covered with R pixels, and pixels corresponding to photoelectric conversion elements covered with G color filters are covered with G pixels and B color filters. A pixel corresponding to the photoelectric conversion element is called a B pixel.

  The solid-state imaging device 2 photoelectrically converts incident light and outputs an analog signal having a level corresponding to the amount of incident light for each pixel. The analog signal is converted to a digital signal by the A / D converter 3 and output, and is written in the frame memory 10 in the pixel signal processing circuit 4 as a color component value of each pixel. At this time, each signal is written in association with the position of each pixel on the imaging surface, and hence the position on the HV coordinate plane.

  As described above, only signals representing any one of R, G, and B color component values can be obtained from each of the photoelectric conversion elements constituting each pixel. That is, for the R pixel, the R component value is known, while the G and B component values are unknown, and for the G pixel, the G component value is known, while the B and R component values are unknown, For the B pixel, the B component value is known, while the R and G component values are unknown. Since each pixel has all R, G, and B color component values, a color image can be obtained. Therefore, an unknown color component value written in the frame memory 10 at each pixel position is an insufficient color component. Also called value. In the pixel signal processing of the present invention, a color component value (insufficient color component value) that is unknown in each pixel is obtained by interpolation.

  The processor 11 shown in FIG. 2 is shown in the flowcharts of FIGS. 4, 5, 6, and 7 for interpolation processing of insufficient color component values for each pixel of the Bayer array type image data shown in FIG. 3. To work. The insufficient color component value (interpolation value) of each pixel calculated by the processor 11 is written in the frame memory 10 in association with the position of each pixel on the HV coordinate plane.

Hereinafter, the interpolation processing of the insufficient color component value performed in the pixel signal processing circuit 4 will be described. For interpolation of missing color component values,
(P1) Processing for obtaining G component values in the R pixel and B pixel,
(P2) Processing for obtaining the R component value and the B component value in the G pixel,
(P3) A process for obtaining the R component value in the B pixel, and (P4) a process for obtaining the B component value in the R pixel,
For example, in the order described above.

  First, the interpolation processing of the G component value in the R pixel and the B pixel in (P1) will be described. First, the interpolation of the G component value in the R pixel will be described with reference to FIGS. FIG. 4 is a flowchart showing the operation of the processor 11 when interpolating the G component value in the R pixel. FIG. 8 schematically shows the color component values of the pixels used in the interpolation of the G component value in the R pixel (R33) among the color component values of each pixel written in the frame memory 10. ing. The symbol at each pixel position represents the color component value of each pixel, and indicates whether the color of the first alphabet (R, G, B) is R, G, or B, followed by 2 A digit number represents the position of each pixel. These symbols representing the color component values of each pixel are also used to represent each pixel. (The same notation is used in the following figures.) In the illustrated example, the position of the interpolation processing target pixel is represented by (33), and the interpolation processing target pixel and its color component value are represented by R33. That is, in the following description, it is assumed that the R pixel R33 is an interpolation target pixel, and the G component value at the position (33) of the R pixel (R33) is generated by interpolation.

The color component value of each pixel used in the processing of each step described below is read from the frame memory 10 as necessary.
First, in steps ST10, ST11, ST12A, ST12B, the color component value (G component value) of the same color as the color component (G component) to be interpolated in the direction in which the image correlation is strong with the interpolation target pixel R33 as the center. A determination is made whether there is a pixel having.
First, in step ST10 and step ST11, it is determined whether the correlation is strong in the horizontal direction or the vertical direction around the interpolation processing target pixel R33. For this determination, a part of color component values of pixels around the interpolation processing target pixel R33 is used to calculate the following values dV and dH (step ST10), and it is determined whether dH <dV. (Step ST11).

dH = | G23-G43 |
dV = | G32-G34 |
... (1)

  Here, dH is the G component value of the pixel in the horizontal direction centered on the interpolation processing target pixel R33, that is, two pixels G23 that are aligned in the horizontal direction with respect to the interpolation processing target pixel R33 and are adjacent to the interpolation processing target pixel R33. , G43 represents the absolute value of the difference between G component values G23 and G43, and dV is the vertical G component value of the pixel in the vertical direction centered on the interpolation processing target pixel R33, that is, vertically aligned with respect to the interpolation processing target pixel R33. The absolute value of the difference between the G component values G32 and G34 of the two pixels G32 and G34 adjacent to the interpolation target pixel. For example, dH <dV when the horizontal correlation is strong in the image like a horizontal stripe image, and dV <dH when the vertical correlation is strong like a vertical stripe image. Note that the equations for obtaining dH and dV are not limited to Equation (1).

  Depending on the magnitude relationship between dV and dH, the process after step ST12A or the process after step ST12B is performed. That is, when dH <dV (when it is determined that the correlation in the horizontal direction is strong), the processing after step ST12A is performed, and otherwise (when it is determined that the correlation in the vertical direction is strong), step ST12B. The following processing is performed.

Next, in step ST12A, the pixel having the G component value in the horizontal direction (the direction in which the correlation is determined to be strong in step ST11) around the interpolation target pixel R33, that is, the same color as the color component value to be interpolated. A determination is made whether there are any pixels with color component values.
When it is determined that it exists, the process proceeds to step ST13A, and the first interpolation is executed. If it is determined that it does not exist, the process proceeds to step ST14A, where the second interpolation is executed. As is apparent from FIG. 8, since there is a pixel having a G component value in the horizontal direction centering on the R pixel, the process proceeds to step ST13A and the first interpolation is executed.

  In step ST13A, the G color component values G23 and G43 of the G pixels G23 and G43 aligned in the horizontal direction with respect to the interpolation processing target pixel R33 and the interpolation processing target pixel R33, and the R color component values R33 of the R pixels R33, R13, and R53, Using R13 and R53, the interpolation calculation of the following equation (2) for obtaining the G component value g33 in the interpolation target pixel R33 is performed. As will be described later, this interpolation calculation is an interpolation calculation that assumes “similarity”.

  Then, the G component value g33 obtained by the interpolation calculation of Expression (2) is written in the frame memory 10 in association with the position of the interpolation processing target pixel R33.

In step ST12B, it is determined whether or not there is a pixel having a G component value in the vertical direction (the direction in which the correlation is determined to be strong in step ST11) around the interpolation target pixel R33.
When it is determined that it exists, the process proceeds to step ST13B, and the first interpolation is executed. If it is determined that it does not exist, the process proceeds to step ST14B, and the second interpolation is executed. As is apparent from FIG. 8, since there is a pixel having a G component value in the vertical direction centered on the R pixel, the process proceeds to step ST13B and the first interpolation is executed.

  In step ST13B, the G color component values G32 and G34 of the G pixels G32 and G34 aligned in the vertical direction with respect to the interpolation processing target pixel R33 and the interpolation processing target pixel R33, and the R component values R33 and R31 of the R pixels R33, R31, and R35. , R35, the interpolation calculation of the following equation (3) for obtaining the G component value g33 in the interpolation target pixel R33 is performed. This interpolation calculation is also an interpolation calculation “assuming a similar relationship”, similar to the equation (2).

  Then, the G component value g33 obtained by the interpolation calculation of Expression (3) is written in the frame memory 10 in association with the position of the interpolation processing target pixel R33.

  The above interpolation processing is executed for all R pixels, and the G component value at the R pixel is obtained by interpolation.

  Next, interpolation of the G component value in the B pixel will be described. FIG. 9 schematically shows the color component values of the pixels used in the interpolation of the G component value in the B pixel B33 among the color component values of each pixel written in the frame memory 10. As is apparent from a comparison of FIG. 9 with FIG. 8, in FIG. 8 and FIG. 9, only the R pixel, the B pixel, and the color component values of each pixel are interchanged. Therefore, the interpolation of the G component value in the B pixel can be performed in the same manner except that R and B of the interpolation of the G component value in the R pixel described above are replaced.

  When the G component values in the R pixel and the B pixel are interpolated by the above interpolation processing, the color component values of each pixel written in the frame memory 10 are schematically shown in FIG. Note that capital letters R, G, and B in FIG. 10 represent R component values, G component values, and B component values given by the solid-state imaging device 2 among the color component values of each pixel, and a small letter g represents interpolation processing. Represents the G component value obtained by.

  Next, the interpolation processing of the R component value and the B component value in the G pixel in (P2) will be described. In the Bayer array, as shown in FIG. 3, the color of pixels aligned in the horizontal direction and the vertical direction with respect to the interpolation processing target pixel differs depending on the row to which the G pixel that is the interpolation processing target pixel belongs. That is, when the G pixels in every other row in FIG. 3, that is, the first, third,... Row (hereinafter referred to as “odd row” for convenience) in FIG. As shown in FIG. 3, the B pixels are aligned in the horizontal direction and the R pixels are aligned in the vertical direction. On the other hand, in FIG. When the G pixels in the fourth, fourth,... Rows (hereinafter referred to as “even-numbered rows”) are set as the interpolation target pixels, as shown in FIG. B pixels are aligned in the direction.

FIG. 5 is a flowchart showing the operation of the processor 11 at the time of interpolation of the R component value in the G pixel. 12 and 13 show the color component values of the pixels used for interpolation of the R component value and the B component value in the G pixel G33 among the color component values of each pixel written in the frame memory 10. Is schematically represented. The symbol at each pixel position represents the color component value possessed by each pixel, but the symbol beginning with a capital letter R, G, or B has the R component value originally possessed by each pixel (ie, given by the solid-state imaging device 2). , G component value and B component value, and a symbol beginning with a lowercase letter g represents a G component value obtained by the interpolation of (P1) above. Similarly to FIG. 8, symbols starting with capital letters R, G, and B are also used to represent each pixel. FIG. 12 shows a case where the interpolation processing target pixel G33 is in an odd row, and FIG. 13 shows a case where the interpolation processing target pixel G33 is in an even row.
First, interpolation of the R component value in the interpolation processing target pixel when the interpolation processing target pixel G33 is in an odd row will be described.

The color component value of each pixel used in the processing of each step described below is read from the frame memory 10 as necessary.
First, in steps ST20, ST21, ST22A, ST22B, the color component value (R component value) of the same color as the color component (R component) to be interpolated in the direction in which the image correlation is strong around the interpolation processing target pixel G33. A determination is made whether there is a pixel having.
First, in steps ST20 and ST21, it is determined whether the correlation is strong in the horizontal direction or the vertical direction around the interpolation processing target pixel G33. For this determination, part of the color component values of the pixels around the interpolation target pixel G33 is used, and the values dV and dH are calculated by the following equation (4) (step ST20). Judgment is made (step ST21). Note that the equations for obtaining dH and dV are not limited to Equation (4).

dH = | g23−g43 |
dV = | g32−g34 |
(4)

  Here, dH is the G component value of the pixel in the horizontal direction around the interpolation processing target pixel G33, that is, two pixels B23 that are aligned in the horizontal direction with respect to the interpolation processing target pixel G33 and are adjacent to the interpolation processing target pixel G33. , B43 represents the absolute value of the difference between the G component values g23 and g43, and dV is the vertical G component value centered on the interpolation target pixel G33, that is, vertically aligned with respect to the interpolation target pixel G33. The absolute value of the difference between the G component values g32 and g34 of the two pixels R32 and R34 adjacent to the interpolation processing target pixel G33. When the correlation in the horizontal direction is strong, dH <dV, and when the correlation in the vertical direction is strong, dV <dH.

  Depending on the magnitude relationship between dV and dH, the process after step ST22A or the process after step ST22B is performed. That is, when dH <dV (when it is determined that the correlation in the horizontal direction is strong), the processing after step ST22A is performed, and otherwise (when it is determined that the correlation in the vertical direction is strong), step ST22B. The following processing is performed.

Next, in step ST22A, it is determined whether or not there is a pixel having an R component value in the horizontal direction (the direction in which the correlation is determined to be strong in step ST21) around the interpolation target pixel G33.
When it is determined that it exists, the process proceeds to step ST23A, and the first interpolation is executed. If it is determined that it does not exist, the process proceeds to step ST24A, where the second interpolation is executed. As is apparent from FIG. 12, since there is no pixel having the R component value in the horizontal direction centering on the G pixel, the process proceeds to step ST24A and the second interpolation is executed.

  The details of the second interpolation performed in step ST24A are shown in FIG. In the second interpolation, interpolation by regression analysis or linear interpolation is selectively performed.

In step ST40 of FIG. 7, pixels around the interpolation target pixel G33, that is,
G component value g12 and R component value R12 of R pixel R12,
G component value g32 and R component value R32 of the R pixel R32,
G component value g52 and R component value R52 of R pixel R52,
G component value g14 and R component value R14 of R pixel R14,
G component value g34 and R component value R34 of the R pixel R34,
Using the G component value g54 and the R component value R54 of the R pixel R54, two-dimensional coordinate data (xi, yi) (i = 1 to 6) represented by the following equation (5) is created. At this time, the color component values g12, g32, g52, g14, g34, and g54 of the same color as the color component value G33 of the interpolation processing target pixel G33 are set to xi, and the color component value to be interpolated in the interpolation processing target pixel G33. Let yi be the color component values R12, R32, R52, R14, R34, and R54 of the same color as r33.

(X1, y1) = (g12, R12)
(X2, y2) = (g32, R32)
(X3, y3) = (g52, R52)
(X4, y4) = (g14, R14)
(X5, y5) = (g34, R34)
(X6, y6) = (g54, R54)
... (5)

  Next, in step ST41, it is determined whether the difference between the maximum value and the minimum value of xi (i = 1 to 6) is equal to or less than a predetermined threshold value. If it is equal to or less than the threshold value, step ST42 is executed, and if not, step ST43 and step ST44 are executed.

  In step ST42, the R component value r33 in the interpolation target pixel G33 is calculated by linear interpolation. That is, an interpolation calculation is performed by the following equation (6), and the obtained value r33 is written in the frame memory 10 in association with the position of the interpolation processing target pixel G33.

  The calculation of Expression (6) is to obtain the average of the color component values R32 and R34 of the same color as the color component value r33 to be interpolated for the pixels located around the interpolation target pixel G33.

  Step ST43 and step ST44 perform interpolation by regression analysis. In interpolation by regression analysis, regression analysis is performed with xi of the two-dimensional coordinate values as explanatory variables and yi as an objective variable, and a regression expression representing the relationship between xi and yi, particularly its slope a and y intercept b And the color component value of the unknown color in the interpolation target pixel is calculated using this regression equation. Specifically, in step ST43, the slope a and the y-intercept b of the straight line y = a × x + b are obtained by the calculation shown in Expression (7).

  In step ST44, the R component value r33 in the interpolation processing target pixel G33 is calculated by the following equation (8) using the straight line y = a × x + b and the G component value G33 in the interpolation processing target pixel G33, and the interpolation processing target pixel G33. Is written in the frame memory 10 in association with the position of.

r33 = a × G33 + b
... (8)

  The above is the process after step ST22A. Next, the processing after step ST22B will be described.

In step ST22B, it is determined whether or not there is a pixel having an R component value in the vertical direction (the direction in which the correlation is determined to be strong in step ST21) around the interpolation target pixel G33.
When it is determined that it exists, the process proceeds to step ST23B, and the first interpolation is executed. If it is determined that it does not exist, the process proceeds to step ST24B, where the second interpolation is executed. As is apparent from FIG. 12, since there is a pixel having an R component value in the vertical direction centered on the G pixel, the process proceeds to step ST23B, and the first interpolation is executed.

  In step ST23B, the G component value G33 of the interpolation processing target pixel G33 and the R color component values R32 and R34 of the R pixels R32 and R34 aligned in the vertical direction with respect to the interpolation processing target pixel G33 and the G component values g32 and g34 are used. Thus, the interpolation calculation for obtaining the R component value r33 in the interpolation processing target pixel G33 is performed by the following equation (9). This interpolation calculation is also an interpolation calculation “assuming a similar relationship”, similar to the equation (2).

  The R component value r33 in the interpolation processing target pixel G33 obtained by the above equation (9) is also written in the frame memory 10 in association with the position of the interpolation processing target pixel G33.

The above is the processing performed after step ST22B in FIG. The above processing is performed for all odd-numbered G pixels, and R component values for odd-numbered G pixels are obtained by interpolation. Next, interpolation of B component values in odd-numbered G pixels will be described with reference to FIGS.
FIG. 6 is a flowchart showing the operation of the processor 11 when the B component value is interpolated.

The color component value of each pixel used in the processing of each step described below is read from the frame memory 10 as necessary.
First, in steps ST30, ST31, ST32A, and ST32B, the color component value (B component value) of the same color as the color component (B component) to be interpolated in the direction in which the correlation of the image is strong around the interpolation target pixel G33. A determination is made whether there is a pixel having.
First, in steps ST30 and ST31, it is determined whether the correlation is strong in the horizontal direction or the vertical direction around the interpolation processing target pixel G33. For this determination, a part of the color component values of the pixels around the interpolation target pixel G33 is used, and the values dV and dH are calculated by the following equation (10) (step ST30). Whether or not is determined (step ST31). Note that the equations for obtaining dH and dV are not limited to Equation (10).

dH = | g23−g43 |
dV = | g32−g34 |
(10)

  Depending on the magnitude relationship between dV and dH, the process after step ST32A or the process after step ST32B is performed. That is, when dH <dV (when it is determined that the correlation in the horizontal direction is strong), the processing after step ST32A is performed, otherwise (when it is determined that the correlation in the vertical direction is strong), step ST32B. The following processing is performed.

Next, in step ST32A, it is determined whether or not there is a pixel having a B component value in the horizontal direction (the direction in which the correlation is determined to be strong in step ST31) around the interpolation target pixel G33.
When it is determined that it exists, the process proceeds to step ST33A, and the first interpolation is executed. If it is determined that it does not exist, the process proceeds to step ST34A, where the second interpolation is executed. As is apparent from FIG. 12, since there is a pixel having a B component value in the horizontal direction centering on the G pixel, the process proceeds to step ST33A, and the first interpolation is executed.

  In step ST33A, the G component value G33 of the interpolation processing target pixel G33 and the B color component values B32 and B43 of the B pixels B23 and B43 aligned in the horizontal direction with respect to the interpolation processing target pixel G33 and the G component values g23 and g43 are used. Thus, the interpolation calculation of the following formula (11) is performed to obtain the B component value b33 in the interpolation target pixel G33. This interpolation calculation is also an interpolation calculation “assuming a similar relationship”, similar to the equation (2).

  The B component value b33 in the interpolation processing target pixel G33 obtained by the above equation (11) is also written in the frame memory 10 in association with the position of the interpolation processing target pixel G33.

  The above is the process after step ST32A in FIG. Next, the processing after step ST32B will be described.

In step ST32B, it is determined whether or not there is a pixel having a B component value in the vertical direction (the direction in which the correlation is determined to be strong in step ST31) around the interpolation target pixel G33.
When it is determined that it exists, the process proceeds to step ST33B, and the first interpolation is executed. If it is determined that it does not exist, the process proceeds to step ST34B, and the second interpolation is executed. As is apparent from FIG. 12, since there is no pixel having a B component value in the vertical direction centering on the G pixel, the process proceeds to step ST34B, and the second interpolation is executed.

  Details of the second interpolation executed in step ST34B are shown in FIG. In the second interpolation, interpolation by regression analysis or linear interpolation is selectively performed.

First, in step ST40, pixels around the interpolation target pixel G33, that is,
G component value g21 and B component value B21 of B pixel B21,
G component value g23 and B component value B23 of the B pixel B23,
G component value g25 and B component value B25 of the B pixel B25,
G component value g41 and B component value B41 of the B pixel B41,
G component value g43 and B component value B43 of the B pixel B43,
Using the G component value g45 and the B component value B45 of the B pixel B45, two-dimensional coordinate data (xi, yi) (i = 1 to 6) represented by the following equation (12) is created. At this time, the color component values g21, g23, g25, g41, g43, and g45 of the same color as the color component value G33 of the interpolation processing target pixel G33 are set to xi, and the color component value to be interpolated in the interpolation processing target pixel G33. Let yi be the color component values B21, B23, B25, B41, B43, and B45 of the same color as b33.

(X1, y1) = (g21, B21)
(X2, y2) = (g41, B41)
(X3, y3) = (g23, B23)
(X4, y4) = (g43, B43)
(X5, y5) = (g25, B25)
(X6, y6) = (g45, B45)
(12)

  Next, in step ST41, it is determined whether the difference between the maximum value and the minimum value of xi (i = 1 to 6) is equal to or less than a predetermined threshold value. If it is equal to or less than the threshold value, step ST42 is executed, and if not, step ST43 and step ST44 are executed.

  In step ST42, the B component value b33 in the interpolation target pixel G33 is calculated by linear interpolation. That is, an interpolation calculation is performed by the following equation (13), and the obtained value b33 is written in the frame memory 10 in association with the position of the interpolation processing target pixel G33.

  The calculation of Expression (13) is to obtain the average of the color component values B23 and B43 of the same color as the color component value b33 to be interpolated for the pixels located around the interpolation target pixel G33.

  Step ST43 and step ST44 perform interpolation by regression analysis. In interpolation by regression analysis, regression analysis is performed with xi of the two-dimensional coordinate values as explanatory variables and yi as an objective variable, and a regression expression representing the relationship between xi and yi, particularly its slope a and y intercept b And the color component value of the unknown color in the interpolation target pixel is calculated using this regression equation. Specifically, the slope a and the y-intercept b of the straight line y = a × x + b are obtained by the calculation shown in the following formula (14) in step ST43.

  In step ST44, the B component value b33 in the interpolation processing target pixel G33 is calculated by the following equation (15) using the straight line y = a × x + b and the G component value G33 in the interpolation processing target pixel G33, and the interpolation processing target pixel G33 is calculated. Is written in the frame memory 10 in association with the position of.

b33 = a × G33 + b
... (15)

  The above is the process after step ST32B. The above processing is performed for all odd-numbered G pixels, and the B component values in the odd-numbered G pixels are obtained by interpolation. As a result, the R component value and the B component value are obtained for all the odd-numbered G pixels.

  Next, the interpolation processing target pixel G33 is in an even-numbered row, and the interpolation of the R component value and the B component value in the interpolation processing target pixel G33 will be described with reference to FIG.

As is apparent from a comparison of FIG. 13 with FIG. 12, in FIG. 13 (when viewed from the center of even-numbered G pixels) and FIG. 12 (when viewed from the center of odd-numbered G pixels), R pixels And B pixels (and their component values) are simply swapped. Therefore, the interpolation of the R component value and the B component value in the G pixel in the even row is performed in the same manner except that the interpolation of the R component value and the B component value in the G pixel in the odd row described above is replaced with R and B. be able to.
Therefore, the interpolation of the R component value in the G pixels in the even rows may be performed by replacing B with R in the above description of the interpolation of the B component values in the G pixels in the odd rows. Similarly, the interpolation of the B component value in the G pixels in the even rows may be performed by replacing R with B in the above description of the interpolation of the R component values in the G pixels in the odd rows.
When the R component value and the B component value are interpolated for all the G pixels by the above interpolation processing, the color component values of each pixel written in the frame memory 10 are as shown in FIG. Capital letters R, G, and B in FIG. 14 represent R component values, G component values, and B component values given by the solid-state imaging device 2 among the color component values of each pixel, and small letters r, g, and b R component value, G component value, and B component value obtained by the interpolation processing are represented.

  Next, the interpolation of the R component value in the B pixel in the above (P3) will be described with reference to FIGS.

FIG. 5 is a flowchart showing the operation of the processor 11 when the R component value is interpolated in the B pixel. FIG. 15 schematically shows the color component values of the pixels used in the interpolation of the R component value in the B pixel B33 among the color component values of each pixel written in the frame memory 10. . The symbol at each pixel position represents the color component value possessed by each pixel, but the symbol beginning with a capital letter R, G, B is the R component originally possessed by each pixel (that is, given by the solid-state imaging device 2). Represents a value, G component value, and B component value, and symbols beginning with lowercase letters r, g, and b are R component values, G component values, B obtained by interpolation of (P1) and (P2) above. Represents a component value.
Similarly to FIG. 8, symbols starting with capital letters R, G, and B are also used to represent each pixel.

The color component value of each pixel used in the processing of each step described below is read from the frame memory 10 as necessary.
First, in steps ST20, ST21, ST22A, and ST22B, the color component value (R component value) of the same color as the color component (R component) to be interpolated in the direction in which the image correlation is strong around the interpolation processing target pixel B33. A determination is made whether there is a pixel having.
First, in step ST20 and step ST21, it is determined whether the correlation is strong in the horizontal direction or the vertical direction around the interpolation processing target pixel B33. For this determination, pixels around the interpolation target pixel B33 are used, and values dV and dH represented by the following equation (16) are calculated (step ST20), and it is determined whether dH <dV. (Step ST21).

dH = | G23-G43 |
dV = | G32-G34 |
... (16)

  Here, dH is the G component value of the pixel in the horizontal direction centering on the interpolation processing target pixel B33, that is, two pixels G23 aligned in the horizontal direction with respect to the interpolation processing target pixel B33 and adjacent to the interpolation processing target pixel B33. , G43 represents the absolute value of the difference between the G component values of G43, and dV is the G component value of the pixel in the vertical direction centered on the interpolation processing target pixel B33, that is, is aligned in the vertical direction with respect to the interpolation processing target pixel B33. This represents the absolute value of the difference between the G component values of the two pixels G32 and G34 adjacent to the processing target pixel. When the correlation in the horizontal direction is strong, dH <dV, and when the correlation in the vertical direction is strong, dV <dH. Note that the equations for obtaining dH and dV are not limited to Equation (16).

  Depending on the magnitude relationship between dV and dH, the process after step ST22A or the process after step ST22B is performed. That is, when dH <dV (when it is determined that the correlation in the horizontal direction is strong), the processing after step ST22A is performed, and otherwise (when it is determined that the correlation in the vertical direction is strong), step ST22B. The following processing is performed.

Next, in step ST22A, it is determined whether or not there is a pixel having an R component value in the horizontal direction (the direction in which the correlation is determined to be strong in step ST21) around the interpolation target pixel B33.
When it is determined that it exists, the process proceeds to step ST23A, and the first interpolation is executed. If it is determined that it does not exist, the process proceeds to step ST24A, where the second interpolation is executed. As is apparent from FIG. 15, since there is a pixel having an R component value in the horizontal direction centering on the B pixel, the process proceeds to step ST23A and the first interpolation is executed.

  In step ST23A, the G component value g33 of the interpolation processing target pixel B33 and the G component values G23 and G43 of the G pixels G23 and G43 aligned in the horizontal direction with respect to the interpolation processing target pixel B33 and the R component values r23 and r43 are used. The interpolation calculation for obtaining the R component value r33 in the interpolation target pixel B33 is performed by the following equation (17). This interpolation calculation is also an interpolation calculation “assuming a similar relationship”, similar to the equation (2).

  The R component value r33 in the interpolation processing target pixel B33 obtained by the above equation (17) is also written in the frame memory 10 in association with the position of the interpolation processing target pixel B33.

The above is the process after step ST22A. Next, the processing after step ST22B will be described.
In step ST22B, it is determined whether or not there is a pixel having an R component value in the vertical direction (the direction in which the correlation is determined to be strong in step ST21) around the interpolation target pixel B33.
When it is determined that it exists, the process proceeds to step ST23B, and the first interpolation is executed. If it is determined that it does not exist, the process proceeds to step ST24B, where the second interpolation is executed. As is apparent from FIG. 15, since there is a pixel having an R component value in the vertical direction centering on the B pixel, the process proceeds to step ST23B, and the first interpolation is executed.

  In step ST23B, the G component value g33 of the interpolation processing target pixel B33 and the G component values G32 and G34 of the G pixels G32 and G34 aligned in the vertical direction with respect to the interpolation processing target pixel B33 and the R component values r32 and r34 are used. The interpolation calculation for obtaining the R component value r33 in the interpolation target pixel B33 is performed by the following equation (18). This interpolation calculation is also an interpolation calculation “assuming a similar relationship”, similar to the equation (2).

  The R component value r33 in the interpolation processing target pixel B33 obtained by the above equation (18) is also written in the frame memory 10 in association with the position of the interpolation processing target pixel G33.

The above processing is performed for all the B pixels, and the R component value in the B pixel is obtained by interpolation.
Next, the interpolation of the B component value in the R pixel in the above (P4) will be described.

  FIG. 16 schematically shows the color component values of the pixels used in the interpolation of the B component value in the R pixel R33 among the color component values of each pixel written in the frame memory 10. Among the symbols at each pixel position, symbols beginning with capital letters R, G, and B are symbols representing the R component value, G component value, and B component value that each pixel originally has (that is, given by the solid-state imaging device 2). Symbols beginning with small letters r, g, and b represent R component values, G component values, and B component values obtained by the interpolation of (P1) and (P2) above. Similarly to FIG. 8, symbols starting with capital letters R, G, and B are also used to represent each pixel.

  As is clear from comparison of FIG. 16 with FIG. 15, the pixel arrangement and the distribution of the color components of both are merely interchanged of the R component and the B component. That is, FIGS. 16 and 15 are the same except that R and B and r and b are interchanged.

  Therefore, the interpolation of the B component value in the R pixel is performed in the same manner except that the interpolation of the R component value in the B pixel described above and R and B and r and b are interchanged.

Thus, all the interpolation processes are completed, and the R component value, the G component value, and the B component value are written in the frame memory 10 for each pixel as shown in FIG. Therefore, a color image can be obtained by outputting the data written in the frame memory 10.
In FIG. 17, capital letters R, G, and B represent R component values, G component values, and B component values given by the solid-state imaging device 2, and small letters r, g, and b represent R components given by interpolation processing. Value, G component value, and B component value.

  The effects of the above-described pixel signal processing method are as follows.

  First, the first interpolation method is interpolation assuming that there is a correlation between changes between different color component values.

FIG. 18 shows the first interpolation performed in step ST13A when it is determined in step ST11 of FIG. 4 that the horizontal correlation is strong during the interpolation of the G component value in the R pixel in (P1) above. It is a figure for demonstrating. In FIG. 18, a curve obtained by plotting the change in the R component value along the horizontal direction around the interpolation target pixel is represented by L1, and a curve obtained by estimating the change in the G component value is represented by L2. Further, in FIG. 18, the H axis corresponds to the horizontal axis on the imaging surface of the image sensor 2, and the Z axis represents the color component value of each pixel.
R1 and R2 are such that the position on the H axis corresponds to the position of the pixels R13 and R53 on the horizontal axis of the imaging surface, and the position on the Z axis corresponds to the average value of the R component values R13 and R53. Yes, R0 is a point where the position on the H axis corresponds to the position of the interpolation processing target pixel R33 on the horizontal axis of the imaging surface, and the position on the Z axis corresponds to the R component value R33.
G1 and G2 are such that the position on the H axis corresponds to the position of the pixels G23 and G43 on the horizontal axis of the imaging surface, and the position on the Z axis corresponds to the average value of the G component values G23 and G43. Yes, G0 is that the position on the H-axis corresponds to the position of the interpolation processing target pixel R33 on the horizontal axis of the imaging surface, and the position on the Z-axis corresponds to the G component value g33 obtained by interpolation. is there.
In the calculation of Expression (2) performed in step ST13A, the G component value g33 is determined so that the triangle R0R1R2 and the triangle G0G1G2 are similar.

In other words, the average value Rav of the R component values R13 and R53 of the pixels R13 and R53 having two R component values (color component values of known colors) around the interpolation processing target pixel R33, and the interpolation processing target pixel R33. The difference (R33−Rav) from the R component value R33 and two G component values around the interpolation processing target pixel R33 (color component values of the same color as the color component value of the insufficient color to be interpolated in the interpolation processing target pixel) ) Of the G component values G23 and G43 of the pixels G23 and G43 having a difference between the G component value g33 in the interpolation processing target pixel R33 obtained by interpolation, and the interpolation processing target Pixels having the two R component values around the pixel R33 (two pixels having color component values R13 and R53 used for calculating the average value Rav of the R component values) R13, A pixel having a distance on the H axis between 53 (distance on the HV plane) Dk and the two G component values around the interpolation target pixel R33 (used for calculating the average value of the G component values) The two pixels having the color component values G23 and G43) G23 and G43 in the interpolation processing target pixel R33 so that the ratio between the distance on the H axis (distance on the HV plane) Du is equal to each other. A G component value g33 is determined.
When the above relationship is expressed by a mathematical formula,

as mentioned above,
Gav is an average value of the color component values G23 and G43 (Gav = (G23 + G43) / 2),
Rav is an average value of the color component values R13 and R53 (Rav = (R13 + R53) / 2)
It is. Therefore,

  g33 = Gav + (Du / Dk) (R33-Rav)

If Du / Dk = f,
g33 = Gav + f (R33-Rav)

When the pixels are evenly arranged, the distance between adjacent pixels is represented by d.
f = Du / Dk = d / 2d = 1/2
Given in.

  In the above description, the case where the correlation is particularly strong in the horizontal direction in the interpolation of the G component value in the R pixel has been described as an example. However, the same description can be made when the correlation is strong in the vertical direction. The same explanation can be made by interpolation of G component values in B pixels.

  FIG. 19 shows that the correlation in the vertical direction is strong in step ST21 of FIG. 5 in the above-described (P2) interpolation of the R component value in the G pixel, particularly in the interpolation of the R component in the odd row G pixel. It is a figure for demonstrating the 1st interpolation performed by step ST23B. In FIG. 19, a curve obtained by plotting the change in the G component value along the vertical direction centering on the pixel to be interpolated is represented by L2, and a curve obtained by estimating the change in the R component value is represented by L1.

Further, in FIG. 19, the V axis corresponds to the vertical axis on the imaging surface of the image sensor 2, and the Z axis represents the color component value of each pixel.
G1 and G2 are such that the position on the V-axis corresponds to the position of the pixels R32 and R34 on the vertical axis of the imaging surface, and the position on the Z-axis corresponds to the average value of the G component values g32 and g34. Yes, G0 is a point where the position on the V-axis corresponds to the position of the interpolation processing target pixel G33 on the vertical axis of the imaging surface, and the position on the Z-axis corresponds to the G component value G33.
R1 and R2 are such that the position on the V-axis corresponds to the position of the pixels R32 and R34 on the vertical axis of the imaging surface, and the position on the Z-axis corresponds to the average value of the R component values R32 and R34. R0 is a point in which the position on the V-axis corresponds to the position of the interpolation target pixel G33 on the vertical axis of the imaging surface, and the position on the Z-axis corresponds to the R component value r33 obtained by interpolation. is there.

  The calculation of equation (9) performed in step ST23B determines the R component value r33 so that the triangle G0G1G2 and the triangle R0R1R2 are similar.

In other words, the average value Gav of the G component values g32 and g34 of the pixels R32 and R34 having two G component values (color component values of known colors) around the interpolation processing target pixel G33, and the interpolation processing target pixel G33. The difference (G33−Gav) from the G component value G33 and the two R component values around the interpolation processing target pixel G33 (the color component value of the same color as the color component value of the insufficient color to be interpolated in the interpolation processing target pixel) ) Of the R component values R32 and R34 of the pixels R32 and R34 having a difference between the R component value r33 in the interpolation processing target pixel G33 obtained by interpolation and the interpolation processing target Pixels having the two G component values around the pixel G33 (two pixels having the G component values g32 and g34 used for calculating the average value Gav of the G component values) R32, A pixel having a distance on the V axis (distance on the HV plane) Dk 34 and a pixel having the two R component values around the interpolation target pixel G33 (used for calculating an average value of the R component values). The two pixels having the color component values R32 and R34) R32 and R34 in the interpolation processing target pixel G33 so that the ratio of the distance on the V axis (distance on the HV plane) Du is equal to each other. An R component value r33 is determined.
When the above relationship is expressed by a mathematical formula,

as mentioned above,
Rav is an average value of color component values R32 and R34 (Rav = (R32 + R34) / 2),
Gav is an average value of the color component values g32 and g43 (Gav = (g32 + g34) / 2),
It is. Therefore,

  r33 = Rav + (Du / Dk) (G33-Gav)

If Du / Dk = f,
r33 = Rav + f (G33-Gav)

When the pixels are evenly arranged, the distance between adjacent pixels is represented by d.
f = Du / Dk = d / d = 1
Given in.

  The above is an example in which the correlation in the vertical direction is strong in the interpolation of the R component values in the odd-numbered G pixels, but the case in which the correlation in the horizontal direction is strong in the interpolation of the R component values in the even-numbered G pixels. The same explanation can be given. The same explanation can be made for interpolation of the B component value in the G pixel, interpolation of the B component value in the R pixel, and interpolation of the R component value in the B pixel.

If the above relationship is generalized, in the interpolation calculation by the first interpolation,
The color component value of the known color in the interpolation target pixel is set as the J color component value JJ,
Jav is an average value of color component values of known colors of pixels around the interpolation target pixel,
The color component value of the unknown color in the interpolation target pixel, that is, the color component value to be interpolated is set as the K color component value KK,
Assuming that the average value of the color component values of unknown colors of pixels around the interpolation target pixel is Kav,
KK is given by the following equation (19).

KK = Kav + f (JJ−Jav) (19)
However, f is a constant representing the ratio between the inter-pixel distances as described above.

  If interpolation is performed with the value of f set as described above, the change in J color and the change in K color are similar. According to this interpolation method, the changes in the two color component values are identical or similar. Therefore, when the two color component values change in the same way, for example, the interpolation is excellent in an area where the hue is the same and the density changes.

  Next, the effect of the second interpolation method will be described with reference to FIGS.

  Prior to the description, calculation of Expression (7) will be described.

  There are a set of two variables (xi, yi), (i = 1 to M), and when a straight line representing the relationship between the two variables xi, yi is obtained by the least square method, the slope a and y intercept b of the straight line are: Is given by equation (20).

  Expression (7) corresponds to the case where M = 6 in Expression (20).

  Obtaining the variable yi as a function of the variable xi using a and b thus obtained is called regression analysis. That is, in the second interpolation method, interpolation is performed by regression analysis. In the regression analysis, the variable yi obtained as a function is called an objective variable, and the other variable xi is called an explanatory variable.

  FIG. 20 is a diagram schematically showing a state near the boundary between the color represented by the region A and the color represented by the region B. Further, in the Bayer array interpolation, it is a diagram when the interpolation of the G component values in the R pixel and the B pixel of (P1) is completed. (That is, the R component value and the B component value in the G pixel of (P2) are then interpolated.)

  In the region along the boundary extending in the vertical direction as shown in FIG. 20, since the correlation is clearly strong in the vertical direction, the insufficient color component value for each pixel is the color of the pixel existing in the vertical direction centering on the interpolation target pixel. It is desirable to interpolate according to the first interpolation method using the component values. However, due to restrictions on the pixel arrangement and the color component value of each pixel, when interpolating the R component value in the pixel G33, there are pixels that have the R component value in the vertical direction centering on the interpolation target pixel G33. Therefore, the interpolation process cannot be performed by the first interpolation method. In this embodiment, in such a case, the interpolation processing by the second interpolation method is performed.

  FIG. 21 is a diagram for explaining the second interpolation method. FIG. 21 illustrates an example in which R component values are interpolated for G pixels. The horizontal axis represents the G component value of each pixel, and the vertical axis represents the R component value of each pixel. In FIG. 21, three points included in the area A are represented by two-dimensional coordinate data (g21, R21), (g23, R23), (g25, R25) using the color component values written in FIG. The three points included in the region B are represented by two-dimensional coordinate data (g41, R41), (g43, R43), (g45, R45) using the color component values written in FIG. It is a point to be done. Further, a straight line representing a color change from the region A to the region B is drawn as L3.

  In the second interpolation method, the straight line L3 is obtained by Expression (20), and interpolation is performed assuming that the insufficient color component value r33 in the interpolation processing target pixel G33 is located on the straight line L3. By performing interpolation in this way, even in an area where the image changes from one color to another color, the color change is represented by a straight line L3, and the missing color component value is obtained on the straight line. Interpolation corresponding to the color change can be performed. Therefore, it can be said that the second interpolation method is a method suitable for interpolation at a color boundary.

  20 and 21 have been described with reference to the G component value and the R component value, but the same description can be made for any combination of color component values.

  In this embodiment, the directions having strong correlation (similar colors) are determined by comparing dH and dV in steps ST11, ST21, and ST31, and steps ST12A, ST12B, ST22A, ST22B, ST32A, and ST32B are determined. When there is a pixel having the same color component value as the color component value to be interpolated in that direction, the first interpolation method is selected. Otherwise, the second interpolation method is selected. As described above, the interpolation method is adaptively selected. By selecting the interpolation method in this way, near the boundary between one color and another color, the color of the same color as the color to be interpolated in the direction of strong correlation due to the restriction of the color component value of the pixel near the interpolation target pixel When there is no pixel having a component value and interpolation processing by the first interpolation method cannot be performed, interpolation processing by the second interpolation method suitable for interpolation at the color boundary can be performed. . That is, it is possible to always select an appropriate interpolation method according to the color component values of pixels in the vicinity of the interpolation target pixel, and to obtain an image without a false color.

  For the calculation of equation (20)

  That is, the division including the divisor of the integrated value of the square of the difference between the explanatory variable xi and the average value is included. In general, division is less accurate when the divisor is close to 0, which leads to an increase in error.

  In equation (20), the divisor is close to 0 when all xi are close to a certain value xo. Whether all xi are close to a certain value xo can be determined by whether or not the difference between the maximum value and the minimum value of xi is equal to or smaller than a predetermined threshold value.

  Therefore, in ST41, it is determined whether or not the difference between the maximum value and the minimum value of the explanatory variable xi is equal to or smaller than a predetermined threshold value. If the difference between the maximum value and the minimum value is equal to or smaller than the predetermined threshold value, interpolation processing without linear regression analysis (linear interpolation) ) Can prevent an increase in error.

Embodiment 2. FIG.
FIG. 22 is a block diagram showing the configuration of the pixel signal processing circuit 4 according to the second embodiment of the present invention. The pixel signal processing circuit 4 can be used in place of the pixel signal processing circuit 4 in FIG. 2 as a part of the imaging apparatus shown in FIG. 1, for example.
The pixel signal processing circuit shown in FIG. 22 has a function similar to that of the pixel signal processing circuit of FIG. 2, but instead of the frame memory 10 and the processor 11, a frame memory 20, a correlation determination circuit 21, and a presence / absence determination circuit. 22 is different in that an interpolation calculation circuit 23 and an interpolation calculation circuit 24 are provided.

The frame memory 20 has the same function as the frame memory 10 of FIG.
The correlation discriminating circuit 21 discriminates a direction having a strong correlation around the interpolation processing target pixel and outputs the result to the presence / absence judging circuit 22.
The presence / absence determination circuit 22 determines whether the interpolation calculation circuit 23 or the interpolation calculation circuit 24 performs the interpolation process on the interpolation target pixel according to a method described later.
Whether or not there is a pixel having a color component value of the same color as the color component value to be interpolated in the direction in which the correlation of the image is strong around the pixel to be interpolated by the correlation determination circuit 21 and the presence / absence determination circuit 22 An image feature determination circuit 25 that performs the determination is configured.
The interpolation calculation circuit 23 and the interpolation calculation circuit 24 each perform an interpolation process by a method described later. Then, the color component value obtained by interpolation by either the interpolation calculation circuit 23 or the interpolation calculation circuit 24 is written in the frame memory 20 in association with the position of the interpolation processing target pixel.

  The correlation determination circuit 21 has the same function as steps ST11, ST21, ST31, the presence / absence determination circuit 22 has the same function as steps ST12A, ST12B, ST22A, ST22B, ST32A, ST32B, and the interpolation calculation circuit 23 has the same function. Steps ST13A, ST13B, ST23A, ST23B, ST33A, ST33B have the same functions, and the interpolation calculation circuit 24 has the same functions as steps ST14A, ST14B, ST24A, ST24B, ST34A, ST34B.

In the following, the interpolation process of the insufficient color component value performed by the pixel signal processing circuit 4 of the second embodiment will be described. For interpolation of missing color component values,
(P1) Processing for obtaining G component values in the R pixel and B pixel,
(P2) Processing for obtaining the R component value and the B component value in the G pixel,
(P3) Processing for obtaining the R component value in the B pixel and (P4) processing for obtaining the B component value in the R pixel are included, and are performed, for example, in the order described above.

  First, the interpolation processing of the G component value in the R pixel in (P1) will be described with reference to FIG. FIG. 8 schematically shows the color component values of pixels used for interpolation of the G component value in the R pixel (R33) among the color component values of each pixel written in the frame memory 20. The meaning of each symbol is the same as in the first embodiment.

  First, the correlation discriminating circuit 21 reads G component values G23, G43, G32, and G34 of the G pixels G23, G43, G32, and G34 from the frame memory 20, and calculates dH and dV by the following equation (21). .

dH = | G23-G43 |
dV = | G32-G34 |
... (21)

  Next, in accordance with the magnitude relationship between dH and dV, a direction having a strong correlation with the interpolation target pixel as a center is determined, and a signal indicating the result (a determination result signal) is output to the presence / absence determination circuit 22. That is, the discrimination result signal output from the correlation discriminating circuit 21 indicates that the horizontal correlation is strong when dH <dV, and otherwise indicates that the vertical correlation is strong. . Note that the equations for obtaining dH and dV are not limited to Equation (21).

  The presence / absence determination circuit 22 examines the color component value stored in the frame memory 20 and determines whether or not there is a pixel having a G component value in a direction having a strong correlation around the interpolation processing target pixel R33. Then, when there is a pixel having a G component value in the direction of strong correlation around the interpolation processing target pixel R33, the interpolation calculation circuit 23 performs interpolation.

First, interpolation when the correlation determination circuit 21 determines that the correlation is strong in the horizontal direction will be described.
In the example shown in FIG. 8, since there is a pixel having a G component value in the horizontal direction centering on the R pixel R33, the presence / absence determination circuit 22 causes the interpolation calculation circuit 23 to perform interpolation.

  The interpolation calculation circuit 23 reads out the G color component values G23 and G43 of the G pixels G23 and G43 and the R color component values R33, R13 and R53 of the R pixels R33, R13 and R53 from the frame memory 20, and uses the following equation (22): ) To calculate the G component value g33 in the interpolation processing target pixel R33.

  The G component value g33 calculated by Expression (22) is output from the interpolation calculation circuit 23, and is written in the frame memory 20 in association with the position of the interpolation processing target pixel R33.

  The above is the interpolation processing when the correlation determination circuit 21 determines that the correlation is strong in the horizontal direction. Next, the operation when the correlation determination circuit 21 determines that the correlation is strong in the vertical direction will be described.

  In the example shown in FIG. 8, since there is a pixel having a G component value in the vertical direction centered on the R pixel R33, the presence / absence determination circuit 22 causes the interpolation calculation circuit 23 to perform interpolation.

  The interpolation calculation circuit 23 reads the G color component values G32, G34 of the G pixels G32, G34 and the R component values R33, R31, R35 of the R pixels R33, R31, R35 from the frame memory 20, and uses the following equation (23). The G component value g33 in the interpolation processing target pixel R33 is interpolated.

  Then, the G component value g33 obtained by Expression (23) is output from the interpolation calculation circuit 23, and is written in the frame memory 20 in association with the position of the interpolation processing target pixel R33.

  The above is the interpolation processing when the correlation determination circuit 21 determines that the correlation is strong in the vertical direction. The above interpolation processing is executed for all R pixels, and the G component value at the R pixel is obtained by interpolation. Next, interpolation of the G component value in the B pixel will be described.

  FIG. 9 schematically shows the color component values of the pixels used for the interpolation of the G component value in the B pixel (B33) among the color component values of each pixel written in the frame memory 20. The meaning of each symbol is the same as in the first embodiment. As is apparent from a comparison of FIG. 9 with FIG. 8, in FIG. 8 and FIG. 9, only the R pixel, the B pixel, and the color component values of each pixel are interchanged. Therefore, the interpolation of the G component value in the B pixel can be performed in the same manner except that R and B of the interpolation of the G component value in the R pixel described above are replaced.

  When the G component values in the R pixel and the B pixel are interpolated by the above interpolation processing, the color component values of each pixel written in the frame memory 20 are schematically shown in FIG. Note that capital letters R, G, and B in FIG. 10 represent R component values, G component values, and B component values given by the solid-state imaging device 2 among the color component values of each pixel, and a small letter g represents interpolation processing. Represents the G component value obtained by.

  Next, the interpolation processing of the R component value and the B component value in the G pixel in (P2) will be described. In the Bayer array, as described above with reference to FIGS. 3 and 11, interpolation is performed depending on whether the G pixel to be subjected to interpolation processing belongs to an odd row or an even row. The colors of the pixels aligned in the horizontal direction and the vertical direction differ from the processing target pixel.

FIG. 12 shows the color component values of the pixels used for the interpolation of the R component value and the B component value in the G pixel G33 in the odd row among the color component values of each pixel written in the frame memory 20. This is schematically shown, and the meaning of each symbol is the same as in the first embodiment.
First, the interpolation of R component values in odd-numbered G pixels will be described.

  First, the correlation discriminating circuit 21 reads the G component values g23 and g43 of the B pixels B23 and B43 and the G component values g32 and g34 of the R pixels R32 and R34 from the frame memory 20, and dH and dV according to the following equation (24). Calculate

dH = | g23−g43 |
dV = | g32−g34 |
... (24)

  Next, in accordance with the magnitude relationship between dH and dV, a direction having a strong correlation with the interpolation target pixel as a center is determined, and a signal indicating the result (a determination result signal) is output to the presence / absence determination circuit 22. That is, the discrimination result signal output from the correlation discriminating circuit 21 indicates that the horizontal correlation is strong when dH <dV, and otherwise indicates that the vertical correlation is strong. . Note that the equations for obtaining dH and dV are not limited to Equation (24).

  The presence / absence determination circuit 22 examines the color component value stored in the frame memory 20 and determines whether or not there is a pixel having an R component value in a direction having a strong correlation around the interpolation processing target pixel G33. Then, if there is a pixel having an R component value in the direction of strong correlation with the interpolation processing target pixel G33 as the center, the interpolation calculation circuit 23 causes the interpolation calculation circuit 24 to perform interpolation.

  First, interpolation when the correlation determination circuit 21 determines that the correlation is strong in the horizontal direction will be described.

  In the example shown in FIG. 12, since there is no pixel having the R component value in the horizontal direction centered on the G pixel G33, the presence / absence determination circuit 22 causes the interpolation calculation circuit 24 to perform interpolation.

Next, an interpolation method performed by the interpolation calculation circuit 24 will be described with reference to FIGS. FIG. 23 is a block diagram showing the internal configuration of the interpolation calculation circuit 24 of FIG. 22. The interpolation processing calculation circuit 24 includes a memory 30 that can store color component values of a plurality of pixels, and color components of the plurality of pixels. A maximum value determination circuit 31 that can determine the maximum value of the values, a minimum value determination circuit 32 that can determine the minimum value of the color component values of a plurality of pixels, and whether the difference between the two numerical values is equal to or less than a predetermined threshold value. The comparison circuit 33 includes a linear interpolation circuit 35 and a regression analysis interpolation circuit 36 that perform interpolation processing to be described later, and an interpolation selection circuit 34.
As will be described later, the interpolation selection circuit 34 causes one of the linear interpolation circuit 35 and the regression analysis interpolation circuit 36 to selectively perform the interpolation.
The interpolation selection circuit 34 has the same function as step ST41, the linear interpolation circuit 35 has the same function as step ST42, and the regression analysis interpolation circuit 36 has the same function as step ST43 and step ST44.

First, by the interpolation calculation circuit 24, the G component value g12 and the R component value R12 of the R pixel R12 from the frame memory 20 are displayed.
G component value g32 and R component value R32 of the R pixel R32,
G component value g52 and R component value R52 of R pixel R52,
G component value g14 and R component value R14 of R pixel R14,
G component value g34 and R component value R34 of the R pixel R34,
The G component value g54 and R component value R54 of the R pixel R54 and the G component value G33 of the interpolation target pixel G33 are read out. Then, the two-dimensional coordinate data represented by the following equation (25) and the G component value G33 of the interpolation target pixel G33 are written in the memory 30.

(X1, y1) = (g12, R12)
(X2, y2) = (g32, R32)
(X3, y3) = (g52, R52)
(X4, y4) = (g14, R14)
(X5, y5) = (g34, R34)
(X6, y6) = (g54, R54)
... (25)

  Next, the maximum value determination circuit 31 calculates the maximum value of xi (i = 1 to 6), and the minimum value determination circuit 32 determines the minimum value of xi (i = 1 to 6), which are output to the comparison circuit 33, respectively. Is done. The comparison circuit 33 checks whether the difference between the maximum value and the minimum value is equal to or less than a predetermined threshold value, and outputs the result to the interpolation selection circuit 34.

  The interpolation selection circuit 34 causes one of the linear interpolation circuit 35 and the regression analysis interpolation circuit 36 to perform an interpolation process according to the input from the comparison circuit 33. When the difference between the maximum value and the minimum value is equal to or less than a predetermined threshold value, interpolation processing is performed by the linear interpolation circuit 35. Otherwise, the regression analysis interpolation circuit 36 performs interpolation processing.

  The linear interpolation circuit 35 calculates the R component value r33 in the interpolation target pixel G33 by linear interpolation. That is, the R component value R32 of the R pixel R32 and the R component value R34 of the R pixel R34 are read from the two-dimensional coordinate data written in the memory 30, and the R component in the interpolation processing target pixel G33 is expressed by the following equation (26). The value r33 is calculated. Then, the value is output from the interpolation calculation circuit 24 to the frame memory 20, and is written in the frame memory 20 in association with the position of the interpolation processing target pixel G33.

  On the other hand, in the regression line analysis circuit 36, the slope a and y intercept b of the straight line y = a × x + b are obtained from the two-dimensional coordinate data represented by the following equation (25) and the equation represented by the following equation (27).

  Next, the regression analysis interpolation circuit 36 reads the G component value G33 in the interpolation processing target pixel G33 from the memory 30, and performs the interpolation processing target by the following equation (28) using the straight line y = a × x + b and the G component value G33. The R component value r33 in the pixel G33 is calculated. Then, the R component value r33 obtained by Expression (28) is output from the interpolation calculation circuit 24 to the frame memory 20, and is written in the frame memory 20 in association with the position of the interpolation processing target pixel G33.

r33 = a × G33 + b
... (28)

  The above is the interpolation processing when the correlation is strong in the horizontal direction around the interpolation processing target pixel G33. Next, the interpolation processing when the correlation is strong in the vertical direction centering on the interpolation processing target pixel G33 will be described.

  As apparent from FIG. 12, since there is a pixel having an R component value in the vertical direction centering on the G pixel, the presence / absence determination circuit 22 causes the interpolation calculation circuit 23 to perform interpolation. The interpolation calculation circuit 23 reads the R component values R32 and R34 of the R pixels R32 and R34 and the G component values g32 and g34 and the G component value G33 of the interpolation processing target pixel G33 from the frame memory 20, and the following equation (29) The R component value r33 in the interpolation processing target pixel G33 is calculated and output to the frame memory 20. The frame memory 20 writes the R color component value r33 in association with the position of the interpolation processing target pixel G33.

  The above is the interpolation of the R component value in the odd-numbered G pixels. The above processing is performed for all odd-numbered G pixels, and R component values for odd-numbered G pixels are obtained by interpolation. Next, interpolation of B component values in odd-numbered G pixels will be described with reference to FIGS.

  First, the correlation determination circuit 21 reads the G component values g23 and g43 of the B pixels B23 and B43 and the G component values g32 and g34 of the R pixels R32 and R34 from the frame memory 20, and is expressed by the following equation (30). dH and dV are calculated.

dH = | g23−g43 |
dV = | g32−g34 |
... (30)

  Next, in accordance with the magnitude relationship between dH and dV, a direction having a strong correlation is determined around the interpolation processing target pixel G33, and a signal indicating the result (a determination result signal) is output to the presence / absence determination circuit 22. That is, the discrimination result signal output from the correlation discriminating circuit 21 indicates that the horizontal correlation is strong when dH <dV, and otherwise indicates that the vertical correlation is strong. . Note that the equations for obtaining dH and dV are not limited to Equation (30).

  The presence / absence determination circuit 22 examines the color component value stored in the frame memory 20 and determines whether or not there is a pixel having a B component value in a direction having a strong correlation around the interpolation processing target pixel G33. Then, when there is a pixel having a B component value in the direction of strong correlation around the interpolation processing target pixel G33, the interpolation calculation circuit 23 performs interpolation.

First, interpolation when the correlation determination circuit 21 determines that the correlation is strong in the horizontal direction will be described.
In the example shown in FIG. 12, there is a pixel having a B component value in the horizontal direction centering on the G pixel. In this case, the interpolation calculation circuit 23 performs interpolation.

  The interpolation calculation circuit 23 reads the B component values B23 and B43 of the B pixels B23 and B43, the G component values g23 and g43, and the G component value G33 of the interpolation processing target pixel G33 from the frame memory 20, and the following equation (31) Thus, the B component value b33 in the interpolation processing target pixel G33 is calculated and output to the frame memory 20. Then, the B component value b33 obtained by the interpolation is written in the frame memory 20 in association with the position of the interpolation processing target pixel G33.

  The above is the interpolation processing when the correlation determination circuit 21 determines that the correlation is strong in the horizontal direction. Next, an interpolation process when the correlation determination circuit 21 determines that the correlation is strong in the vertical direction will be described.

  First, the presence / absence determination circuit 22 checks whether or not there is a pixel having a B component value in a direction having a strong correlation (that is, the vertical direction) centering on the interpolation processing target pixel G33. As a result, the interpolation calculation circuit 23 and the interpolation calculation circuit 24 are checked. Interpolation processing is performed in either. As apparent from FIG. 12, there is no pixel having the B component value in the vertical direction centering on the G pixel, and in this case, the interpolation processing circuit 24 performs the interpolation processing.

  Next, the interpolation processing performed by the interpolation calculation circuit 24 will be described with reference to FIGS.

First, the G component value g21 and B component value B21 of the B pixel B21 from the frame memory 20 by the interpolation calculation circuit 24,
G component value g23 and B component value B23 of the B pixel B23,
G component value g25 and B component value B25 of the B pixel B25,
G component value g41 and B component value B41 of the B pixel B41,
G component value g43 and B component value B43 of the B pixel B43,
The G component value g45 and B component value B45 of the B pixel B45 and the G component value G33 of the interpolation target pixel G33 are read out. Then, the two-dimensional coordinate data represented by the following equation (32) and the G component value G33 of the interpolation target pixel G33 are written in the memory 30.

(X1, y1) = (g21, B21)
(X2, y2) = (g41, B41)
(X3, y3) = (g23, B23)
(X4, y4) = (g43, B43)
(X5, y5) = (g25, B25)
(X6, y6) = (g45, B45)
... (32)

  Next, the maximum value determination circuit 31 and the minimum value determination circuit 32 check the maximum and minimum values of xi (i = 1 to 6), respectively, and output them to the comparison circuit 33.

  The comparison circuit 33 checks whether the difference between the maximum value and the minimum value is equal to or less than the threshold value, and outputs the result to the interpolation selection circuit 34.

  The interpolation selection circuit 34 selects the interpolation process in the linear interpolation circuit 35 when the difference between the maximum value and the minimum value is equal to or less than the threshold value, and selects the interpolation process in the regression analysis interpolation circuit 36 otherwise.

  The linear interpolation circuit 35 calculates the B component value b33 in the interpolation target pixel G33 by linear interpolation. That is, the B component value B23 of the B pixel B23 and the B component value B43 of the B pixel B43 are read from the two-dimensional coordinate data written in the memory 30, and the B component in the interpolation processing target pixel G33 is obtained by the following equation (33). The value b33 is calculated, outputted from the interpolation calculation circuit 24 to the frame memory 20, and written in the frame memory 20 in association with the position of the interpolation processing target pixel G33.

  The regression analysis interpolation circuit 36 first obtains the slope a and the y-intercept b of the straight line y = a × x + b by the two-dimensional coordinate data represented by the equation (32) and the calculation represented by the following equation (34).

  Next, the regression analysis interpolation circuit 36 reads the G component value G33 in the interpolation processing target pixel G33 from the memory 30, and performs the interpolation processing target by the following equation (35) using the straight line y = a × x + b and the G component value G33. The B component value b33 in the pixel G33 is calculated. Then, the B component value b33 obtained by Expression (35) is output from the interpolation calculation circuit 24 to the frame memory 20, and is written in the frame memory 20 in association with the position of the interpolation processing target pixel G33.

b33 = a × G33 + b
... (35)

The above is the interpolation processing of the B component value in the G pixels in the odd rows when it is determined that the correlation is strong in the vertical direction. The above processing is performed for all odd-numbered G pixels, and the B component values in the odd-numbered G pixels are obtained by interpolation. As a result, the R component value and the B component value are obtained for all the odd-numbered G pixels. Next, interpolation of R component values and B component values in G pixels in even rows will be described with reference to FIG.
FIG. 13 shows the color component values of the pixels used in the interpolation of the R component value and the B component value in the G pixel G33 in the even row among the color component values of each pixel written in the frame memory 20. This is schematically shown, and the meaning of each symbol is the same as in the first embodiment.

As is apparent from a comparison of FIG. 13 with FIG. 12, in FIG. 13 (when viewed from the center of even-numbered G pixels) and FIG. 12 (when viewed from the center of odd-numbered G pixels), R pixels And B pixels (and their component values) are simply swapped. Therefore, the interpolation of the G component value in the B pixel can be performed in the same manner except that R and B of the interpolation of the G component value in the R pixel described above are replaced.
Therefore, the interpolation of the R component value in the G pixels in the even rows may be performed by replacing B with R in the above description of the interpolation of the B component values in the G pixels in the odd rows. Similarly, the interpolation of the B component value in the G pixels in the even rows may be performed by replacing R with B in the above description of the interpolation of the R component values in the G pixels in the odd rows.
When the R component value and the B component value are interpolated for all the G pixels by the above interpolation processing, the color component values of each pixel written in the frame memory 20 are as shown in FIG.
14 represent the R component value, the G component value, and the B component value given by the solid-state imaging device 2 among the color component values of each pixel, and the small letters r, g, b represents an R component value, a G component value, and a B component value obtained by the interpolation processing.

  Next, the interpolation of the R component value in the B pixel (P3) will be described with reference to FIGS.

  FIG. 15 schematically shows the color component values of the pixels used in the interpolation of the R component value in the B pixel B33 among the color component values of each pixel written in the frame memory 20, The meaning of the symbols is the same as in the first embodiment.

  First, the correlation determination circuit 21 reads G component values G23, G43, G32, and G34 of the G pixels G23, G43, G32, and G34 from the frame memory 20, and calculates dH and dV by the following equation (36). .

dH = | G23-G43 |
dV = | G32-G34 |
... (36)

  Next, in accordance with the magnitude relationship between dH and dV, a direction having a strong correlation is determined around the interpolation processing target pixel B33, and a signal indicating the result (a determination result signal) is output to the presence / absence determination circuit 22. That is, the discrimination result signal output from the correlation discriminating circuit 21 indicates that the horizontal correlation is strong when dH <dV, and otherwise indicates that the vertical correlation is strong. . Note that the equations for obtaining dH and dV are not limited to Equation (36).

  The presence / absence determination circuit 22 checks the color component value stored in the frame memory 20 and determines whether or not there is a pixel having an R component value in a direction having a strong correlation around the interpolation processing target pixel B33. Then, when there is a pixel having an R component value in the direction of strong correlation around the interpolation processing target pixel B33, the interpolation calculation circuit 23 performs interpolation.

First, an interpolation process when the correlation determination circuit 21 determines that the correlation is strong in the horizontal direction will be described.
In the example shown in FIG. 15, since there is a pixel having an R component value in the horizontal direction centering on the B pixel B <b> 33, the presence / absence determination circuit 22 causes the interpolation calculation circuit 23 to perform interpolation.

  The interpolation calculation circuit 23 reads the R component values r23 and r43 of the G pixels G23 and G43 and the G component values G23 and G43 and the G component value g33 of the interpolation processing target pixel B33 from the frame memory 20, and the following equation (37) Thus, the R component value r33 in the interpolation target pixel B33 is calculated and output to the frame memory 20. The output R component value r33 is written in the frame memory 20 in association with the position of the interpolation processing target pixel B33.

The above is the interpolation processing when the correlation determination circuit 21 determines that the correlation is strong in the horizontal direction.
Next, an interpolation process when the correlation determination circuit 21 determines that the correlation is strong in the vertical direction will be described.

In the example shown in FIG. 15, since there is a pixel having an R component value in the vertical direction centering on the interpolation processing target pixel B <b> 33, the presence / absence determination circuit 22 causes the interpolation calculation circuit 23 to perform interpolation processing.
The interpolation calculation circuit 23 reads the R component values r32 and r34 of the G pixels G32 and G34 and the G component values G32 and G34 and the G component value g33 of the interpolation processing target pixel B33 from the frame memory 20, and the following equation (38) Thus, the R component value r33 in the interpolation target pixel B33 is calculated.

  The R component value r33 calculated by Expression (38) is output from the interpolation calculation circuit 23, and is written in the frame memory 20 in association with the position of the interpolation processing target pixel R33.

The above processing is performed for all the B pixels, and the R component value in the B pixel is obtained by interpolation.
Next, the interpolation of the B component value in the R pixel in the above (P4) will be described.

FIG. 16 schematically shows the color component values of the pixels used when interpolating the B component value in the R pixel R33 among the color component values of each pixel written in the frame memory 20. The meaning of the symbols is the same as in the first embodiment.
As is clear from comparison of FIG. 16 with FIG. 15, the distribution of the color components is merely the replacement of the R component and the B component. That is, FIGS. 16 and 15 are the same except that R and B and r and b are interchanged.

  Therefore, the interpolation of the B component value in the R pixel is performed in the same manner except that the interpolation of the R component value in the B pixel described above and R and B and r and b are interchanged.

As described above, all the interpolation processes are completed, and the R component value, the G component value, and the B component value are written in the frame memory 20 for each pixel as shown in FIG. Therefore, a color image can be obtained by outputting the data written in the frame memory 20.
In FIG. 17, capital letters R, G, and B represent R component values, G component values, and B component values given by the solid-state imaging device 2, and small letters r, g, and b represent R values given by interpolation processing. Indicates a component value, a G component value, and a B component value.

  According to the second embodiment described above, interpolation processing equivalent to that in the first embodiment can be performed. Therefore, the same effect as in the first embodiment can be obtained.

  In the above description, FIG. 1 is used as the configuration of the imaging apparatus. However, the present invention is not limited to this, and various examples are conceivable. For example, a white balance circuit is added, or a function for recording the output from the pixel signal processing circuit 4 on a recording medium is added.

  Further, the method of creating the two-dimensional coordinate data for obtaining the regression line in the present invention is not limited to the one described in the first and second embodiments.

  Furthermore, the object of the interpolation processing in the first and second embodiments is not limited to the Bayer array, and it is sufficient that pixels of a plurality of color components are regularly arranged.

  The configuration of the pixel signal processing circuit 4 is not limited to that described in the first and second embodiments. For example, a configuration using a line memory having the required number of lines instead of the frame memory 10 and the frame memory 20 may be used. Conceivable.

It is a block diagram which shows the imaging device provided with the pixel signal processing circuit of Embodiment 1 or Embodiment 2 of this invention. It is a block diagram showing the internal structure of the pixel signal processing circuit of Embodiment 1 of this invention. It is a figure showing the Bayer arrangement | sequence of the color filter provided in the solid-state image sensor of the imaging device of FIG. 3 is a flowchart showing an interpolation process of the pixel signal processing circuit of FIG. 1. 3 is a flowchart showing an interpolation process of the pixel signal processing circuit of FIG. 1. 3 is a flowchart showing an interpolation process of the pixel signal processing circuit of FIG. 1. 6 is a flowchart showing processing performed by a second interpolation method of the interpolation processing of the pixel signal processing circuit of FIG. 1. It is a figure showing the pixel arrangement centering on R pixel and the color component value which each pixel has. It is a figure showing the pixel arrangement centering on B pixel and the color component value which each pixel has. It is a figure showing the color component value which each pixel arrangement | sequence in the state after the interpolation of the G component value in R pixel and B pixel finished. It is a figure which shows the Bayer arrangement centering on G pixel in an odd number row, and the Bayer arrangement centering on G pixel in an even number row. It is a figure showing the pixel arrangement centering on G pixel in an odd number row, and the color component value which each pixel has. It is a figure showing the pixel arrangement centering on G pixel in an even number line, and the color component value which each pixel has. It is a figure showing the color component value which each pixel arrangement | sequence in the state after the interpolation of the G component value in R pixel and B pixel, and the interpolation of the R component value and B component value in G pixel has ended. FIG. 15 is a diagram illustrating a pixel array centered on a B pixel and a color component value of each pixel in the same state as in FIG. 14. FIG. 15 is a diagram illustrating a pixel array centered on an R pixel and color component values of each pixel in the same state as in FIG. 14. Interpolation of G component value in R pixel and B pixel, interpolation of R component value and B component value in G pixel, interpolation of R component value in B pixel, and pixel arrangement in a state where interpolation of B component value in R pixel is finished FIG. 4 is a diagram illustrating color component values of pixels. It is a figure which shows the concept of the interpolation process in a 1st interpolation method. It is a figure which shows the concept of the interpolation process in a 1st interpolation method. It is a figure for demonstrating the concept of the interpolation process by regression analysis. It is a figure for demonstrating the concept of the interpolation process by regression analysis. It is a block diagram showing the internal structure of the pixel signal processing circuit of Embodiment 2 of this invention. FIG. 23 is a block diagram illustrating an internal configuration of an interpolation calculation circuit in FIG. 22.

Explanation of symbols

  4 pixel signal processing circuit, 10 frame memory, 11 processor, 20 frame memory, 21 correlation determination circuit (correlation determination means), 22 presence / absence determination circuit (means for selecting an interpolation method), 23 interpolation operation circuit (first interpolation means) ), 24 interpolation operation circuit (second interpolation means), 25 image feature determination circuit (image feature determination means), ST11 correlation determination step, ST12A step of determining presence / absence of color component value of insufficient color in strong correlation direction, ST13A First interpolation step, ST14A Second interpolation step, ST12B Step of determining presence / absence of color component value of insufficient color in strong correlation direction, ST13B First interpolation step, ST14B Second interpolation step, ST21 Correlation discrimination Step, ST22A in the direction of strong correlation ST23A first interpolation step, ST24A second interpolation step, ST22B step of determining presence / absence of insufficient color component value in a direction with strong correlation, ST23B first interpolation step Interpolation Step, ST24B Second Interpolation Step, ST31 Correlation Determination Step, ST32A Step for Determining Presence / Absence of Color Component Value of Insufficient Color in Strong Correlation, ST33A First Interpolation Step, ST34A Second Interpolation Step, ST32B Correlation Determining whether or not there is a color component value of an insufficient color in a strong direction, ST33B first interpolation step, ST34B second interpolation step.

Claims (20)

Each color component value of a plurality of pixels arranged on a two-dimensional plane and having at least one color component value among first to Nth (N is an integer of 2 or more) different color component values. The Kth color (K is any number from 1 to N, but different from J) of the interpolation target pixel having the Jth color component value (J is any number from 1 to N). In a pixel signal processing method for generating component values by interpolation,
An image feature determination step for determining whether or not there is a pixel having the Kth color component value in a direction having a strong correlation around the interpolation target pixel;
When it is determined that “exists” in the image feature determination step, interpolation processing is executed by the first interpolation method. When it is determined that “does not exist” in the image feature determination step, the first An interpolation step for performing an interpolation process with a second interpolation method different from the one interpolation method,
The interpolation by the first and second interpolation methods is both the J-th color component value of the interpolation processing target pixel and the pixels around the interpolation processing target pixel, and the pixels around the interpolation processing target pixel. A pixel signal processing method, wherein the K-th color component value of the interpolation target pixel is interpolated based on the K-th color component value of the pixel. The image feature determination step includes
A correlation determination step of determining a direction in which pixels with strong correlation are arranged around the interpolation target pixel;
And a presence / absence determination step for determining whether or not there is a pixel having a color component value of the same color as the color component value to be interpolated in the direction determined to have a strong correlation in the correlation determination step. The pixel signal processing method according to claim 1.   In the interpolation by the first interpolation method, an average value of the J-th color component values of pixels around the interpolation processing target pixel is subtracted from the J-th color component value of the interpolation processing target pixel. The value obtained by multiplying the value by a predetermined ratio f and the average value of the K-th color component values of the pixels around the interpolation processing target pixel are added to add the value in the interpolation processing target pixel. The pixel signal processing method according to claim 2, wherein the Kth color component value is interpolated. Interpolation by the first interpolation method is
When calculating the average value of the J-th color component value, the J-th color component value of pixels existing in the direction in which the correlation is determined to be strong in the correlation determination step centering on the interpolation target pixel Use only
When calculating an average value of the K-th color component value, the K-th color component value of pixels existing in a direction in which the correlation is determined to be strong in the correlation determination step centering on the interpolation target pixel The pixel signal processing method according to claim 3, wherein only the pixel signal is used.   In the interpolation by the first interpolation method, the ratio f is calculated by calculating the distance between pixels used when calculating the average value of the Jth color component value and the average value of the Kth color component value. The pixel signal processing method according to claim 3, wherein the pixel signal processing method is determined in accordance with a distance between pixels used when performing the processing. Interpolation by the second interpolation method is
Two-dimensional coordinate data (xi, yi) (i = i) in which the Jth color component value of the pixels around the interpolation target pixel is the x component (xi) and the Kth color component value is the y component (yi). 1 to M) M (M is an integer of 2 or more)
By performing regression analysis on the two-dimensional coordinate data, a regression equation representing the relationship between the Jth color component value and the Kth color component value around the interpolation target pixel is obtained,
5. The Kth color component value for the interpolation processing target pixel is interpolated from the regression equation and the Jth color component value of the interpolation processing target pixel. The pixel signal processing method described. Interpolation by the second interpolation method is
In the above regression analysis,

The pixel signal processing method according to claim 6, wherein the K-th color component value is obtained by a regression equation y = a × x + b using a and b given by: Interpolation by the second interpolation method is
The following equation in which the K-th color component value KK and the J-th color component value JJ of the interpolation target pixel are substituted for x and y of the regression equation y = a × x + b.

KK = a × JJ + b

The pixel signal processing method according to claim 7, wherein an interpolation calculation for obtaining the Kth color component value KK in the interpolation target pixel is performed. Interpolation by the second interpolation method is
9. The pixel signal according to claim 7, wherein linear interpolation is performed when a difference between a maximum value and a minimum value of the x component (xi) (i = 1 to M) is a predetermined threshold value or less. Processing method.   The pixel signal processing method according to claim 9, wherein the linear interpolation is performed by obtaining an average of color component values of pixels around the interpolation processing target pixel. Each color component value of a plurality of pixels arranged on a two-dimensional plane and having at least one color component value among first to Nth (N is an integer of 2 or more) different color component values. The Kth color (K is any number from 1 to N, but different from J) of the interpolation target pixel having the Jth color component value (J is any number from 1 to N) In a pixel signal processing apparatus that generates component values by interpolation,
Image feature determination means for determining whether or not there is a pixel having the Kth color component value in a direction having a strong correlation around the interpolation target pixel;
A first interpolation means for performing interpolation by the first interpolation method;
Second interpolation means for performing interpolation by a second interpolation method different from the first interpolation method;
The image feature determination means causes the first interpolation means to perform an interpolation process when it is determined to be “present” in the above determination, and when it is determined to be “not present” in the above determination, Interpolating means to perform interpolation processing,
Each of the first and second interpolation means has the J-th color component value of the pixel to be interpolated and the pixels around the pixel to be interpolated and the pixels around the pixel to be interpolated. A pixel signal processing device for interpolating the Kth color component value of the interpolation target pixel based on the Kth color component value. The image feature determination means is
Correlation determining means for determining the direction in which pixels with strong correlation are arranged around the interpolation target pixel;
And presence / absence determining means for determining whether or not there is a pixel having a color component value of the same color as the color component value to be interpolated in a direction determined to have a strong correlation by the correlation determining means. The pixel signal processing device according to claim 11.   The first interpolation means obtains a value obtained by subtracting an average value of the Jth color component values of pixels around the interpolation processing target pixel from the Jth color component value of the interpolation processing target pixel. The value obtained by multiplying the predetermined ratio f and the average value of the Kth color component values of the pixels around the interpolation processing target pixel are added to add the Kth value in the interpolation processing target pixel. The pixel signal processing device according to claim 12, wherein the color component values of the pixel signal are interpolated. The first interpolation means includes
When calculating the average value of the J-th color component value, the J-th color component value of pixels existing in the direction in which the correlation determination unit determines that the correlation is strong with the interpolation processing target pixel as the center Use only
When calculating the average value of the K-th color component value, the K-th color component value of pixels existing in the direction in which the correlation determination unit determines that the correlation is strong with the interpolation target pixel as the center The pixel signal processing device according to claim 13, wherein only the pixel signal processing device is used.   The first interpolation means calculates the ratio f, the distance between pixels used when calculating the average value of the Jth color component value, and the average value of the Kth color component value. The pixel signal processing device according to claim 13, wherein the pixel signal processing device is determined in accordance with a distance between pixels used in the step. The second interpolation means includes
Two-dimensional coordinate data (xi, yi) (i = i) in which the Jth color component value of the pixels around the interpolation target pixel is the x component (xi) and the Kth color component value is the y component (yi). 1 to M) M (M is an integer of 2 or more)
By performing regression analysis on the two-dimensional coordinate data, a regression equation representing the relationship between the Jth color component value and the Kth color component value around the interpolation target pixel is obtained,
15. The K-th color component value for the interpolation processing target pixel is interpolated from the regression equation and the J-th color component value of the interpolation processing target pixel. The pixel signal processing device described. The second interpolation means includes
In the above regression analysis,

17. The pixel signal processing apparatus according to claim 16, wherein the K-th color component value is obtained by a regression equation y = a × x + b using a and b given by: The second interpolation means includes
The following equation in which the K-th color component value KK and the J-th color component value JJ of the interpolation target pixel are substituted for x and y of the regression equation y = a × x + b.

KK = a × JJ + b

The pixel signal processing apparatus according to claim 17, wherein an interpolation calculation for obtaining the Kth color component value KK in the interpolation target pixel is performed. The second interpolation means includes
19. The pixel signal according to claim 17, wherein linear interpolation is performed when a difference between a maximum value and a minimum value of the x component (xi) (i = 1 to M) is equal to or less than a predetermined threshold value. Processing equipment.   The pixel signal processing apparatus according to claim 19, wherein the linear interpolation is performed by calculating an average of color component values of pixels around the interpolation processing target pixel.

Images