JP2005217478A - Pixel signal processing method and apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a pixel signal processing method and apparatus capable of suppressing production of a false color in an interpolation result even in a region having no similarity relation in a behavior of changes in color components and obtaining an optimum interpolation result independently of a way of the changes in the color component values in regions in the vicinity of a target pixel in the case of producing deficient color components of a plurality of pixels arranged in a way of e.g., the Bayer arrangement through interpolation. <P>SOLUTION: The apparatus and method carries out the interpolation by first and second interpolation methods different from each other to produce first and second interpolation values, and selects and outputs either of the first and second interpolation values depending on whether or not pixel signals are monotonously changed (increased or decreased). The discrimination of the behavior of the change and the first and second interpolation may be carried out on the basis of a J-th color component value of the target pixel and a color component value of a pixel existing in a direction providing a strong correlation among pixels in the regions in the vicinity of the target pixel. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a pixel signal processing device and a pixel signal processing method, and in particular, based on a set of pixel signals arranged on a two-dimensional plane, each having any one of a plurality of color component values. The present invention relates to a pixel signal processing method and apparatus for generating another color component value at a target pixel position (interpolation target pixel) where a pixel signal having one of the color component values exists.

  Such pixel signal processing, for example, generates one color component value of each of a plurality of color components (spectral sensitivity characteristics), for example, red (R), green (G), and blue (B). A plurality of types of photoelectric conversion elements are further provided with, for example, an image sensor arranged in a Bayer shape on a two-dimensional plane, and are used as a part of a color image pickup device, and among pixel signals output from the image sensor, It is used to interpolate missing color component values (insufficient color component values) at each pixel position.

  In a conventional image pickup apparatus having an image pickup element in which color filters of three primary colors of red, green, and blue are arranged in a Bayer shape, for example, as shown in Patent Document 1 below, Based on the local output signal distribution for each pixel, the output signal of each pixel is replaced with an average value, and an interpolation method based on the linear similarity ratio between the known color geometric figure and the insufficient color geometric figure is assumed. Yes.

JP 2001-197512 A (paragraphs 0048 to 0049, FIG. 7)

  This conventional method assumes that there is a similar relationship in the state of change of each color component (for example, R, G, B components in a Bayer array) in a region near the pixel of interest. For this reason, there is a problem that a false color is generated in an interpolation result in an area where the color component change state is not similar (for example, a boundary between one color and another color), and an area in the vicinity of the target pixel. There is a problem that the interpolation method may not be appropriate depending on how the color component values change.

  The present invention can suppress the occurrence of a false color in the interpolation result even in a region where there is no similarity in the color component change in the region in the vicinity of the target pixel. It is an object of the present invention to provide a pixel signal processing apparatus and method that can always perform interpolation using an optimal interpolation method regardless of how color component values change within a region.

The present invention
A set of pixel signals representing respective color component values of a plurality of pixels arranged on a two-dimensional plane and each having any of first to Nth (N is an integer of 2 or more) different color components. Based on the Kth (K is any one of 1 to N excluding J) at the position of the pixel of interest having the Jth (J is any one of 1 to N) color component value. In a pixel signal processing method for generating a pixel signal representing a color component value by interpolation,
Determining a mode of a change in a color component value of a pixel that is located in a region near the target pixel and has the same color component as the target pixel and the target pixel;
According to the determination result in the step of determining the mode of change,
The first interpolation calculation step for generating the first interpolation value by performing the interpolation by the first interpolation method or the second interpolation by performing the interpolation by a second interpolation method different from the first interpolation method. An interpolation calculation step for executing any one of the second interpolation calculation steps for generating a value,
The first interpolation calculation step and the second interpolation calculation step are based on the J-th color component value of the pixel of interest and the color component value of a pixel located in a region near the pixel of interest. Provided is a pixel signal processing method characterized by performing interpolation.

  According to the present invention, the occurrence of false colors can be suppressed, and the optimum interpolation can be performed according to how the color component value of the pixel changes in the area near the target pixel. is there.

  Embodiments of the present invention will be described below with reference to the drawings. The embodiment described below is suitable for use as a part of a full-color image capturing apparatus, for example, a digital still camera, but the present invention is not limited to this.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a configuration of an image pickup apparatus including the pixel signal processing apparatus according to Embodiment 1 of the present invention. Light incident from the lens 1 forms an image on the two-dimensional image sensor 2. The two-dimensional imaging device 2 has a plurality of photoelectric conversion elements arranged two-dimensionally, and the plurality of photoelectric conversion elements are arranged in a Bayer type, for example, as shown in FIG. Covered with color filters having spectral sensitivity characteristics corresponding to the three primary colors (R), green (G), and blue (B), and from each photoelectric conversion element, an analog of a color component corresponding to the color of the color filter A signal is output.

  In FIG. 2, horizontal and vertical represent the horizontal direction (H) and vertical direction (V) of the imaging surface, respectively. The photoelectric conversion element constitutes a pixel, and the position occupied by each photoelectric conversion element on the imaging surface corresponds to the pixel position. Since each pixel is two-dimensionally arranged on the imaging surface of the imaging device, their position can be represented by a coordinate value on the HV coordinate plane.

  The two-dimensional image sensor 2 photoelectrically converts incident light and outputs an analog pixel signal at a level corresponding to the amount of incident light for each pixel. The analog pixel signal is converted into a digital pixel signal by the A / D converter 3, output, and stored in the frame memory 11 in the interpolation processing circuit 4. At this time, the signal of each pixel is stored in association with the position of each pixel on the imaging surface, and hence the position on the HV coordinate plane.

  Since the pixel signal stored in the frame memory 11 as described above represents the value of the signal output from the image sensor 2, it is called an “output signal of the image sensor” or simply “output signal”, which will be described later. Thus, it is distinguished from a pixel signal obtained by obtaining an average of surrounding pixels and a pixel signal obtained by interpolation.

  As described above, only signals representing one component value of R, G, and B can be obtained from each of the photoelectric conversion elements constituting each pixel. That is, the output signal at each pixel position recorded in the frame memory 11 includes a pixel having only an R component value (R pixel), a pixel having only a G component (G pixel), and a pixel having only a B component (B pixel). ) In other words, only one color pixel signal is known at each pixel position, and the other color pixel signals are unknown. In the interpolation processing of the present invention, the color component value of an unknown color (insufficient color) at each pixel position is obtained by interpolation.

  In the present embodiment, the pixel signal at each pixel position has a color component of any one of R, G, and B. In general, according to the present invention, the pixel signal at each pixel position is The present invention is applicable when any of the first to Nth (N is an integer of 2 or more) color component values is used.

  The synthesis circuit 5 combines the interpolation result OUT4 input from the interpolation processing circuit 4 and the signal OUT3 obtained from the solid-state imaging device 2 input from the A / D converter 3 to obtain a full color image.

FIG. 3 shows the configuration of the interpolation processing circuit 4 used in the first embodiment.
As illustrated, the interpolation processing circuit 4 includes a frame memory 11, a read control circuit 16, a processor 35, and a memory 36. The processor 35 operates as shown in the flowcharts of FIGS. 4, 5, and 6 for interpolation processing of insufficient color component values for each pixel of the Bayer array type image data shown in FIG. 2. The insufficient color component value (interpolation value) calculated by the processor 35 is written in the memory 36.

Hereinafter, the interpolation processing of the insufficient color component performed by the interpolation processing circuit 4 will be described. For interpolation of missing color components,
(P1) Processing for obtaining G component values at the positions of the R pixel and the B pixel;
(P2) Processing for obtaining the R component value and the B component value at the position of the G pixel,
(P3) includes processing for obtaining the B component value at the position of the R pixel, and (P4) processing for obtaining the R component value at the position of the B pixel, and is performed, for example, in the order described above.

  First, the G component value interpolation process at the positions of the R pixel and the B pixel will be described. First, interpolation of the G component value at the position of the R pixel will be described with reference to FIGS. FIG. 4 is a flowchart showing the operation of the processor 35 at the time of interpolation of the G component value at the position of the R pixel, and FIG. 7 is a pixel used when performing interpolation of the G component value at the position of the R pixel and its color component value. Indicates. The R pixel R44 in FIG. 7 becomes the target pixel (interpolation target pixel).

In step ST11, the color component values of a plurality of pixels located in the region near the target pixel R44 shown in FIG. That is, the R component values R44, R42, R46, R24, R64 of the target pixel R44, the R pixels R42, R46 aligned in the vertical direction with respect to the target pixel R44, and the R pixels R24, R64 aligned in the horizontal direction with respect to the target pixel R44. G components G41, G43, G45, G47 in the vertical direction with respect to the target pixel R44, and G component values G41, G43, G45, G74 aligned in the horizontal direction with respect to the target pixel R44. G47, G14, G34, G54, and G74 are read out.
Note that, as described above, a pixel having a certain color component and the color component value of the pixel may be represented by the same symbol.

Next, in step ST12, it is determined whether the correlation is strong in the horizontal direction or the vertical direction around the target pixel. For this determination, the following values dV and dH are calculated using a part of the pixel signal (color component value) read in step ST11.
dV = | G34-G54 |
dH = | G43-G45 |

Here, dV is the color component value of the pixel in the vertical direction centered on the target pixel, that is, the absolute value of the difference between the color component values of two pixels aligned in the vertical direction with respect to the target pixel and adjacent to the target pixel. DH represents the color component value of the pixel in the horizontal direction around the target pixel, that is, the absolute value of the difference between the color component values of two pixels that are aligned in the horizontal direction with respect to the target pixel and are adjacent to the target pixel. .
For example, dH <dV when the horizontal correlation is strong in the image like a fine horizontal stripe image, and dV <dH when the vertical correlation is strong like a fine vertical stripe image.

  Depending on the magnitude relationship between dV and dH, the process after step ST13a or the process after step ST13b is performed. That is, when dH <dV (when it is determined that the correlation in the horizontal direction is strong), the processing after step ST13a is performed, and otherwise (when it is determined that the correlation in the vertical direction is strong), step ST13b The following processing is performed. Note that the equations for obtaining dH and dV are not limited to the above equations.

The processing after step ST13a will be described.
In step ST13a, the pixel is located in a region in the vicinity of the target pixel, aligned in the horizontal direction (the direction determined to have a strong correlation in the correlation determination step ST12) with the target pixel as the center, and has the same color component as the target pixel. A change mode of the color component value of the pixel and the target pixel is determined. That is, it is determined whether or not the color component value of the pixel changes monotonously (increases or decreases) in relation to the pixel position.
This is performed, for example, as follows. That is, the pixel of interest R44 and the pixel of interest R44 are aligned in the horizontal direction (the direction in which the correlation is determined to be strong), close to the pixel of interest (for example, at a position separated by two pixels), and centered on the pixel of interest. The magnitude relationship between the R component values R44, R24, R64 of the pixels R24, R64 located on the opposite sides is examined. That is, it is determined whether or not the R component values R24, R44, and R64 satisfy one of the following conditions 1 and 2.

Condition 1: R24 <R44 and R64 <R44
Condition 2: R24> R44 and R64> R44

  When condition 1 is satisfied, the color component value changes upwardly, and when condition 2 is satisfied, the color component value changes downwardly, and both conditions 1 and 2 are satisfied. If not, it can be said that the color component value changes monotonously.

  Depending on the result of the determination in step ST13a, either step ST14a or step ST15a and subsequent processing is performed. That is, when either of the conditions 1 and 2 is satisfied, the process of step ST14a is performed, and otherwise, the process of step ST15a and the subsequent processes is performed.

  Step ST14a is an interpolation operation assuming a similar relationship, and calculates the G component value G44 of the pixel of interest R44 by the following equation (2).

  In step ST15a, the following two two-dimensional data are calculated. Each of these two-dimensional data is aligned with the target pixel in the horizontal direction (direction determined to have a strong correlation), close to the target pixel (for example, at a position separated by two pixels), and centered on the target pixel. The known color component at the target pixel position obtained with respect to the positions of the pixels located on the opposite sides of each other as x component (xi) and the insufficient color (unknown) color component as y component (yi) , I = 1, 2).

  As can be seen from the above equation, as the x component xi, the R component at each pixel position obtained from the image sensor 2 is used as it is, and as the y component yi, for example, in the horizontal direction around each pixel position. The average of the G components of adjacent pixels on both sides is used.

  In step ST16a, it is checked whether or not the absolute value of the difference between the x components x1 and x2 is equal to or smaller than a certain threshold value. When the absolute value of the difference between x1 and x2 is equal to or smaller than the threshold value, the process of step ST18a is performed, and otherwise, the process of step ST17a is performed.

  Step ST17a is an interpolation operation that assumes a monotonous change, and calculates the G component value G44 for the pixel of interest R44 by the following equation (3).

In step ST18a, the G component value G44 for the target pixel R44 is calculated using linear interpolation. That is, the G component value G44 for the target pixel R44 is given by the following equation using the color component value of the pixel adjacent to the target pixel.
G44 = (G34 + G54) / 2

The processing after step ST13b will be described.
In step ST13b, the pixel is located in a region in the vicinity of the target pixel, aligned in the vertical direction (the direction in which the correlation is determined to be strong in the correlation determination step ST12) with the target pixel as the center, and has the same color component as the target pixel. It is determined whether the color component values of the pixel and the pixel of interest are changing monotonously in relation to the pixel position.
Therefore, for example, the magnitude relationship between the R component values R42, R44, and R46 of the R pixels R42, R44, and R46 is examined. Then, depending on the result, either step ST14b or step ST15b and subsequent processing is performed. That is, when the R component values R42, R44, and R46 satisfy one of the following conditions 1 and 2, the process of step ST14b is performed, and otherwise, the process of step ST15b and the subsequent processes is performed.

Condition 1: R42 <R44 and R46 <R44
Condition 2: R42> R44 and R46> R44

  Step ST14b is an interpolation operation that assumes a similar relationship, and calculates the G component value G44 of the pixel of interest R44 by the following equation.

  In step ST15b, the following two two-dimensional data are calculated. Each of these two-dimensional data is aligned in the vertical direction (the direction determined to have a strong correlation) with respect to the target pixel, close to the target pixel (for example, at a position separated by two pixels), and centered on the target pixel. The known color component at the target pixel position obtained with respect to the positions of the pixels located on the opposite sides of each other as x component (xi) and the insufficient color (unknown) color component as y component (yi) , I = 1, 2).

  In step ST16b, it is checked whether or not the absolute value of the difference between the x components x1 and x2 is equal to or smaller than a certain threshold value. When the absolute value of the difference between x1 and x2 is equal to or smaller than the threshold value, the process of step ST18b is performed, and otherwise, the process of step ST17b is performed.

  Step ST17b is an interpolation operation that assumes a monotonous change, and calculates a G component value G44 for the pixel of interest R44 using the following equation.

In step ST18b, the G component value G44 for the target pixel R44 is calculated using linear interpolation. That is, the G component value G44 for the target pixel R44 is given by the following equation using the color component value of the pixel adjacent to the target pixel.
G44 = (G43 + G45) / 2

The deficient color component value (interpolated value) interpolated in any of the above-described steps ST14a, ST17a, ST18a, ST14b, ST17b, ST18b is the G component value obtained by interpolation at the position of the target pixel R44 in step ST19. It is written in the memory 36 in association with the pixel position.
The above interpolation processing is performed on the G component values at the positions of all R pixels.

  The above is the interpolation processing of the G component value at the position of the R pixel, but the interpolation of the G component value at the position of the B pixel is similarly performed except that R and B are interchanged.

Next, the R component value and B component value interpolation processing at the G pixel position will be described.
In the Bayer array, when the G pixel is centered, the arrangement of the R, G, and B pixels is as shown in FIG. 8 when the pixel of interest is in an odd row (a) and in an even row ( Unlike b), the arrangement is such that R and B are interchanged. First, the interpolation of the R component value and the B component value at the position of the G pixel in the odd row will be described with reference to FIGS.

  FIG. 5 is a flowchart showing the operation of the processor 35 at the time of interpolation of the R component value at the G pixel position, and FIG. 9 is used when the R component value and the B component value are interpolated at the odd row G pixel positions. Pixel and its color component value.

  In FIG. 9, the G pixel G44 is the target pixel, but no R pixel exists in the vertical direction of the target pixel G44, and no B pixel exists in the horizontal direction. Therefore, only pixels in the horizontal direction can be used for interpolation of the R component values at the positions of the G pixels in the odd rows, and only pixels in the vertical direction can be used for interpolation of the B component values. Accordingly, the direction to be referred to for the interpolation processing is naturally limited to one direction, and thus the correlation determination as in step ST12 in FIG. 4 is not performed.

  First, an interpolation process of R component values at the positions of odd-numbered G pixels will be described.

  In step ST21, the color component values of a plurality of pixels located in the region in the vicinity of the target pixel G44 shown in FIG. That is, the target pixel G44, the G component values G44, G24, G64 of the G pixels G24, G64 aligned in the horizontal direction with respect to the target pixel G44, and the R pixels R14, R34, aligned in the horizontal direction with respect to the target pixel. The R component values R14, R34, R54, and R74 of R54 and R74 are read out.

Next, in step ST22, a pixel that is located in a region near the target pixel, is aligned in the horizontal direction (the direction in which the R pixel exists) around the target pixel, and has the same color component as the target pixel, and It is determined whether or not the color component value changes monotonously in relation to the pixel position.
Therefore, for example, the magnitude relationship between the G component values G24, G44, and G64 of the G pixels G24, G44, and G64 is examined. Then, depending on the result, either step ST23 or step ST24 and subsequent processes are performed. That is, when the G component values G24, G44, and G64 satisfy any one of the following conditions 1 and 2, the process of step ST23 is performed, and otherwise, the process of step ST24 and the subsequent processes is performed.

Condition 1: G24 <G44 and G64 <G44
Condition 2: G24> G44 and G64> G44

  Step ST23 is an interpolation operation assuming a similar relationship, and calculates the R component value R44 of the pixel of interest G44 by the following equation.

  In step ST24, the following two two-dimensional data are calculated. Each of these two-dimensional data is aligned in the horizontal direction (the direction in which the R pixel exists) with respect to the target pixel, is close to the target pixel (for example, at a position separated by two pixels), and is centered on the target pixel. The known color component at the target pixel position obtained with respect to the position of the pixel located on the opposite side is the x component (xi), and the insufficient color (unknown) color component is the y component (yi) (where i = 1, 2).

  As can be seen from the above formula, as the x component xi, the G component at each pixel position obtained from the image sensor 2 is used as it is, and as the y component yi, for example, in the horizontal direction around each pixel position. The average of the R components of adjacent pixels on both sides is used.

  In step ST25, it is checked whether the absolute value of the difference between the x components x1 and x2 is equal to or smaller than a threshold value. If the absolute value of the difference between x1 and x2 is equal to or smaller than the threshold value, the process of step ST27 is performed next. Otherwise, the process of step ST26 is performed next.

  Step ST26 is an interpolation operation assuming a monotonous change, and the R component value R44 of the pixel of interest G44 is calculated by the following equation.

In step ST27, an R component value R44 for the target pixel G44 is calculated using linear interpolation. That is, the R component value R44 for the target pixel G44 is given by the following equation using the color component value of the pixel adjacent to the target pixel.
R44 = (R34 + R54) / 2

  The insufficient color component value (interpolated value) interpolated in any of the above-described steps ST23, ST26, ST27 is stored in the memory 36 as the R component value obtained by interpolation at the position of the target pixel G44 in step ST28. Written in association with

  The above interpolation processing is performed on the R component values at the positions of all the odd-numbered G pixels.

  Next, the B component value interpolation process at the position of the odd-numbered G pixel will be described. As shown in FIG. 8A, the pixel having the B component is located in the vertical direction (vertical direction) around the G pixel as the target pixel, and does not exist in the horizontal direction (horizontal direction).

  In step ST21, the color component values of a plurality of pixels located in the region in the vicinity of the target pixel G44 shown in FIG. That is, the target pixel G44, the G component values G44, G42, and G46 of the G pixels G42 and G46 aligned in the vertical direction with respect to the target pixel G44, and the B pixels B41, B43, B component values B41, B43, B45, and B47 of B45 and B47 are read out.

Next, in step ST22, a pixel that is located in a region near the target pixel, is aligned in the vertical direction (the direction in which the B pixel exists) around the target pixel, has the same color component as the target pixel, and It is determined whether or not the color component value changes monotonously in relation to the pixel position.
Therefore, for example, the magnitude relationship between the G component values G42, G44, and G46 of the G pixels G42, G44, and G46 is examined. Then, depending on the result, either step ST23 or step ST24 and subsequent processes are performed. That is, when the G component values G42, G44, and G46 satisfy any one of the following conditions 1 and 2, the process of step ST23 is performed, and otherwise, the process of step ST24 and the subsequent processes is performed.

Condition 1: G42 <G44 and G46 <G44
Condition 2: G42> G44 and G46> G44

  In step ST23, the B component value B44 of the target pixel G44 is calculated by the following equation.

  In step ST24, the following two two-dimensional data are calculated. Each of these two-dimensional data is aligned in the vertical direction (the direction in which the B pixel exists) with respect to the target pixel, is close to the target pixel (for example, at a position separated by two pixels), and is centered on the target pixel. The known color component at the target pixel position obtained with respect to the position of the pixel located on the opposite side is the x component (xi), and the insufficient color (unknown) color component is the y component (yi) (where i = 1, 2).

  In step ST25, it is checked whether or not the absolute value of the difference between the x components x1 and x2 is equal to or less than a threshold value. When the absolute value of the difference between x1 and x2 is equal to or smaller than the threshold value, the process of step ST27 is performed next, and otherwise, the process of step ST26 is performed.

  In step ST26, the B component value B44 of the target pixel G44 is calculated by the following equation.

In step ST27, the B component value B44 for the target pixel G44 is calculated using linear interpolation. That is, the B component value B44 for the target pixel G44 is given by the following equation using the color component values of the pixels adjacent to the target pixel.
B44 = (B43 + B45) / 2

  The insufficient color component value (interpolation value) interpolated in any of the above-described steps ST23, ST26, ST27 is stored in the memory 36 as the B component value obtained by interpolation at the position of the target pixel G44 in step ST28. Written in association with

  The above interpolation processing is performed for the B component values at the positions of all G pixels in all odd rows.

  The description of the interpolation of the R component value at the position of the G pixel in the odd row and the description of the interpolation of the B component value at the position of the G pixel in the odd row are the vertical direction (up and down direction) and the horizontal direction (left and right direction). Is just replaced.

  The above is the interpolation processing of the R component value and the B component value at the position of the G pixel in the odd row, but the interpolation of the R component value and the B component value at the position of the G pixel in the even row is the same as in the case of the odd row. To be done. However, “vertical direction” and “horizontal direction” are interchanged. From another point of view, the interpolation of the R component value and the B component value at the position of the G pixel in the even row replaces the interpolation between the R component value and the B component value at the position of the G pixel in the odd row, and R and B. The other operations are the same.

Next, the B component value interpolation processing at the position of the R pixel will be described with reference to FIGS.
FIG. 6 is a flowchart showing the operation of the processor 35 when interpolating the B component value at the R pixel position, and FIG. 10 shows the pixel and its color component value used when interpolating the B component value at the R pixel position. Indicates. The R pixel R44 in FIG. 10 is the target pixel.

  In step ST31, the color component values of a plurality of pixels located in the region near the target pixel R44 shown in FIG. That is, the target pixel R44, R pixels R62 and R26 aligned in the direction from the upper right to the lower left (45 ° direction) with respect to the target pixel, the upper left to the lower right direction (−45 ° direction) ) R component values R44, R62, R26, R22, R66 of the R pixels R22, R66 aligned with ()) and the B pixel aligned with respect to the target pixel from the upper right to the lower left (45 degree direction). B71, B53, B35, B17, B component values B71, B53, B35, B17, B11 of B pixels B11, B33, B55, B77 aligned in the direction from the upper left to the lower right (the direction of −45 degrees), B33, B55, and B77 are read out.

Next, in step ST32, it is determined whether the correlation is strong in the direction of 45 degrees or the direction of -45 degrees around the target pixel. For this determination, the following values d (45) and d (−45) are calculated using a part of the pixel signal (color component value) read in step ST31.
d (45) = | B53−B35 |
d (−45) = | B55−B33 |

  d (45) is the color component value of a pixel on a straight line in the direction of 45 degrees with respect to the pixel of interest in the horizontal direction (upper right to lower left, ie, upward to the right), that is, 45 with respect to the pixel of interest. The absolute value of the difference between the color component values of two pixels adjacent to the target pixel is aligned in the direction of degrees, and d (−45) is the direction in which the angle from the horizontal direction around the target pixel is −45 degrees ( The color component values of the pixels on the straight line in the direction from the upper left to the lower right, that is, the upward left direction), that is, the color component values of two pixels that are aligned in the direction of −45 degrees with respect to the target pixel Represents the absolute value of the difference.

Similar to the processing in step ST12, the value calculated using the color component values of the pixels in the direction of strong correlation in the image is smaller.
Depending on the magnitude relationship between d (45) and d (−45), the process after step ST33a or the process after step ST33b is performed. That is, when d (45) <d (−45) (when the correlation is strong in the direction of 45 degrees), the processing after step ST33a is performed, and otherwise (when the correlation is strong in the direction of −45 degrees) Step ST33b and subsequent processes are performed. Note that the equations for obtaining d (45) and d (−45) are not limited to the above.

The processing after step ST33a will be described.
In step ST33a, the pixel is located in a region in the vicinity of the target pixel, aligned in a 45 degree direction (the direction in which the correlation is determined to be strong in the correlation determination step ST32) around the target pixel, and the same color component as the target pixel. It is determined whether or not the color component values of the pixel having the pixel and the target pixel change monotonously in relation to the pixel position.
Therefore, for example, the magnitude relationship between the R component values R62, R44, and R26 of the R pixels R62, R44, and R26 is examined. And the process below step ST34a or step ST35a is performed by the result. That is, when R62, R44, and R26 satisfy one of the following conditions 1 and 2, the process of step ST34a is performed, and otherwise, the process of step ST35a and the subsequent processes is performed.

Condition 1: R62 <R44 and R26 <R44
Condition 2: R62> R44 and R26> R44

  Step ST34a is an interpolation operation assuming a similar relationship, and calculates the B component value B44 of the target pixel R44 by the following equation.

  In step ST35a, the following two two-dimensional data are calculated. Each of these two-dimensional data is aligned in a direction of 45 degrees with respect to the target pixel (direction determined to have a strong correlation), and is close to the target pixel (for example, a position separated by two pixels in the direction of 45 degrees) The known color component at the target pixel position (x component) obtained for the position of the pixel located on the opposite side with respect to the target pixel (position separated by two pixels in the horizontal and vertical directions) (Xi) and the color component of the insufficient color (unknown) as the y component (yi) (where i = 1, 2).

  As can be seen from the above formula, as the x component xi, the R component at each pixel position obtained from the image sensor 2 is used as it is, and as the y component yi, for example, 45 degrees around each pixel position. The average of the B components of adjacent pixels on both sides of the direction is used.

  In step ST36a, it is checked whether or not the absolute value of the difference between the x components x1 and x2 is equal to or smaller than a certain threshold value. When the absolute value of the difference between x1 and x2 is equal to or smaller than the threshold value, the process of step ST38a is performed. Otherwise, the process of step ST37a is performed.

  Step ST37a is an interpolation operation that assumes a monotonous change, and calculates a B component value B44 for the pixel of interest R44 using the following equation.

In step ST38a, the B component value B44 for the target pixel R44 is calculated using linear interpolation. That is, the B component value B44 for the target pixel B44 is given by the following equation using the color component values of the pixels adjacent to the target pixel.
B44 = (B53 + B35) / 2

Next, the processing after step 33b will be described.
First, in step ST33b, as in step 33a, it is located in the region in the vicinity of the target pixel, and is in the direction of −45 degrees around the target pixel (the direction in which the correlation is determined to be strong in the correlation determination step ST32). It is determined whether the pixels that are aligned and have the same color component as the target pixel, and the color component value of the target pixel change monotonously in relation to the pixel position.
Therefore, for example, the magnitude relationship between the R component values R22, R44, and R66 of the R pixels R22, R44, and R66 is examined. Then, depending on the result, either step ST34b or processing after step ST35b is performed. That is, when the R component values R22, R44, and R66 satisfy one of the following conditions 1 and 2, the process of step ST34b is performed, and otherwise, the process of step ST35b and the subsequent processes is performed.

Condition 1: R22 <R44 and R66 <R44
Condition 2: R22> R44 and R66> R44

  Step ST34b is an interpolation operation that assumes a similar relationship, and calculates the B component value B44 of the pixel of interest R44 by the following equation.

  In step ST35b, the following two two-dimensional data are calculated. Each of these two-dimensional data is aligned in the direction of −45 degrees with respect to the target pixel (the direction in which the correlation is determined to be strong) and close to the target pixel (for example, two pixels are separated in the direction of −45 degrees). The color of the deficient color (unknown) is defined as the x component (xi), which is the known color component at the target pixel position, which is obtained with respect to the position of the pixel located on the opposite side of the target pixel. The component is the y component (yi) (where i = 1, 2).

  In step ST36b, it is checked whether or not the absolute value of the difference between the x components x1 and x2 is equal to or less than a certain threshold value. When the absolute value of the difference between x1 and x2 is equal to or smaller than the threshold value, the process of step ST38b is performed, and otherwise, the process of step ST37b is performed.

  Step ST37b is an interpolation operation that assumes a monotonous change, and calculates a B component value B44 for the pixel of interest R44 using the following equation.

In step ST38b, a B component value B44 for the target pixel R44 is calculated using linear interpolation. That is, the B component value B44 for the target pixel R44 is given by the following equation using the color component values of the pixels adjacent to the target pixel.
B44 = (B33 + B55) / 2

  The deficient color component value (interpolated value) interpolated in any of the above steps ST34a, ST37a, ST38a, ST34b, ST37b, ST38b is the B component value obtained by interpolation at the position of the target pixel R44 in step ST39. It is written in the memory 36 in association with the pixel position.

  The above interpolation processing is performed on the B component values at the positions of all R pixels.

  The above is the process of interpolating the B component value at the position of the R pixel, but the interpolation of the R component value at the position of the B pixel is similarly performed except that R and B are interchanged.

  As described above, all the insufficient color component values for the R, G, and B pixels are calculated, and all the insufficient color component values for all the pixels are written in the memory 36.

  When all the insufficient color component values for all the pixels are written in the memory 36, the contents in the memory 36 are output to the outside of the interpolation processing circuit 4 as the interpolation result OUT4. The interpolation result OUT4 and the signal OUT3 obtained by the solid-state imaging device 2 inputted from the A / D converter 3 are synthesized by the synthesis circuit 5, so that the colors of the R, G, and B colors for all the pixels are obtained. An image signal (full color image) having the same components is output.

Hereinafter, characteristics of the interpolation processing (hereinafter referred to as “interpolation processing assuming monotonic change”) performed in steps ST17a, ST17b, ST26, ST37a, and ST37b, and “similarity” performed in steps ST14a, ST14b, ST23, ST34a, and ST34b. The characteristics of the “interpolation process assuming the relationship” and the determination of the color component value change mode performed in steps ST13a, ST13b, ST22, ST33a, and ST33b to select them will be described.
First, the determination regarding the mode of change will be described.

  For example, in the case where determination is made regarding the mode of change of the R component value as in step 13a, the mode of change can be classified into FIGS. 11 (a) to 11 (d).

  11A to 11D, the R component values between the R pixels R24, R44, and R64 on the coordinate plane in which the horizontal axis corresponds to the position of the R pixel and the vertical axis corresponds to the R component value of each pixel. The change pattern with respect to the pixel position is represented by a graph.

As shown in FIG. 11A or 11B, an image containing a lot of high-frequency components is convex upward or convex downward. In the case of FIG. 11A, Condition 1 is satisfied, and FIG. In the case of b), condition 2 is satisfied. On the other hand, in an image containing a lot of low frequency components, the color signal is monotonically increasing as shown in FIG. 11C or monotonically decreasing as shown in FIG. Both 2 are not satisfied. As will be described in detail below, in the cases of FIGS. 11 (a) and 11 (b), the interpolation process assuming a similar relationship is excellent, and in the cases of FIGS. 11 (c) and 11 (d), monotonicity is performed. Interpolation processing assuming changes is excellent.
Therefore, in the case of FIGS. 11A and 11B, that is, when either condition 1 or condition 2 is satisfied, interpolation processing assuming a similarity relationship in step ST14a is performed, and FIGS. In the case of d), that is, when neither condition 1 nor condition 2 is satisfied, the process after step ST15a, in particular, the interpolation process (ST17a) assuming a monotonic change is performed. However, when the amount of change is extremely small, linear interpolation (ST18a) is performed instead of interpolation processing (ST17a) assuming monotonous change.
Thereby, the optimal interpolation process can be selected and executed according to the mode of change.

  The operations in steps ST13b, ST22, ST33a, and ST33b are the same as those in step 13a.

  Hereinafter, features of the interpolation processing assuming monotonic changes performed in steps ST17a, ST17b, ST26, ST37a, and ST37b will be described in detail.

In the region near the target pixel, when the known color component value at the target pixel position changes monotonously as shown in FIGS. 11C and 11D, the color component value of the insufficient color also monotonously. Often changed. That is, it changes monotonously in the same direction as the known color (increases with the known color or decreases with the known color), or changes monotonously in the opposite direction (the insufficient color decreases with respect to the increase in the known color, or the known color In many cases, the deficient color increases).
The former is, for example, a case where the brightness is gradually changing, and the latter is, for example, a case where the color is smoothly switched from one color to another.

In that case, it can be assumed that there is a relationship as shown in FIG. 12A or 12B approximately between the color changes. 12A and 12B, the H axis corresponds to the horizontal axis on the imaging surface of the imaging device 2, and the Z axis represents the color component value of each pixel. X0, x1, and x2 indicate color component values of known colors (colors having known color component values in the target pixel) in the target pixel and pixels adjacent thereto, and a solid line connecting these indicates the change. y1 and y2 are color component values of an insufficient color (a color component value is unknown in the target pixel) in a pixel close to the target pixel. y0 is an interpolation value of the insufficient color component value in the target pixel. 12A shows a case where the color component value of the known color and the color component value of the insufficient color change in the same direction, and FIG. 12B shows the color component value of the known color and the color of the insufficient color. The case where the component value is changing in the opposite direction is shown.
In the interpolation process assuming monotonous change, as described above, the color component value of the known color and the color component value of the insufficient color change monotonously in the same direction and in the same direction, or monotonously in the same direction and in the opposite direction. The color component value y0 of the insufficient color at the target pixel position is determined so that the following relationship is satisfied.
(Y0-y1) / (x0-x1) = (y1-y2) / (x1-x2)
If this equation is transformed,

It becomes. That is, the color component value y0 of the insufficient color at the target pixel is obtained by the above formula.
As described above, x0 and y0 are the color component value of the known color and the color component value of the insufficient color at the target pixel position. For example, when obtaining the G component value at the target pixel R44 (in the case of Expression (3)), R44 becomes x0 and G44 becomes y0.

The concept of interpolation processing assuming monotonic changes can also be explained as follows.
That is, the slope of the straight line passing through (x1, y1) and (x2, y2) in the two-dimensional plane shown in FIG. 13 where the color component value of the known color is the x component and the color component value y component of the insufficient color is ,
(Y1-y2) / (x1-x2)
And y increment Δy (= y0−y1) accompanying the change of x from x1 to x0 is
Δy0 = {(y1−y2) / (x1−x2)} × (x0−x1)
Since this straight line passes through (x1, y1), a value obtained by adding y1 to Δy0 is y0. That is, the equation that gives y0 is as shown in equation (1) above. As described above, the interpolation processing assuming monotonic change is interpolation based on the assumption that there is an interpolation value on a straight line passing through (x1, y1) and (x2, y2) in the xy coordinate plane. It can also be called “interpolation on the top”.

  Interpolation processing assuming monotonic change is suitable for cases where the brightness gradually changes in the image as described above, or when the color is smoothly switched from one color to another. It is excellent for interpolation processing in an image area containing a lot of frequency components.

  The calculation of equation (1) used in the interpolation process assuming a monotonic change includes division. In general, division cannot be performed when the divisor ((x1-x2) in the above equation) becomes zero. In addition, when the divisor is a value close to zero, the accuracy of the calculation deteriorates, which leads to an increase in error. Therefore, when the absolute value of the difference between x1 and x2 is zero or a value close to zero, it is desirable to use a different interpolation process without using Equation (1). Therefore, in the above embodiment, in steps ST16a, ST16b, ST25, ST36a, ST36b, it is checked whether or not the absolute value of the difference between x1 and x2 is less than a certain threshold value. Interpolation other than interpolation, for example, linear interpolation is performed.

  When the absolute value of the difference between x1 and x2 is less than or equal to the above threshold value, it can be estimated that there is almost no color change, and it is considered that a sufficiently satisfactory result can be obtained even with simple interpolation such as linear interpolation. In addition, linear interpolation is convenient because it is easy to calculate.

  Next, an interpolation process performed in steps ST14a, ST14b, ST23, ST34a, and ST34b assuming a similarity relationship will be described with reference to FIG. Interpolation processing assuming a similar relationship assumes that changes in two color component values have a similar relationship with each other.

FIG. 14 is a diagram for explaining how to determine the G component value at the position of the target pixel R44, which is interpolated by the interpolation processing in step 14a. In the figure, the H axis corresponds to the horizontal axis on the imaging surface of the image sensor 2, and the Z axis represents the component value of each pixel. R1 and R2 are points in which the R component values of the pixels R24 and R64 are replaced with average values of the R component values R24 and R64, and R0 is a point that represents the R component value R44 of the pixel R44.
G1 and G2 are points where the G component values of the pixels G34 and G54 are replaced by their average values, and G0 is a point representing the G component value interpolated on the pixel R44.

  In the calculation of equation (2) performed in step ST14a and the like, the value of G0 is determined so that the triangle R0R1R2 and the triangle G0G1G2 are similar. That is, with respect to the known color component R0 in the target pixel R44, a known color component triangle (geometrical figure) formed by the average value of the R component value of the target pixel R44 and the R component value of the pixel having the R component around the target pixel R44, As for the insufficient color component G0 to be interpolated on the target pixel R44, the insufficient color component formed by the average value of the G component values of the pixel having the value G0 obtained by interpolation and the G component (insufficient color component) around the target pixel R44. G0 is determined so that the triangles (geometric figures) are similar.

  In other words, the difference between the average value of the R component values of the pixels (R24, G64) having two R components around the target pixel R44 and the value of the R component at the position of the target pixel R44, and the above average value The ratio of the distance on the H axis between the two pixels (R24, R64) and the target pixel (R44) used in the calculation of (the distance on the two-dimensional plane (HV plane)) and the periphery of the target pixel R44 The difference between the average value of the G component values of the pixels having the two G components (G34, G54) and the G component value G0 on the target pixel R44 obtained by interpolation is used to calculate the average value. The value G0 of the G component at the position of the target pixel is determined so that the ratio of the distance between the two pixels (G34, G54) and the target pixel (R44) on the two-dimensional plane is equal to each other.

In FIG. 14, L1 is a curve representing a change in the color component value (for example, R component value) of the known color, L2 is a curve representing a change in the color component value (for example, the G component value) of the insufficient color, A point indicated by, for example, represents a color component value of a pixel signal given as an output of the image sensor 2, and a point indicated by ◯ represents a color component value of a pixel signal given by interpolation processing.
From FIG. 14, it can be seen that when the color component values of the two colors change in the same way, a satisfactory result can be obtained by the interpolation process assuming a similar relationship.

  In other words, in the interpolation process assuming the similarity relationship, the interpolation process is performed so that the increase / decrease in the color component values of the R, G, and B colors match in the local region. Therefore, the interpolation process assuming a similarity relationship is excellent in an interpolation process in a region where the increase / decrease of the color component is large in the local region, particularly an image containing a lot of high frequency components.

  As an example of the description, the interpolation processing performed in step ST14a has been taken up, but the same can be said for the interpolation processing performed in steps ST14b, ST23, ST34a, and ST34b.

Interpolation processing assuming monotonic change (however, if the change is extremely small, linear interpolation is used instead) and interpolation processing assuming similarity is selected and executed according to the situation, and the result of interpolation is good ( Specific numerical examples will be described with reference to FIGS. 15 and 16.
15 and 16, the horizontal axis represents the position of each pixel in the direction in which the pixels are aligned with GRGR (for example, odd rows), the vertical axis represents the color component value of each pixel, and ■ represents the color component value of the G pixel. ♦ represents the color component value of the R pixel, ○ represents the result of the interpolation processing assuming monotonic change, Δ represents the result of the interpolation processing assuming similarity, and each numerical value represents each color component value Represents.

  In the above description, when the color component value changes slowly as shown in FIG. 15, the result of the interpolation process assuming monotonic change is when the color component value changes in a short cycle as shown in FIG. It can be seen that the result of the interpolation process assuming a similar relationship is better. It can be seen that a good interpolation result can always be obtained regardless of the mode of change by selecting the interpolation process according to the mode of change.

  In this embodiment, further, in steps ST12 and ST32, the color component values of pixels existing on two straight lines that pass through the pixel of interest and are orthogonal to each other are used to determine the direction of strong correlation in the image, and interpolation is performed. Since the pixels and color component values used for processing are narrowed down to those existing in the direction in which it is determined that the correlation is strong centering on the pixel of interest, highly accurate interpolation processing can be performed.

Embodiment 2. FIG.
Next, an example in which the interpolation processing described in the first embodiment is realized by hardware will be described.
The configuration of the interpolation processing circuit 4 used in the second embodiment is as shown in FIG. 17. In addition to the frame memory 11, the read control circuit 16, and the memory 36 similar to those in FIG. 3, the processor 35 in FIG. In addition, a correlation determination circuit 12, a first interpolation calculation circuit 13, a second interpolation calculation circuit 14, and a selection circuit 15 are provided.

  The correlation discriminating circuit 12 has the same function as step ST12 in FIG. 4 and step S32 in FIG. 6, and the selection circuit 15 is in steps 13a and 13b in FIG. 4, step ST22 in FIG. 5, and steps ST33a and ST33b in FIG. The interpolation calculation circuit 13 performs interpolation assuming a similar relationship, and is equivalent to steps ST14a and ST14b in FIG. 4, step ST23 in FIG. 5, and steps ST34a and ST34b in FIG. The interpolation calculation circuit 14 has a function and performs interpolation or linear interpolation assuming a monotonic change, and steps ST15a, ST16a, ST17a, ST18a, ST15b, ST16b, ST17b, ST18b of FIG. 4 and steps of FIG. ST24, ST25, ST26, ST27, steps ST35a, ST36a in FIG. With T37a, ST38a, ST35b, ST36b, ST37b, the same function as ST38b.

  The interpolation processing circuit 4 shown in FIG. 17 can also be used as a part of the imaging apparatus shown in FIG.

A part of the signal stored in the frame memory 11 is output by the read control circuit 16 (OUT11) and input to the correlation determination circuit 12, the interpolation calculation circuit 13, the interpolation calculation circuit 14, and the selection circuit 15.
The correlation determination circuit 12 determines the direction of strong correlation in the image based on the output OUT11 from the frame memory 11, and the determination result is supplied to the interpolation calculation circuit 13, the interpolation calculation circuit 14, and the selection circuit 15.

The interpolation calculation circuit 13 performs the later-described interpolation (interpolation assuming a similar relationship) based on the output OUT11 from the frame memory 11 and the determination result of the correlation determination circuit 12, and outputs the interpolation result OUT13.
Based on the output OUT11 from the frame memory 11 and the determination result of the correlation determination circuit 12, the interpolation calculation circuit 14 performs the later-described interpolation (interpolation assuming a monotonic change or linear interpolation) and outputs an interpolation result OUT14.

  Based on the output OUT11 from the frame memory 11 and the discrimination result of the correlation discriminating circuit 12, the selection circuit 15 is located in a region in the vicinity of the pixel of interest, and the correlation discriminating unit 12 correlates around the pixel of interest. The pixel is aligned in the direction determined to be strong and has the same color component as the target pixel, and the change mode of the color component value of the target pixel is determined. That is, it is determined whether or not the color component value of the pixel changes monotonously in relation to the pixel position. Then, based on the determination result as to whether or not the change is monotonous, one of the interpolation results OUT13 and OUT14 is selected, and the selection result is output as the selected interpolation signal OUT15. That is, when it is determined that the change is monotonous, OUT14 is selected. Otherwise, OUT13 is selected.

  The output OUT15 of the selection circuit 15 is written in the memory 36.

  Next, the interpolation process of the insufficient color component value performed in the interpolation processing circuit 4 of the second embodiment will be described. For the R pixel, the G component value and the B component value become the B component value and the R component value for the G pixel, and for the B pixel, the R component value and the G component value become the insufficient color component value. Interpolation processing of these insufficient color component values will be described in order.

  First, the G component value interpolation process at the position of the R pixel will be described.

  The color component value of each pixel shown in FIG. 7 is read from the frame memory 11 and output as OUT11. That is, the R component values R44, R42, R46, R24, R64 of the target pixel R44, the R pixels R42, R46 aligned in the vertical direction with respect to the target pixel R44, and the R pixels R24, R64 aligned in the horizontal direction with respect to the target pixel R44. G components G41, G43, G45, G47 in the vertical direction with respect to the target pixel R44, and G component values G41, G43, G45, G74 aligned in the horizontal direction with respect to the target pixel R44. G47, G14, G34, G54, and G74 are read out.

  The correlation determination circuit 12 determines whether the correlation is strong in the horizontal direction or the vertical direction around the target pixel. For this determination, the following values dV and dH are calculated using a part of the pixel signal (color component value) read out as described above, and the magnitude relationship is examined.

dV = | G34-G54 |
dH = | G43-G45 |

Here, dV is aligned in the vertical direction with respect to the target pixel and represents the absolute value of the difference between the color component values of two pixels adjacent to the target pixel, and dH is aligned in the horizontal direction with respect to the target pixel. This represents the absolute value of the difference between the color component values of two pixels adjacent to the pixel.
When dH <dV, it is determined that the horizontal correlation is strong. Otherwise, it is determined that the vertical correlation is strong.
The operations of the interpolation calculation circuit 13, the interpolation calculation circuit 14, and the selection circuit 15 change depending on the magnitude relationship between dV and dH.
First, operations of the interpolation calculation circuit 13, the interpolation calculation circuit 14, and the selection circuit 15 when dH <dV are described.

  The interpolation calculation circuit 13 calculates and outputs the G component value OUT13 of the target pixel R44 by the following equation.

  The interpolation calculation circuit 14 calculates the G component value OUT14 of the target pixel R44 by the following equation.

  However,

  X1, y1, x2, and y2 are calculated in the interpolation calculation circuit 14.

The interpolation calculation circuit 14 further determines whether or not the absolute value of the difference between x1 and x2 is equal to or smaller than a threshold value. When the absolute value is equal to or smaller than the threshold value, the G component value OUT14 for the target pixel R44 is calculated and output using linear interpolation. . That is, the G component value OUT14 for the target pixel R44 is calculated and output using the color component value of the pixel adjacent to the target pixel according to the following formula.
OUT14 = (G34 + G54) / 2

  The selection circuit 15 is located in a region in the vicinity of the target pixel, is aligned in the direction in which the correlation determination unit 12 determines that the correlation is strong, and has the same color component as the target pixel. And a mode of change in the color component value of the target pixel is determined. That is, it is determined whether or not the color component value of the pixel changes monotonously in relation to the pixel position. This is performed, for example, as follows. That is, for example, the magnitude relationship between the R component values R24, R44, and R64 of the R pixels R24, R44, and R64 is examined. That is, it is determined whether or not the R component values R24, R44, and R64 satisfy one of the following conditions 1 and 2.

Condition 1: R24 <R44 and R64 <R44
Condition 2: R24> R44 and R64> R44

  When either of the conditions 1 and 2 is satisfied, the interpolation signal OUT13 from the interpolation calculation circuit 13 is selected. Otherwise, the interpolation signal OUT14 from the interpolation calculation circuit 14 is selected. The selected interpolation signal is output as OUT15.

  The above is the operation of the interpolation calculation circuit 13, the interpolation calculation circuit 14, and the selection circuit 15 when dH <dV. Next, operations of the interpolation calculation circuit 13, the interpolation calculation circuit 14, and the selection circuit 15 when dH <dV are not described.

  The interpolation calculation circuit 13 calculates and outputs the G component value OUT13 of the target pixel R44 by the following equation.

  The interpolation calculation circuit 14 calculates and outputs the G component value OUT14 of the target pixel R44 by the following equation.

However,

  X1, y1, x2, and y2 are calculated in the interpolation calculation circuit 14.

  The interpolation calculation circuit 14 further determines whether or not the absolute value of the difference between x1 and x2 is equal to or smaller than a threshold value. When the absolute value is equal to or smaller than the threshold value, the G component value OUT14 for the target pixel R44 is calculated and output using linear interpolation. . That is, the G component value OUT14 for the target pixel R44 is calculated and output using the color component value of the pixel adjacent to the target pixel according to the following formula.

  OUT14 = (G43 + G45) / 2

  The selection circuit 15 is located in a region in the vicinity of the target pixel, is aligned in a direction in which the correlation determination unit 12 determines that the correlation is strong with the target pixel as a center, and has the same color component as the target pixel. In addition, it is determined whether the color component value of the target pixel changes monotonously in relation to the pixel position. Therefore, for example, the magnitude relationship between the R component values R42, R44, and R46 of the R pixels R42, R44, and R46 is examined. Then, according to the result, either OUT13 or OUT14 is output as OUT15. That is, OUT13 is output when the R component values R42, R44, and R46 satisfy either of the following conditions 1 and 2, and OUT14 is output otherwise.

Condition 1: R42 <R44 and R46 <R44
Condition 2: R42> R44 and R46> R44

  The above is the interpolation processing of the G component value at the position of the R pixel, but the interpolation of the G component value at the position of the B pixel is similarly performed except that R and B are interchanged.

  Next, the R component value and B component value interpolation processing at the G pixel position will be described.

  In the Bayer array, when the G pixel is centered, the arrangement of the R, G, and B pixels is as shown in FIG. 8 when the pixel of interest is in an odd row (a) and in an even row ( Unlike b), the arrangement is such that R and B are interchanged. First, interpolation of R component values and B component values at the positions of G pixels in odd rows will be described with reference to FIG.

  The color component values of a plurality of pixels located in the area in the vicinity of the target pixel G44 shown in FIG. 9 are read from the frame memory 4 and output as OUT11. That is, the target pixel G44, the G component values G44, G24, G64, G42, and G46 of the G pixels G24, G64, G42, and G46 aligned in the horizontal and vertical directions with respect to the target pixel G44, and the horizontal with respect to the target pixel R component values R14, R34, R54, R74 of R pixels R14, R34, R54, R74 aligned in the direction, and B component values B41 of B pixels B41, B43, B45, B47 aligned in the vertical direction with respect to the target pixel , B43, B45, B47 are read out.

  In FIG. 9, the G pixel G44 is the target pixel, but no R pixel exists in the vertical direction of the target pixel G44, and no B pixel exists in the horizontal direction. Therefore, only pixels in the horizontal direction can be used for interpolation of the R component values at the positions of the G pixels in the odd rows, and only pixels in the vertical direction can be used for interpolation of the B component values. Therefore, the direction to be referred for the interpolation process is naturally limited to one direction, and the correlation determination circuit 12 does not perform the correlation determination.

  First, the interpolation process of the R component value in the G pixels in the odd rows will be described.

  The interpolation calculation circuit 13 calculates and outputs the R component value OUT13 of the target pixel G44 by the following equation.

  The interpolation calculation circuit 14 calculates the R component value OUT14 of the target pixel G44 by the following equation.

However,

  X1, y1, x2, and y2 are calculated in the interpolation calculation circuit 14.

The interpolation calculation circuit 14 further determines whether or not the absolute value of the difference between x1 and x2 is equal to or smaller than a threshold value. When the absolute value is equal to or smaller than the threshold value, the G component value OUT14 for the target pixel R44 is calculated and output using linear interpolation. . That is, the R component value OUT14 for the target pixel G44 is calculated and output using the color component value of the pixel adjacent to the target pixel according to the following equation.
OUT14 = (R34 + R54) / 2

  The selection circuit 15 is located in a region near the target pixel, is aligned in the horizontal direction (the direction in which the R pixel exists) around the target pixel, and has the same color component as the target pixel, and the target pixel It is determined whether or not the color component value changes monotonously in relation to the pixel position. Therefore, for example, the magnitude relationship between the G component values G24, G44, and G64 of the G pixels G24, G44, and G64 is examined. Then, according to the result, either OUT13 or OUT14 is output as OUT15. That is, when the G component values G24, G44, and G64 satisfy one of the following conditions 1 and 2, OUT13 is selected, and when not, OUT14 is selected and output.

Condition 1: G24 <G44 and G64 <G44
Condition 2: G24> G44 and G64> G44

The above is the interpolation processing of the R component value at the position of the odd-numbered G pixel.
Next, the B component value interpolation process at the position of the odd-numbered G pixel will be described.

  The interpolation calculation circuit 13 calculates and outputs the B component value OUT13 of the target pixel G44 using the following equation.

  The interpolation calculation circuit 14 calculates the B component value OUT14 of the target pixel G44 by the following equation.

However,

  X1, y1, x2, and y2 are calculated in the interpolation calculation circuit 14.

  The interpolation calculation circuit 14 further determines whether or not the absolute value of the difference between x1 and x2 is equal to or smaller than a threshold value. When the absolute value is equal to or smaller than the threshold value, the B component value OUT14 for the target pixel G44 is calculated and output using linear interpolation. . That is, the B component value OUT14 for the target pixel G44 is calculated and output by the following formula using the color component value of the pixel adjacent to the target pixel.

  OUT14 = (B43 + B45) / 2

  The selection circuit 15 is located in a region near the target pixel, is aligned in the vertical direction (the direction in which the B pixel exists) around the target pixel, and has the same color component as the target pixel, and the target pixel It is determined whether or not the color component value changes monotonously in relation to the pixel position. Therefore, for example, the magnitude relationship between the G component values G42, G44, and G46 of the G pixels G42, G44, and G46 is examined. Then, according to the result, either OUT13 or OUT14 is output as OUT15. That is, when the G component values G42, G44, and G46 satisfy either of the following conditions 1 and 2, OUT13 is selected and when it is not, OUT14 is selected and output.

Condition 1: G42 <G44 and G46 <G44
Condition 2: G42> G44 and G46> G44

The above interpolation processing is performed for the B component values at the positions of all G pixels in all odd rows.
The above is the interpolation processing of the B component value at the position of the G pixel in the odd row.
The above is the interpolation processing of the R component value and the B component value at the position of the G pixel in the odd row, but the interpolation processing of the R component value and the B component value at the position of the G pixel in the even row is that R and B are interchanged. The same is done.

Next, the B component value interpolation processing at the position of the R pixel will be described with reference to FIG.
The color component values of a plurality of pixels located in the region in the vicinity of the target pixel R44 shown in FIG. 10 are read from the frame memory 4 and output as OUT11. That is, the target pixel R44, R pixels R62 and R26 aligned in the direction from the upper right to the lower left (45 ° direction) with respect to the target pixel, the upper left to the lower right direction (−45 ° direction) ) R component values R44, R62, R26, R22, R66 of the R pixels R22, R66 aligned with ()) and the B pixel aligned with respect to the target pixel from the upper right to the lower left (45 degree direction). B71, B53, B35, B17, B component values B71, B53, B35, B17, B11 of B pixels B11, B33, B55, B77 aligned in the direction from the upper left to the lower right (the direction of −45 degrees), B33, B55, and B77 are read out.

  The correlation discriminating circuit 12 discriminates whether the correlation is strong in the upward direction (45 degrees direction) or the upward direction (−45 degrees direction) with respect to the target pixel. For this determination, the following values d (45) and d (−45) are calculated using a part of the pixel signal (color component value) read out as described above, and the magnitude relationship is examined. .

d (45) = | B53−B35 |
d (−45) = | B55−B33 |

d (45) is a color component of two pixels that are aligned in a direction whose angle to the horizontal direction is 45 degrees centering on the pixel of interest (the direction from the upper right to the lower left, that is, the direction of rising to the right) and adjacent to the pixel of interest. This represents the absolute value of the difference between the values, and d (−45) is aligned in a direction where the angle with the horizontal direction is −45 degrees with respect to the pixel of interest (the direction from the upper left to the lower right, that is, the upward direction to the left). Represents the absolute value of the difference between the color component values of two pixels adjacent to the target pixel.
When d (45) <d (−45), it is determined that the correlation in the upward direction is strong, and otherwise, it is determined that the correlation in the upward direction is strong.
The operations of the interpolation calculation circuit 13, the interpolation calculation circuit 14, and the selection circuit 15 change depending on the magnitude relationship between d (45) and d (−45). First, operations of the interpolation operation circuit 13, the interpolation operation circuit 14, and the selection circuit 15 when d (45) <d (−45) are described.

  The interpolation calculation circuit 13 calculates and outputs the G component value OUT13 of the target pixel R44 by the following equation.

  The interpolation calculation circuit 14 calculates the G component value OUT14 of the target pixel R44 by the following equation.

However,

  X1, y1, x2, and y2 are calculated in the interpolation calculation circuit 14.

The interpolation calculation circuit 14 further determines whether or not the absolute value of the difference between x1 and x2 is less than or equal to a threshold value. When the absolute value is less than or equal to the threshold value, the R component value OUT14 for the pixel of interest G44 is calculated and output using linear interpolation. . That is, the R component value OUT14 for the target pixel G44 is calculated and output using the color component value of the pixel adjacent to the target pixel according to the following equation.
OUT14 = (B53 + B35) / 2

  The selection circuit 15 is located in a region in the vicinity of the target pixel, is aligned in the direction in which the correlation determination unit 12 determines that the correlation is strong with the target pixel as a center, and has a pixel having the same color component as the target pixel. It is determined whether the color component value of the target pixel changes monotonously in relation to the pixel position. Therefore, for example, the magnitude relationship between the R component values R62, R44, and R26 of the R pixels R62, R44, and R26 is examined. Depending on the result, either OUT13 or OUT14 is output as OUT15. That is, OUT13 is output when R62, R44, and R26 satisfy one of the following conditions 1 and 2, and OUT14 is output otherwise.

Condition 1: R62 <R44 and R26 <R44
Condition 2: R62> R44 and R26> R44

  The above is the operation when d (45) <d (−45). Next, the operation when d (45) <d (−45) is not described.

  The interpolation calculation circuit 13 calculates and outputs the B component value OUT13 of the target pixel R44 by the following equation.

  The interpolation calculation circuit 14 calculates the B component value OUT14 of the target pixel R44 by the following equation.

However,

  X1, y1, x2, and y2 are calculated in the interpolation calculation circuit 14.

The interpolation calculation circuit 14 further determines whether or not the absolute value of the difference between x1 and x2 is equal to or smaller than a threshold value. When the absolute value is equal to or smaller than the threshold value, the B component value OUT14 for the target pixel R44 is calculated and output using linear interpolation. . That is, the B component value OUT14 for the target pixel R44 is calculated and output using the color component value of the pixel adjacent to the target pixel according to the following formula.
OUT14 = (B33 + B55) / 2

  The selection circuit 15 is located in a region in the vicinity of the target pixel, is aligned in a direction in which the correlation determination unit 12 determines that the correlation is strong with the target pixel as a center, and has the same color component as the target pixel. In addition, it is determined whether the color component value of the target pixel changes monotonously in relation to the pixel position. Therefore, for example, the magnitude relationship between the R component values R22, R44, and R66 of the R pixels R22, R44, and R66 is examined. Then, according to the result, either OUT13 or OUT14 is output as OUT15. That is, OUT13 is output when R22, R44, and R66 satisfy one of the following conditions 1 and 2, and OUT14 is output otherwise.

Condition 1: R22 <R44 and R66 <R44
Condition 2: R22> R44 and R66> R44

The above interpolation processing is performed on the B component values at the positions of all R pixels.
The above is the interpolation processing of the B component value at the position of the R pixel.
The R component value interpolation process at the B pixel position is performed in the same manner as the B component value interpolation process at the R pixel position, except that R and B are interchanged.

  Thus, the interpolation process is completed, and all insufficient color component values for all pixels are written in the memory 36. Next, the contents of the memory 36 are output to the outside of the interpolation processing circuit 4 as an interpolation result OUT4. The interpolation result OUT4 is combined with the signal OUT3 obtained by the solid-state imaging device 2 input from the A / D converter 3 by the combining circuit 5, so that the color components of the R, G, and B colors for all the pixels are obtained. Thus, an image signal (full color image) having a uniform number is output.

  As in the first embodiment, the calculation formulas for dH, dV, d (45), and d (−45) are not limited to those described in this example.

  Further, the configuration of the full-color image capturing apparatus according to the second embodiment is not limited to that illustrated in FIG. For example, a circuit that performs white balance processing or a circuit that performs gamma correction may be added.

  The internal configuration of the interpolation processing circuit 4 in the second embodiment is not limited to that shown in FIG. For example, a configuration using a line memory having the number of lines necessary for the interpolation processing instead of the frame memory 11 can be considered.

  In addition, the target of the interpolation processing in the first and second embodiments is not limited to the Bayer type arrangement, and it is sufficient that pixels of a plurality of color components are regularly arranged.

It is a block diagram which shows the structure of the imaging device provided with the pixel signal processing apparatus of Embodiment 1 of this invention. It is a figure which shows a Bayer type pixel arrangement. 2 is a block diagram illustrating a configuration of an interpolation processing circuit according to the first embodiment. FIG. 3 is a flowchart illustrating interpolation processing according to the first embodiment. 3 is a flowchart illustrating interpolation processing according to the first embodiment. 3 is a flowchart illustrating interpolation processing according to the first embodiment. It is a figure which shows the pixel utilized when interpolating G component value in the position of R pixel, and its color component value. (A) And (b) is a figure which shows the pixel arrangement | positioning when centering on the G pixel of an odd number row, and the pixel arrangement | positioning centering on the G pixel of an even number row, respectively, in a Bayer arrangement. It is a figure which shows the pixel utilized when interpolating R component value and B component value in the position of G pixel of an odd-numbered row, and its color component value. It is a figure which shows the pixel utilized when interpolating B component value in the position of R pixel, and its color component value. (A) to (d) show changes in R component values among R pixels R24, R44, and R64 on a coordinate plane in which the horizontal axis corresponds to the position of the R pixel and the vertical axis corresponds to the R component value of each pixel. FIG. It is a figure explaining the concept of the interpolation process supposing the monotone change. It is a figure explaining the concept of the interpolation process supposing the monotone change. It is a figure for demonstrating the concept of the interpolation process supposing the similarity relationship. It is a figure for demonstrating that a preferable result is obtained by the interpolation process on which the monotone change was assumed when a color component value changes gently. It is a figure which shows that a preferable result is obtained by the interpolation process which assumed the similarity relationship when a color component value changes with a short period. It is a block diagram which shows the structure of the interpolation processing circuit 4 of Embodiment 2 of this invention.

Explanation of symbols

  ST12 correlation determination step, ST13a step of determining the mode of change, ST13b step of determining the mode of change, ST14a step of performing an interpolation process assuming a similar relationship, ST14b step of performing an interpolation process assuming a similar relationship, ST17a monotonic change ST17b Step of performing interpolation processing assuming monotone change, ST22 Step of determining change mode, ST23 Step of performing interpolation processing assuming similar relationship, ST26 Interpolation processing assuming monotone change ST32 correlation determination step, ST33a step of determining the change mode, ST33b step of determining the mode of change, ST34a step of performing an interpolation process assuming the similarity relationship, ST34b similarity relationship ST37a Step of performing interpolation processing assuming monotone change, ST37b Step of performing interpolation processing assuming monotone change, 12 Correlation determining circuit, 13 Interpolation operation circuit, 14 Interpolation operation circuit, 15 selection circuit.

Claims (24)

A set of pixel signals representing respective color component values of a plurality of pixels arranged on a two-dimensional plane and each having any of first to Nth (N is an integer of 2 or more) different color components. Based on the Kth (K is any one of 1 to N excluding J) at the position of the pixel of interest having the Jth (J is any one of 1 to N) color component value. In a pixel signal processing method for generating a pixel signal representing a color component value by interpolation,
Determining a mode of a change in a color component value of a pixel that is located in a region near the target pixel and has the same color component as the target pixel and the target pixel;
According to the determination result in the step of determining the mode of change,
The first interpolation calculation step for generating the first interpolation value by performing the interpolation by the first interpolation method or the second interpolation by performing the interpolation by a second interpolation method different from the first interpolation method. An interpolation calculation step for executing any one of the second interpolation calculation steps for generating a value,
The first interpolation calculation step and the second interpolation calculation step are based on the J-th color component value of the pixel of interest and the color component value of a pixel located in a region near the pixel of interest. A pixel signal processing method characterized by performing interpolation. A correlation determination step for determining a direction in which the correlation between pixel signals located in a region in the vicinity of the target pixel is strong;
The step of determining the change mode is a pixel and a target pixel that are located in a region in the vicinity of the target pixel, are aligned in a direction in which the correlation is determined to be strong in the correlation determination step, and have the same color component as the target pixel The pixel signal processing method according to claim 1, wherein a mode of change of the color component value of the pixel is determined.   The first interpolation calculation step and the second interpolation calculation step are located in a region near the pixel of interest and the J-th color component value of the pixel of interest, and are strongly correlated in the correlation determination step. The pixel signal processing method according to claim 2, wherein the interpolation is performed based on color component values of pixels aligned in the determined direction. In the second interpolation calculation step, among the pixels aligned in the direction in which the correlation is determined to be strong in the correlation determination step, the J-th component value of the pixel having the J-th color component is an x component, Two two-dimensional data (x1, y1), (x2, y2) having a value calculated based on the Kth component value of the pixel having the Kth color component as the y component, and the above of the target pixel The following formula (1) using the J-th color component value x0:

4. The pixel signal processing method according to claim 3, wherein the Kth color component value y0 at the position of the target pixel is calculated. The second interpolation calculation step includes:
When the absolute value of the difference between the x components x1 and x2 of the two two-dimensional data is less than a certain threshold value, the color component value of the same color as the target pixel of the pixel located in the region near the target pixel is used. Perform linear interpolation,
The pixel signal processing method according to claim 4, wherein when the absolute value of the difference between the x components x1 and x2 of the two two-dimensional data exceeds the threshold value, interpolation using the equation (1) is performed. . The first interpolation calculation step includes:
In a three-dimensional orthogonal coordinate system in which an arrangement of pixels on a two-dimensional plane is represented on a two-dimensional coordinate plane including an H axis and a V axis orthogonal to each other, and a color component value is represented on a Z axis perpendicular to the two-dimensional plane.
The Jth of the pixels at the respective positions of the plurality of pixels having the Jth color component that are located around the target pixel and aligned in the direction in which the correlation is determined to be strong in the correlation determination step. A first polygon formed by an average value of color component values and a value of the color component of the target pixel at the position of the target pixel;
The Kth of the pixels at the respective positions of the plurality of pixels having the Kth color component aligned in the direction in which the correlation is determined to be strong in the correlation determination step is located around the target pixel. The second polygon formed by the average value of the color components and the value of the Kth color component at the position of the target pixel obtained by interpolation in the second interpolation calculation step is similar to each other.
4. The pixel signal processing method according to claim 3, wherein a value of the Kth color component at the position of the target pixel is determined. The first interpolation calculation step includes:
An average value of the values of the J-th color component of a plurality of pixels having the J-th color component, which are positioned around the target pixel and aligned in the direction in which the correlation is determined to be strong in the correlation determination step; A ratio between a difference between the value of the J-th color component at the position of the target pixel and a distance on the two-dimensional plane between the plurality of pixels used for calculating the average value and the target pixel;
An average value of the values of the K-th color component of a plurality of pixels having the K-th color component, which are positioned around the target pixel and aligned in a direction in which the correlation is determined to be strong in the correlation determination step; The difference between the value of the Kth color component at the position of the target pixel obtained by interpolation in the second interpolation calculation step and the plurality of pixels used for the calculation of the average value and between the target pixels In order that the ratio to the distance on the two-dimensional plane is the same,
4. The pixel signal processing method according to claim 3, wherein a value of the Kth color component at the position of the target pixel is determined.   The step of determining the mode of change is a pixel that is located in a region in the vicinity of the target pixel, is aligned in a direction in which the correlation is determined to be strong in the correlation determination step, and has a color component of the same color as the target pixel It is determined whether or not the color component value of the pixel of interest is changing monotonously in relation to the position of each pixel. If it is determined that the color component value is changing monotonously, 4. The pixel signal processing method according to claim 3, wherein an output is selected and output in the first interpolation calculation step otherwise. In the step of determining the change mode, the color component value of the target pixel is represented by x0, and the two pixels having the same color component as the target pixel are aligned in the direction in which the correlation is determined to be strong in the correlation determination step. When the color component values possessed by x1 and x2, x0, x1, and x2 are the following conditions 1 and 2, that is,
Condition 1: x0> x1 and x0> x2
Condition 2: x0 <x1 and x0 <x2
The pixel signal processing method according to claim 8, wherein it is determined that the color component value changes monotonously when none of the above is satisfied.   A signal obtained from each of the photoelectric conversion elements of the imaging element in which the first to Nth types of photoelectric conversion elements each generating one of the first to Nth color components are arranged on a two-dimensional plane. The pixel signal processing method according to claim 1, wherein at least a part of the set of pixel signals is used. A set of pixel signals representing respective color component values of a plurality of pixels arranged on a two-dimensional plane and each having any of first to Nth (N is an integer of 2 or more) different color components. Based on the Kth (K is any one of 1 to N excluding J) at the position of the pixel of interest having the Jth (J is any one of 1 to N) color component value. In a pixel signal processing apparatus that generates a pixel signal representing a color component value by interpolation,
First interpolation calculation means for generating the first interpolation value by performing the interpolation by the first interpolation method;
Second interpolation calculation means for generating the second interpolation value by performing the interpolation by a second interpolation method different from the first interpolation method;
Any one of the first interpolation value and the second interpolation value is located in a region in the vicinity of the target pixel and has the same color component as the target pixel and the change of the color component value of the target pixel. And selecting means for selecting and outputting
The first interpolation calculation means and the second interpolation calculation means are based on the J-th color component value of the target pixel and the color component value of a pixel located in a region near the target pixel. A pixel signal processing apparatus characterized by performing interpolation. Correlation determining means for determining a direction of strong correlation of pixel signals located in a region in the vicinity of the target pixel;
The selection means is located in a region in the vicinity of the target pixel, aligned in the direction in which the correlation determination means determines that the correlation is strong, and has the same color component as the target pixel and the color component value of the target pixel. 12. The pixel signal processing apparatus according to claim 11, wherein one of the first interpolation value and the second interpolation value is selected and output in accordance with a change mode.   The first interpolation calculation means and the second interpolation calculation means are located in a region in the vicinity of the target pixel and the J-th color component value of the target pixel, and the correlation determination unit has a strong correlation. 13. The pixel signal processing apparatus according to claim 12, wherein the interpolation is performed based on color component values of pixels aligned in the determined direction. The second interpolation calculation means uses the J-th color component value of the pixel having the J-th color component as the x-component among the pixels aligned in the direction in which the correlation determination means determines that the correlation is strong. , Two two-dimensional data (x1, y1), (x2, y2) having the y component as a value calculated based on the Kth color component value of the pixel having the Kth color component, and the pixel of interest The following equation (1) using the above-mentioned J-th color component value x0:

14. The pixel signal processing apparatus according to claim 13, wherein the Kth color component value y0 at the position of the target pixel is calculated. The second interpolation calculation means includes:
When the absolute value of the difference between the x components x1 and x2 of the two two-dimensional data is less than a certain threshold value, the color component value of the same color as the target pixel of the pixel located in the region near the target pixel is used. Perform linear interpolation,
The pixel signal processing device according to claim 14, wherein when the absolute value of the difference between the x components x1 and x2 of the two two-dimensional data exceeds the threshold value, interpolation using the equation (1) is performed. . The first interpolation calculation means includes:
In a three-dimensional orthogonal coordinate system in which an arrangement of pixels on a two-dimensional plane is represented on a two-dimensional coordinate plane including an H axis and a V axis orthogonal to each other, and a color component value is represented on a Z axis perpendicular to the two-dimensional plane.
The Jth of the pixels at the respective positions of the plurality of pixels that are located around the pixel of interest and are aligned in the direction in which the correlation determination unit determines that the correlation is strong and have the Jth color component. A first polygon formed by an average value of color component values of the target pixel and a value of the color component of the target pixel at the position of the target pixel;
The Kth of the pixels at the respective positions of the plurality of pixels that are located around the pixel of interest and are aligned in the direction in which the correlation determination unit determines that the correlation is strong and have the Kth color component. The second polygon formed by the average value of the color components and the value of the Kth color component at the position of the target pixel obtained by interpolation by the second interpolation calculation means are similar to each other. ,
14. The pixel signal processing apparatus according to claim 13, wherein a value of the Kth color component at the position of the target pixel is determined. The first interpolation calculation means includes:
An average value of the values of the J-th color component of a plurality of pixels located around the pixel of interest, aligned in a direction in which the correlation determination unit determines that the correlation is strong, and having the J-th color component; A ratio of a difference between the value of the J-th color component at the position of the target pixel and a distance on the two-dimensional plane between the plurality of pixels used for calculating the average value and the target pixel; ,
An average value of the values of the K-th color component of a plurality of pixels located around the pixel of interest, aligned in the direction in which the correlation determination unit determines that the correlation is strong, and having the K-th color component; The difference between the value of the K-th color component at the position of the target pixel obtained by interpolation by the second interpolation calculation means, and the plurality of pixels used for the calculation of the average value and between the target pixels So that their ratio to the distance on the two-dimensional plane is the same.
14. The pixel signal processing apparatus according to claim 13, wherein a value of the Kth color component at the position of the target pixel is determined.   The selection unit is located in a region in the vicinity of the target pixel, is aligned in a direction in which the correlation determination unit determines that the correlation is strong, and has the same color component as the target pixel and the color component of the target pixel It is determined whether or not the value is changing monotonously in relation to the position of each pixel. If it is determined that the value is changing monotonously, the output of the second interpolation calculation means is the other than that The pixel signal processing apparatus according to claim 13, wherein the output of the first interpolation calculation means is selected and output at the time. The selection means represents the color component value of the pixel of interest by x0, aligns the color component values of the two pixels having the same color component as the pixel of interest, aligned in the direction in which the correlation determination means determines that the correlation is strong. When represented by x1, x2, x0, x1, x2 are the following conditions 1 and 2, that is,
Condition 1: x0> x1 and x0> x2
Condition 2: x0 <x1 and x0 <x2
The pixel signal processing apparatus according to claim 18, wherein the pixel component value is determined to be monotonously changing when none of the above is satisfied.   Each of the photoelectric conversion elements further includes an imaging element in which first to Nth types of photoelectric conversion elements each generating one of the first to Nth color components are arranged on a two-dimensional plane. The pixel signal processing apparatus according to claim 11, wherein a signal obtained from is used as at least part of the set of pixel signals. A set of pixel signals representing respective color component values of a plurality of pixels arranged on a two-dimensional plane and each having any of first to Nth (N is an integer of 2 or more) different color components. Based on this, the Kth color (K is any one of 1 to N excluding J) at the position of the pixel of interest having the Jth color component (J is any one of 1 to N). In a pixel signal processing method for generating a pixel signal representing a component value by interpolation,
Of the pixels that exist in one direction around the pixel of interest,
The color component value of the pixel of the Jth color component is the x component,
Two pieces of two-dimensional data (x1, y1), (x2, y2) having the value calculated based on the color component value of the pixel of the Kth color component as the y component, and the Jth color of the pixel of interest The following formula (1) using the component value x0,

An interpolation step (14) for calculating a Kth color component value y0 at the position of the pixel of interest. A correlation determination step (12) for determining a direction of strong correlation of pixel signals in a region in the vicinity of the target pixel;
In the interpolation calculation step, the J-th color component value of the pixel of interest and the color component value of a pixel existing in the direction of strong correlation determined in the correlation determination step among the pixels in the vicinity of the pixel of interest The pixel signal processing method according to claim 21, wherein the interpolation is performed based on: A set of pixel signals representing respective color component values of a plurality of pixels arranged on a two-dimensional plane and each having any of first to Nth (N is an integer of 2 or more) different color components. Based on this, the Kth color (K is any one of 1 to N excluding J) at the position of the pixel of interest having the Jth color component (J is any one of 1 to N). In a pixel signal processing apparatus that generates a pixel signal representing a component value by interpolation,
Of the pixels that exist in one direction around the pixel of interest,
The color component value of the pixel of the Jth color component is the x component,
Two pieces of two-dimensional data (x1, y1), (x2, y2) having the value calculated based on the color component value of the pixel of the Kth color component as the y component, and the Jth color of the pixel of interest The following formula (1) using the component value x0,

A pixel signal processing apparatus comprising: interpolation means (14) for calculating a Kth color component value y0 at the position of the target pixel. Correlation determining means (12) for determining the direction of strong correlation of pixel signals in the region in the vicinity of the target pixel,
The interpolation calculation means includes the J-th color component value of the pixel of interest and the color component value of a pixel existing in the direction of strong correlation determined by the correlation determination means among the pixels in the vicinity of the pixel of interest. 24. The pixel signal processing apparatus according to claim 23, wherein the interpolation is performed based on:

Images