JP2010081583A - Image signal processing apparatus, imaging apparatus, image signal processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image signal processing apparatus, an imaging apparatus, an image signal processing method and a program, which permit the improvement of signal correction and resolution of a defective pixel, further, which is excellent in color reproductivity. <P>SOLUTION: An interpolation processing unit 141 carries out interpolating processing of respective first color detecting signal, second color detecting signal and a third color detecting signal employing a detecting signal CDT4 for color interpolation, a first color detecting signal CDT1, a second color detecting signal CDT2 and a third color detecting signal CDT3 to produce a plurality of signals X1, X2, Y1, Y2, Z1, Z2 with respect to respective color detecting signals while a composition processing unit 142 synthesizes the plurality of output signals outputted from the interpolation processing unit to produce a first color signal CX, a second color signal CY and a third color signal CZ whereby the defective of picture element is compensated and signal strength of double precision can be obtained. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image signal processing apparatus that processes a color detection signal by a solid-state image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and an imaging apparatus. The present invention particularly relates to an image signal processing device, an imaging device, an image signal processing method, and a program for processing a readout signal of a two-dimensional pixel array using a plurality of color filters.

As an array of color filters of the image sensor, a Bayer array using two green (G1, G2), one red (R), and one blue (B) among the three primary colors with good color reproducibility. It has been known.
This Bayer array is an array in which luminance resolution is more important than color.

By the way, in such a color filter array, an imaging apparatus having a pixel array in which a transparent filter is arranged so as to increase sensitivity while maintaining good color reproducibility has been proposed (for example, see Patent Document 1). ).
This pixel unit is composed of four pixels of R, G, B, and white (W) that does not have a color filter.
This white pixel is mainly used for a luminance signal.

  In recent years, a technique for improving the S / N ratio by multiplying a white signal by a ratio of signals obtained from R, G, and B pixels of peripheral elements to generate R, G, and B signals has been proposed. (See Patent Document 2).

USP6477685B1 specification JP 2008-22521 A JP 2007-329380 A

  However, in the technique disclosed in Patent Document 2 described above, the R, G, B ratio around the white pixel is obtained from the average of the signals of several pixels around the pixel, so there is a concern that the resolution may be lowered.

As miniaturization progresses and pixel pixels become smaller, there arises a problem that electrical noise generated by thermal fluctuations or the like cannot be ignored with respect to an electrical signal generated by the incidence of light.
This problem can be improved by increasing the ratio of the entire pixel, but the area of the pixel is naturally limited because the area of the image sensor is the upper limit.

On the other hand, as miniaturization progresses and the number of pixels mounted on one image sensor increases, the number of defects such as white spots generated per image sensor tends to increase, and the number of pixels increases. However, a technique for suppressing the number of white spots is desired.
Currently, a single image sensor cannot obtain a sufficiently large signal intensity, and thus a plurality of elements are used as one unit, but only an image with a resolution smaller than the number of pixels can be obtained. There is a problem that it is difficult to take cost merit.

  As a technique for obtaining a high-resolution color signal, a method of interpolating colors from adjacent pixels has been proposed (see, for example, Patent Document 3).

  However, with this technique, the algorithm is complicated and it is difficult to obtain high color reproducibility because the color signal is interpolated using a technique such as linear interpolation.

  An object of the present invention is to provide a technique for complementing pixel defects with an algorithm, and to provide an image signal processing device, an imaging device, an image signal processing method, and a program capable of improving resolution with a simple algorithm.

  An image signal processing apparatus according to a first aspect of the present invention includes an interpolation processing unit including a function of performing interpolation processing of a plurality of input color detection signals and generating a plurality of signals for each color detection signal, and the interpolation processing unit Each of the output signals is combined to generate a color signal, and the interpolation processing unit includes a first color detection element from the first color detection element having transparency in the wavelength range of the first electromagnetic wave. The first color detection signal, the second color detection signal from the second color detection element having transparency in the wavelength range of the second electromagnetic wave, and the third color detection having transparency in the wavelength range of the third electromagnetic wave A third color detection signal from the element, and a color interpolation detection signal from a color interpolation element having transparency in a wavelength range including at least a part of the first, second, and third wavelength ranges. And input the color interpolation detection signal above. Using the first color detection signal, the second color detection signal, and the third color detection signal, the first color detection signal, the second color detection signal, and the third color detection signal Each interpolation processing is performed to generate a plurality of signals for each color detection signal, and the synthesis processing unit synthesizes a plurality of output signals of each color detection signal output from the interpolation processing unit to obtain the first color A signal, a second color signal, and a third color signal are generated.

  Preferably, the interpolation processing unit, for one interpolation target color detection signal, uses the interpolation target color detection signal as a first output signal, and outputs the interpolation target color detection signal from the color interpolation detection signal. Subtracting the two color detection signals except the two to generate a second output signal, and the synthesis processing unit synthesizes the first output signal and the second output signal output from the interpolation processing unit. A color signal is generated.

  Preferably, the interpolation processing unit removes the first color detection signal from the color interpolation detection signal with the first color detection signal as the first output signal for the first color detection signal. If both the two second and third color detection signals are greater than or equal to a predetermined value, they are subtracted to generate a second output signal. If either of the two color detection signals is less than or equal to the predetermined value, the value is 0. A second output signal is generated, and for the second color detection signal, the second color detection signal is used as a third output signal, and the second color detection signal is removed from the color interpolation detection signal 2 If both of the first and third color detection signals are equal to or greater than a predetermined value, they are subtracted to generate a fourth output signal. 4 output signal, and for the third color detection signal, the third color detection signal As the fifth output signal, when both of the two first and second color detection signals excluding the third color detection signal from the color interpolation detection signal are equal to or larger than a predetermined value, they are subtracted to obtain the sixth output signal. The output signal is generated, and if one of the two color detection signals is equal to or less than a predetermined value, a sixth output signal having a value of 0 is generated, and the synthesis processing unit outputs the first output signal output from the interpolation processing unit. The output signal and the second output signal are combined to generate a first color signal, and the third output signal output from the interpolation processing unit and the fourth output signal are combined to generate a second color signal. A color signal is generated, and the fifth output signal output from the interpolation processor and the sixth output signal are combined to generate a third color signal.

  Preferably, when the first output signal and the second output signal have a predetermined level, the synthesis processing unit adds the first output signal and the second output signal to generate a color signal. When only one of the first output signal and the second output signal has a predetermined level, the signal obtained by adding the first output signal and the second output signal is multiplied by a value greater than 1 When the color signal is generated and both the first output signal and the second output signal have not reached the predetermined level, the color signal is generated using the color detection signal around the corresponding color detection signal. Generate.

  Preferably, when the first output signal and the second output signal have a predetermined level when generating the first color signal, the synthesis processing unit generates the first color signal and the second output signal. Output signals are added to generate a first color signal, and when only one of the first output signal and the second output signal has a predetermined level, the first output signal and the second output signal When the signal obtained by adding the output signals is multiplied by a value greater than 1 to generate a first color signal, both the first output signal and the second output signal have not reached a predetermined level. When the first color signal is generated using the color detection signal around the color detection signal to be generated, and the second color signal is generated, the third output signal and the fourth output signal have predetermined levels. And adding the third output signal and the fourth output signal to obtain the second When a color signal is generated and only one of the third output signal and the fourth output signal has a predetermined level, it is greater than 1 by adding the third output signal and the fourth output signal. The second color signal is generated by multiplying the values, and when both the third output signal and the fourth output signal have not reached a predetermined level, the color detection signal around the corresponding color detection signal Is used to generate a third color signal, and when the third color signal is generated, when the fifth output signal and the sixth output signal have predetermined levels, the fifth output signal When the sixth output signal is added to generate a third color signal, and only one of the fifth output signal and the sixth output signal has a predetermined level, the fifth output signal and the fifth output signal The signal obtained by adding the output signal of 6 is multiplied by a value greater than 1 to obtain the third A color signal is generated, and when both the fifth output signal and the sixth output signal have not reached a predetermined level, the third color signal is generated using the color detection signal around the corresponding color detection signal. Is generated.

  Preferably, the first color detection signal, the second color detection signal, and the third color detection signal are weighted according to a light attenuation rate and supplied to the interpolation processing unit. It further has a part.

  Preferably, the interpolation processing unit and the synthesis processing unit, when the color interpolation detection signal is out of an allowable range, the first color detection signal, the second color detection signal, and the third color detection signal. The color detection signal is directly processed as a color signal.

  An image pickup apparatus according to a second aspect of the present invention includes a plurality of pixels having different wavelength ranges to be transmitted arranged in an array, and converts the light transmitted through the pixels into an electric signal, and a plurality of color detection signals and color interpolation signals. A sensor unit including a plurality of pixel units for outputting detection signals, and an image signal processing device having a function of performing interpolation and synthesis processing on the plurality of color detection signals and color interpolation detection signals by the sensor unit. The image signal processing apparatus performs an interpolation process on a plurality of color detection signals output from the sensor unit, and includes an interpolation processing unit including a function of generating a plurality of signals for each color detection signal, and the interpolation processing unit A synthesis processing unit that synthesizes each output signal and generates each color signal, and the pixel unit of the sensor unit has transparency in the wavelength range of the first electromagnetic wave, and the first electromagnetic wave Detect A first color detection element that generates one color detection signal and a second color detection element that is transparent in the wavelength range of the second electromagnetic wave and detects the second electromagnetic wave to generate a second color detection signal. A color detection element, a third color detection element that is transparent in the wavelength range of the third electromagnetic wave, detects the third electromagnetic wave and generates a third color detection signal, and the first, Color interpolation having transparency in a fourth wavelength range including at least a part of the second and third wavelength ranges and generating a color interpolation detection signal by detecting electromagnetic waves in the fourth wavelength range And the interpolation processing unit uses the color interpolation detection signal, the first color detection signal, the second color detection signal, and the third color detection signal by the sensor unit. , The first color detection signal, the second color detection signal, and the third color detection signal. Each interpolation process is performed to generate a plurality of signals for each color detection signal, and the synthesis processing unit synthesizes a plurality of output signals of each color detection signal output from the interpolation processing unit, respectively. The second color signal, the second color signal, and the third color signal are generated.

  The image signal processing method according to the third aspect of the present invention is the first color detection signal from the first color detection element having transparency in the wavelength range of the first electromagnetic wave, and the transmission in the wavelength range of the second electromagnetic wave. A second color detection signal from the second color detection element having the third property, a third color detection signal from the third color detection element having the transmission property in the wavelength range of the third electromagnetic wave, and the first, In the wavelength range including at least a part of the second and third wavelength ranges, the first color detection signal and the second color detection signal are transmitted using the color interpolation detection signal from the color interpolation element having transparency. Each of the color detection signal and the third color detection signal is generated, a plurality of signals are generated for each color detection signal, and a plurality of signals of the generated color detection signals are respectively combined to generate a first signal. Color signal, a second color signal, and Generating a third color signal.

  According to a fourth aspect of the present invention, the first color detection signal from the first color detection element having transparency in the wavelength range of the first electromagnetic wave, and the second having transparency in the wavelength range of the second electromagnetic wave. A second color detection signal from the first color detection element, a third color detection signal from the third color detection element having transparency in the wavelength range of the third electromagnetic wave, and the first, second, and second color detection signals. In the wavelength range including at least a part of the wavelength range of 3, using the color interpolation detection signal from the transparent color interpolation element, the first color detection signal, the second color detection signal, The first color signal is obtained by interpolating each of the third color detection signals, generating a plurality of signals for each color detection signal, and combining the plurality of signals of the generated color detection signals. , Second color signal, and third color signal Is a program for executing the image signal processing including a process of generating a computer.

According to the present invention, for example, in the case of blue, in addition to the light reception signal from the blue color filter, a blue signal can be obtained separately by subtracting the red and green signals from the white signal.
The calculation is Blue = W + B / b (B) -R / r (R) -G / g (G) (r, g, and b are light transmittances by the color filter). Synthesized. Other colors are also generated by the same calculation.

According to the present invention, it is possible to realize color interpolation of defective pixels with higher accuracy than usual, and it is possible to realize an imaging apparatus capable of improving resolution with a simple algorithm and having excellent color reproducibility.
Further, the present invention can generate colors that are close to nature according to brightness.

It is a block diagram which shows the structural example of the principal part of the imaging device which concerns on embodiment of this invention. It is a circuit diagram which shows an example of the unit pixel of this embodiment. It is a figure which shows typically an example of the pixel unit of the pixel array part of this embodiment. 3 is a diagram conceptually showing spectral characteristics of each color filter pixel R, G, B and white pixel W. FIG. It is a block diagram which shows the structural example of the signal interpolation synthetic | combination process part which concerns on the 1st embodiment. 6 is a first flowchart for explaining the operation of the imaging apparatus having the first pixel units of R, G, B, and W according to the first embodiment. FIG. 6 is a second flowchart for explaining the operation of the imaging apparatus having the first pixel units of R, G, B, and W according to the first embodiment. FIG. 10 is a third flowchart for explaining the operation of the imaging apparatus having the first R, G, B, and W pixel units of the first embodiment. FIG. 10 is a fourth flowchart for explaining the operation of the imaging apparatus having the first pixel units of R, G, B, and W according to the first embodiment. It is a figure which shows typically the other example of the pixel unit of the pixel array part of the 1st embodiment. FIG. 6 is a first flowchart for explaining the operation of the imaging apparatus having the second pixel unit of Cy, Mg, Y, and W according to the first embodiment. FIG. 6 is a second flowchart for explaining the operation of the imaging apparatus having the second pixel unit of Cy, Mg, Y, and W according to the first embodiment. It is a 3rd flowchart in order to explain operation of an imaging device which has the 2nd pixel unit of Cy, Mg, Y, and W of a 1st embodiment. FIG. 14 is a fourth flowchart for explaining the operation of the imaging apparatus having the second pixel unit of Cy, Mg, Y, and W according to the first embodiment. It is a block diagram which shows the structural example of the signal interpolation synthetic | combination process part which concerns on the 2nd embodiment. It is a figure which shows the 1st example of the basic algorithm which concerns on the 2nd embodiment. It is a figure which shows the 2nd example of the basic algorithm which concerns on the 2nd embodiment. It is a figure which shows the 3rd example of the basic algorithm which concerns on the 2nd embodiment. It is a figure which shows the 4th example of the basic algorithm which concerns on the 2nd embodiment. It is a figure for demonstrating the coordinate interpolation method of pixel data. It is a figure which shows the coordinate interpolation method shown as a comparative example. It is a 1st flowchart for demonstrating the 1st process which concerns on the 2nd embodiment. It is a 2nd flowchart for demonstrating the 1st process which concerns on the 2nd embodiment. It is a 3rd flowchart for demonstrating the 1st process which concerns on the 2nd embodiment. It is a 4th flowchart for demonstrating the 1st process which concerns on the 2nd embodiment. It is a 1st flowchart for demonstrating the 2nd process which concerns on the 2nd embodiment. It is a 2nd flowchart for demonstrating the 2nd process which concerns on the 2nd embodiment. It is a 3rd flowchart for demonstrating the 2nd process which concerns on this 2nd Embodiment. It is a 4th flow chart for explaining the 2nd processing concerning this 2nd embodiment. It is a figure which shows the other structural example of the imaging device which concerns on this embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
The description will be given in the following order.
1. First Embodiment 2. FIG. Second embodiment

<1. First Embodiment>
FIG. 1 is a block diagram illustrating a configuration example of a main part of an imaging apparatus according to an embodiment of the present invention.

As illustrated in FIG. 1, the imaging apparatus 100 includes an optical lens system 110, an image sensor unit 120, a weighting unit 130, a signal interpolation / synthesis processing unit 140, and a signal processing unit 150.
The weighting unit 130, the signal interpolation / synthesis processing unit 140, and the signal processing unit 150 constitute an image signal processing apparatus.

  The optical lens 110 forms a subject image on the imaging surface of the image sensor of the image sensor unit 120.

The image sensor unit 120 includes an image sensor that converts image light of a subject formed through the optical lens system 110 into an analog electric signal.
The image sensor unit 120 converts this analog electric signal into a digital signal by an analog / digital (AD) converter (not shown), and finally outputs it as digital raw (RAW) image data.

The image sensor unit 120 includes, for example, a pixel array unit in which sensor pixel units are arrayed in a predetermined array form.
In the pixel array portion, a transfer selection line, a reset line, and a select line are wired in each row (row) of the pixel array, and a signal line is wired in each column (column) of the pixel array.

  FIG. 2 is a circuit diagram showing an example of the pixel of this embodiment. In FIG. 2, a CMOS sensor is shown as an example.

  2 includes a photodiode 121, a transfer transistor 122, an amplifying transistor 123, a select transistor 124, a reset transistor 125, and a floating diffusion (FD) node ND121.

The photodiode 121 photoelectrically converts incident light into signal charges (for example, electrons) having a charge amount corresponding to the amount of light, and accumulates the signal light.
The transfer transistor 122 is connected between the cathode of the photodiode 121 and the FD node ND121, and the gate is connected to the transfer selection line TRFL. The transfer transistor 122 is turned on to transfer the signal charge stored in the photodiode 121 to the FD node ND121.
The amplification transistor 123 and the select transistor 124 are connected in series between the power supply potential VDD and the signal line SGNL.
The amplifying transistor 123 has a gate connected to the FD node ND121, amplifies the potential of the FD node ND121, and outputs it to the signal line SGNL via the select transistor 124.
The gate of the select transistor 124 is connected to the select line SELL.
The reset transistor 125 has a source connected to the FD node ND121, a drain connected to a predetermined potential line, a gate connected to the reset line RSTL, and a function of resetting the potential of the FD node ND121.

The transfer selection line TRFL, select line SELL, and reset line RSTL wired to each row of the pixel array are selectively driven by the vertical scan circuit (VSCN) 126, and the signal line SGNL is supplied from the pixel to the horizontal scan circuit (HSCN) 127. The read signal is selectively transferred.
The drive timing of the vertical scan circuit 126 and the horizontal scan circuit 127 is controlled by a timing control unit (TC) 128.

  3A and 3B are diagrams schematically illustrating an example of a pixel unit of the pixel array unit of the present embodiment.

The pixel unit 200 in FIG. 3A is formed by three color detection elements 201, 202, 203 and a color interpolation element 204 arranged in a so-called square pattern.
The first color detection element 201 has transparency in the wavelength range of the first electromagnetic wave, detects the first electromagnetic wave, and generates a first color detection signal CDT1.
The second color detection element 202 has transparency in the wavelength range of the second electromagnetic wave, and detects the second electromagnetic wave to generate a second color detection signal CDT2.
The third color detection element 203 has transparency in the wavelength range of the third electromagnetic wave, and detects the third electromagnetic wave to generate a third color detection signal CDT3.
The color interpolation element 204 is transparent in a fourth wavelength range including at least a part of the first, second, and third wavelength ranges, and detects an electromagnetic wave in the fourth wavelength range to detect color. An interpolation detection signal CDT4 is generated.

In the present embodiment, the first color detection element 201 is formed by a color filter pixel R having a spectral sensitivity characteristic peak including a color filter in red (R).
The second color detection element 202 is formed by a color filter pixel G having a spectral sensitivity characteristic peak including a color filter in green (G).
The third color detection element 203 is formed by a color filter pixel B having a spectral sensitivity characteristic peak including a color filter in blue (B).
The color interpolation element 204 is formed by white (white) pixels W having a high spectral transmittance that includes R, G, and B colors that do not include a color filter.
Note that the color interpolation element 204 is not necessarily white.

  FIG. 3A shows an example of pixel units arranged in a square, but the present invention also includes pixel units having various arrangements, for example, a so-called honeycomb structure as shown in FIG. Applicable.

FIG. 4 is a diagram conceptually showing the spectral characteristics of the color filter pixels R, G, B and the white pixel W.
In FIG. 4, the horizontal axis indicates the wavelength and the vertical axis indicates the relative output.
As can be seen from FIG. 4, the white pixel W has sensitivity over substantially the entire visible light region (wavelength 360 nm to 700 nm).
That is, since the white pixel W has a wide wavelength region component, it is easy to reproduce the color at the boundary with the clear pixel (all color signals are included).

The first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3 generated by the image sensor unit 120 are supplied to the weighting unit 130.
On the other hand, the color interpolation detection signal CDT 4 generated by the image sensor unit 120 is supplied to the signal interpolation synthesis processing unit 140.

The weighting unit 130 weights the first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3, which are RAW data by the image sensor unit 120, according to the light attenuation rate. To the signal interpolation synthesis processing unit 140.
The weighting unit 130 performs weighting using a weighting coefficient g in consideration of the light attenuation rate by the color filter in the image sensor unit 120.

The light transmittance in the color filter is obtained by inputting R, G, B monochromatic light as a signal, and obtained from the ratio of the output signal from the monochromatic pixel (pixel) and the output signal from the white (white) pixel, and the weighting unit 130. Holds the transmittance of each color as a table.
The R color transmittance r (R) is given by R / W, the G color transmittance g (G) is given by G / W, and the B color transmittance b (B) is given by B / W.
A table that changes in accordance with the output of a monochrome pixel can also be applied.

The signal interpolation synthesis processing unit 140 performs interpolation processing of a plurality of input color detection signals, generates a plurality of signals for each color detection signal, synthesizes each generation signal, generates each color signal, and generates each color The signal is output to the signal processing unit 150.
When the color interpolation detection signal CDT4 is out of the allowable range, the signal interpolation synthesis processing unit 140 applies the color of the first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3 as they are. It has a function of processing as a signal.
When the upper limit threshold of the allowable range of the color interpolation detection signal is W1 and the lower limit threshold is W2 (W1> W2) in a certain pixel, the signal interpolation synthesis processing unit 140 performs the following processing.
When W> W1 and W <W2, the signal interpolation / synthesis processing unit 140 processes the first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3 as they are as color signals. To do.

  FIG. 5 is a block diagram illustrating a configuration example of the signal interpolation / synthesis processing unit 140 according to the first embodiment.

  As shown in FIG. 5, the signal interpolation / synthesis processing unit 140 includes an interpolation processing unit 141 and a synthesis processing unit 142.

The interpolation processing unit 141 uses the color interpolation detection signal CDT4 from the image sensor unit 120 to perform interpolation processing on each of the first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3. A plurality of signals are generated for each color detection signal.
The interpolation processing unit 141 generates signals X1 and X2 for the first color detection signal CDT1, generates signals Y1 and Y2 for the second color detection signal CDT2, and signals Z1 and Z2 for the third color detection signal CDT3. Is generated.
The interpolation processing unit 141 outputs the generated signals X1 and X2, Y1 and Y2, and Z1 and Z2 to the synthesis processing unit 142.

Specifically, for the first color detection signal CDT1, the interpolation processing unit 141 uses the first color detection signal CDT1 as it is as the first output signal X1, and the following processing is performed when Y1 and Z1 are equal to or larger than a predetermined value. I do.
The interpolation processing unit 141 subtracts the second and third color detection signals excluding the first color detection signal from the color interpolation detection signal CDT4 to generate the second output signal X2. When either Y1 or Z1 is less than or equal to a predetermined value, X2 is zero.
The second output signal X2 is given by the following equation.

[Equation 1]
X2 = W-gy * Y-gz * Z

  When weighting by the weighting unit 130 is not performed, for example, for R, X2 is simply given by X2 = W−G−B.

The interpolation processing unit 141 performs the next process on the second color detection signal CDT2 when the second color detection signal CDT2 is used as it is as the third output signal Y1 and X1 and Z1 are equal to or larger than a predetermined value.
The interpolation processing unit 141 subtracts the first and third color detection signals excluding the second color detection signal from the color interpolation detection signal CDT4 to generate a fourth output signal Y2. When either X1 or Z1 is equal to or less than a predetermined value, Y2 is set to zero.
The fourth output signal Y2 is given by the following equation.

[Equation 2]
Y2 = W-gx * X-gz * Z

  When weighting by the weighting unit 130 is not performed, for example, for G, Y2 is simply given by Y2 = W−R−B.

The interpolation processing unit 141 performs the following processing on the third color detection signal CDT3 when the third color detection signal CDT3 is used as it is as the fifth output signal Z1 and X1 and Y1 are equal to or larger than a predetermined value.
The interpolation processing unit 141 subtracts the first and second color detection signals excluding the third color detection signal from the color interpolation detection signal CDT4 to generate a sixth output signal Z2. If either X1 or Y1 is less than or equal to a predetermined value, Z2 is zero.
The sixth output signal Z2 is given by the following equation.

[Equation 3]
Z2 = W-gx * X-gy * Y

  When weighting by the weighting unit 130 is not performed, for example, for B, Z2 is simply given by Z2 = WRG.

  The synthesis processing unit 142 includes the first output signal X1 and the second output signal X2, the third output signal Y1 and the fourth output signal Y2, and the fifth output signal Z1 and the sixth output signal from the interpolation processing unit 141. The output signals Z2 are combined and output to the signal processing unit 150.

The synthesis processing unit 142 synthesizes the first output signal X1 and the second output signal X2, and outputs the first color signal CX that is the synthesized signal X1 + X2 to the signal processing unit 150.
The combined signal X1 + X2 is given by the following equation.

[Equation 4]
X1 + X2 = W + X1-gy * Y-gz * Z

When weighting by the weighting unit 130 is not performed, for R, for example, when all the RGB pixels related to interpolation and synthesis are valid, the synthesized signal X1 + X2 is simply given by [X1 + X2 = W + R−G−B] A color signal CX is generated.
When the R pixel involved in interpolation and synthesis is defective and is at a level that is difficult to use in the interpolation process, for example, 0, the synthesis processing unit 142 outputs [R = W−G−B], the first color signal CX is generated.
When only one of the first output signal X1 and the second output signal X2 has a predetermined level, the synthesis processing unit 142 adds 1 to the signal obtained by adding the first output signal X1 and the second output signal X2. The first color signal CX is generated by multiplying a large value, for example, 2.
When two or more pixels involved in interpolation and synthesis are defective and are at a level that is difficult to use for interpolation processing, for example, 0, the synthesis processing unit 142 outputs a normal peripheral signal as an R signal to be output to the signal processing unit 150 The first color signal CX is generated by interpolation using the color detection signal.

The synthesis processing unit 142 synthesizes the third output signal Y1 and the fourth output signal Y2, and outputs a color signal CY that is the synthesized signal Y1 + Y2 to the signal processing unit 150.
The combined signal Y1 + Y2 is given by the following equation.

[Equation 5]
Y1 + Y2 = W + Y1-gx * X-gz * Z

When weighting by the weighting unit 130 is not performed, for G, for example, when all of the RGB pixels related to interpolation and synthesis are valid, the synthesized signal Y1 + Y2 is simply given as [Y1 + Y2 = W + G−R−B]. A color signal CY is generated.
When the G pixel involved in the interpolation and synthesis is defective and is at a level that is difficult to use for the interpolation processing, for example, 0, the synthesis processing unit 142 uses the G signal output to the signal processing unit 150 as [G = W−R−B] is generated as a second color signal CY.
When only one of the third output signal Y1 and the fourth output signal Y2 has a predetermined level, the synthesis processing unit 142 adds 1 to the signal obtained by adding the third output signal Y1 and the fourth output signal Y2. The second color signal CY is generated by multiplying a large value, for example, 2.
When two or more pixels related to interpolation and synthesis are defective and the level is difficult to use for interpolation processing, for example, 0, the synthesis processing unit 142 outputs a normal peripheral signal as a G signal to be output to the signal processing unit 150. A second color signal CY is generated by interpolation processing using the color detection signal.

The synthesis processing unit 142 synthesizes the fifth output signal Z1 and the sixth output signal Z2, and outputs a color signal CZ that is the synthesized signal Z1 + Z2 to the signal processing unit 150.
The combined signal Z1 + Z2 is given by the following equation.

[Equation 6]
Z1 + Z2 = W + Z1-gx * X-gy * Y

When weighting by the weighting unit 130 is not performed, for example, for B, when all of the RGB pixels related to interpolation and synthesis are valid, the synthesized signal Z1 + Z2 is simply given by [Z1 + Z2 = W + B−R−G] A color signal CZ is generated.
When the B pixel involved in interpolation and synthesis is defective and is at a level that is difficult to use in the interpolation process, for example, 0, the synthesis processing unit 142 uses [G = W−R−G] is generated as a third color signal CZ.
When only one of the fifth output signal Z1 and the sixth output signal Z2 has a predetermined level, the synthesis processing unit 142 adds 1 to the signal obtained by adding the fifth output signal Z1 and the sixth output signal Z2. The third color signal CZ is generated by multiplying a large value, for example, 2.
When two or more pixels related to interpolation and synthesis are defective and the level is difficult to use for interpolation processing, for example, 0, the synthesis processing unit 142 outputs a normal peripheral signal as a B signal to the signal processing unit 150. A third color signal CZ is generated by interpolation processing using the color detection signal.

  The signal processing unit 150 receives the color signals CX, CY, and CZ from the signal interpolation / synthesis processing unit 140 and performs processing such as white balance and compression.

[Operation of Imaging Device Having R, G, B, W Pixel Units]
Next, the operation according to the above configuration will be described.
6 to 9 are flowcharts for explaining the operation of the imaging apparatus having the R, G, B, and W pixel units according to the first embodiment.

<Step ST1>
In step ST1, the weighting unit 130 weights the signal extracted from the R, G, B RAW pixel data by the image sensor unit 120 using the weighting coefficient g in consideration of the light attenuation rate by the color filter. .
The weighting unit 130 weights the first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3, which are RAW data from the image sensor unit 120.
The first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3 that have undergone the weighting process are supplied to the interpolation processing unit 141 of the signal interpolation synthesis processing unit 140.
The color interpolation detection signal CDT4 from the image sensor unit 120 is supplied to the interpolation processing unit 141 without passing through the weighting unit 130.

<Step ST2>
In step ST2, the signal interpolation composition processing unit 140 determines whether the magnitude (level) of the color interpolation detection signal CDT4 indicating the white signal W is greater than or equal to the threshold value W1 (W> W1) or less than the threshold value W2 (W <W2). Determine whether or not.

<Step ST3>
If it is determined in step ST2 that the magnitude (level) of the color interpolation detection signal CDT4 is greater than or equal to the threshold value W1 (W> W1) or less than or equal to the threshold value W2 (W <W2), the signal interpolation synthesis processing unit 140 performs step ST3. The generated color signal is processed as a color signal as it is.

<Step ST4>
If it is determined in step ST2 that the magnitude (level) of the color interpolation detection signal CDT4 is equal to or less than the threshold value W1 and equal to or greater than the threshold value W2 (W2 <W <W1), the interpolation processing unit 141 is targeted in step ST4. Specify colors X, Y, and Z.
Then, the interpolation processing unit 141 identifies the target color X and performs the process of FIG.
The interpolation processing unit 141 identifies the target color Y and performs the process of FIG.
The interpolation processing unit 141 identifies the target color Z and performs the process of FIG.
The interpolation processing unit 141 specifies, for example, X as R, Y as G, and Z as B.

<Step ST5>
When the target color X is specified and the colors other than X are set to Y and Z, the interpolation processing unit 141 determines whether or not the X signal is equal to or higher than a preset threshold value VTHX in step ST5 of FIG. To do.

<Step ST6>
If it is determined in step ST5 that the X signal is equal to or greater than the preset threshold value VTHX, the interpolation processing unit 141 uses the weighted X signal in step ST6 as it is as the first output signal X1 and sets its value to gx * X.

<Step ST7>
If it is determined in step ST5 that the X signal is not equal to or greater than the preset threshold value VTHX, the interpolation processing unit 141 sets the first output signal X1 to 0 in step ST7.

<Step ST8>
In step ST8, the interpolation processing unit 141 determines whether the colors Y and Z other than X are greater than or equal to the threshold values VTHY and VTHZ.

<Step ST9>
If it is determined in step ST8 that the colors Y and Z other than X are greater than or equal to the threshold values VTHY and VTHZ, the interpolation processing unit 141 calculates the second output signal X2 to [W-gy * Y-gz * Z] by calculation in step ST9. ].

<Step ST10>
If it is determined in step ST8 that the colors Y and Z other than X are not greater than or equal to the threshold values VTHY and VTHZ, the interpolation processing unit 141 sets the second output signal X2 to 0 in step ST10.

  The interpolation processing unit 141 outputs the obtained first output signal X1 and second output signal X2 to the synthesis processing unit 142.

<Step ST11>
In step ST11, the synthesis processing unit 142 receives the first output signal X1 and the second output signal X2 from the interpolation processing unit 141, and both signal levels of the first output signal X1 and the second output signal X2 are set. It is determined whether or not it is greater than zero.

<Step ST12>
If it is determined in step ST11 that the first output signal X1 and the second output signal X2 are greater than 0, the synthesis processing unit 142 performs the following processing.
That is, the synthesis processing unit 142 obtains a synthesized signal X1 + X2 obtained by adding the first output signal X1 and the second output signal X2 in step ST12, and outputs the synthesized signal X1 + X2 as the first color signal CX.

<Step ST13>
If it is determined in step ST11 that the first output signal X1 and the second output signal X2 are not greater than 0, the synthesis processing unit 142 determines that the first output signal X1 and the second output signal X2 are 0 in step ST13. It is determined whether or not there is.

<Step ST14>
If it is determined in step ST13 that only one of the first or second output signals X1 and X2 is 0, the synthesis processing unit 142 performs the following processing.
That is, the synthesis processing unit 142 obtains a synthesized signal [2 * (X1 + X2)] obtained by adding the first output signal X1 and the second output signal X2 in step ST14, and uses this synthesized signal as the first color signal CX. Output.

<Step ST15>
If it is determined in step ST13 that both the first and second output signals X1 and X2 are 0, the synthesis processing unit 142 interpolates or extrapolates the signal from the surrounding signal X in step ST16. X is calculated and output as the first color signal CX.

<Step ST16>
When the target color Y is specified and the colors other than Y are X and Z, the interpolation processing unit 141 determines whether or not the Y signal is greater than or equal to a preset threshold value VTHY in step ST16 of FIG. .

<Step ST17>
If it is determined in step ST16 that the Y signal is greater than or equal to the preset threshold value VTHY, the interpolation processing unit 141 uses the weighted Y signal in step ST17 as it is as the third output signal Y1, and sets its value to gy * Y.

<Step ST18>
If it is determined in step ST16 that the Y signal is not equal to or greater than the preset threshold value VTHY, the interpolation processing unit 141 sets the third output signal Y1 to 0 in step ST18.

<Step ST19>
In step ST19, the interpolation processing unit 141 determines whether the colors X and Z other than Y are equal to or higher than the threshold values VTHX and VTHZ.

<Step ST20>
If it is determined in step ST19 that the colors X and Z other than Y are greater than or equal to the threshold values VTHX and VTHZ, the interpolation processing unit 141 calculates the fourth output signal Y2 by [W−gx * X−gz * Z] in step ST20. ].

<Step ST21>
If it is determined in step ST19 that the colors X and Z other than Y are not equal to or higher than the threshold values VTHX and VTHZ, the interpolation processing unit 141 sets the fourth output signal Y2 to 0 in step ST21.

  The interpolation processing unit 141 outputs the obtained third output signal Y1 and fourth output signal Y2 to the synthesis processing unit 142.

<Step ST22>
In step ST22, the synthesis processing unit 142 receives the third output signal Y1 and the fourth output signal Y2 from the interpolation processing unit 141, and both signal levels of the third output signal Y1 and the fourth output signal Y2 are set. It is determined whether or not it is greater than zero.

<Step ST23>
If it is determined in step ST22 that the third output signal Y1 and the fourth output signal Y2 are greater than 0, the synthesis processing unit 142 performs the following processing.
That is, the synthesis processing unit 142 obtains a synthesized signal Y1 + Y2 obtained by adding the third output signal Y1 and the fourth output signal Y2 in step ST23, and outputs this synthesized signal Y1 + Y2 as the second color signal CY.

<Step ST24>
If it is determined in step ST22 that the third output signal Y1 and the fourth output signal Y2 are not greater than 0, the synthesis processing unit 142 determines that the third output signal Y1 and the fourth output signal Y2 are 0 in step ST24. It is determined whether or not there is.

<Step ST25>
If it is determined in step ST24 that only one of the third or fourth output signals Y1 and Y2 is 0, the synthesis processing unit 142 performs the following processing.
That is, the synthesis processing unit 142 obtains a synthesized signal [2 * (Y1 + Y2)] obtained by adding the third output signal Y1 and the fourth output signal Y2 in step ST25, and uses this synthesized signal as the second color signal CY. Output.

<Step ST26>
If it is determined in step ST24 that both the third and fourth output signals Y1 and Y2 are 0, the synthesis processing unit 142 interpolates or extrapolates the signal Y from the surrounding signal Y in step ST26. And is output as the second color signal CY.

<Step ST27>
The interpolation processing unit 141 specifies the target color Z, and when colors other than Z are X and Y, the interpolation processing unit 141 determines whether or not the Z signal is greater than or equal to a preset threshold value VTHZ in step ST27 of FIG.

<Step ST28>
If it is determined in step ST27 that the Z signal is equal to or greater than the preset threshold value VTHZ, the interpolation processing unit 141 uses the Z signal weighted in step ST28 as it is as the fifth output signal Z1 and sets its value to gz * Z.

<Step ST29>
If it is determined in step ST27 that the Z signal is not greater than or equal to the preset threshold value VTHZ, the interpolation processing unit 141 sets the fifth output signal Z1 to 0 in step ST29.

<Step ST30>
In step ST30, the interpolation processing unit 141 determines whether the colors X and Y other than Z are greater than or equal to the threshold values VTHX and VTHY.

<Step ST31>
If it is determined in step ST30 that the colors X and Y other than Z are greater than or equal to the threshold values VTHX and VTHY, the interpolation processing unit 141 calculates the sixth output signal Z2 by [W−gx * X−gy * Y] in step ST31. ].

<Step ST32>
If it is determined in step ST30 that the colors X and Y other than Z are not greater than or equal to the threshold values VTHX and VTHY, the interpolation processing unit 141 sets the sixth output signal Z2 to 0 in step ST32.

  The interpolation processing unit 141 outputs the obtained fifth output signal Z1 and sixth output signal Z2 to the synthesis processing unit 142.

<Step ST33>
In step ST33, the synthesis processing unit 142 receives the fifth output signal Z1 and the sixth output signal Z2 from the interpolation processing unit 141, and both signal levels of the fifth output signal Z1 and the sixth output signal Z2 are set. It is determined whether or not it is greater than zero.

<Step ST34>
If it is determined in step ST33 that the fifth output signal Z1 and the sixth output signal Z2 are greater than 0, the synthesis processing unit 142 performs the following processing.
That is, the composition processing unit 142 obtains a composite signal Z1 + Z2 obtained by adding the fifth output signal Z1 and the sixth output signal Z2 in step ST34, and outputs the composite signal Z1 + Z2 as the third color signal CZ.

<Step ST35>
If it is determined in step ST33 that the fifth output signal Z1 and the sixth output signal Z2 are not greater than 0, the synthesis processing unit 142 determines that the fifth output signal Z1 and the sixth output signal Z2 are 0 in step ST35. It is determined whether or not there is.

<Step ST36>
If it is determined in step ST35 that only one of the fifth and sixth output signals Z1, Z2 is 0, the synthesis processing unit 142 performs the following process.
That is, the synthesis processing unit 142 obtains a synthesized signal [2 * (Z1 + Z2)] obtained by adding the fifth output signal Z1 and the sixth output signal Z2 in step ST36, and uses this synthesized signal as the third color signal CZ. Output.

<Step ST37>
If it is determined in step ST35 that both the fifth and sixth output signals Z1 and Z2 are 0, the synthesis processing unit 142 interpolates or extrapolates the signal from the surrounding signal Z in step ST37. Z is calculated and output as the third color signal CZ.

The first to third signals CX, CY, and CZ of the pixel signals X, Y, and Z obtained in this way can double the signal intensity when there is no defect in the pixels of the pixel unit. it can.
Further, when the number of defective pixels is one, there is no improvement in signal strength, but it is possible to synthesize an alternative signal for a defective pixel.

  In the above description, the case where the threshold values VTHX, VTHY, and VTHZ corresponding to the respective color pixels are used has been described as an example. However, these threshold values can be used as the same value.

  The above description is an example in which the first pixel unit is formed of R, G, B, and W. For example, as shown in FIG. 10, cyan (Cy), magenta (Mg), yellow ( It is also possible to use pixel units formed of Y) and white (W).

[Operation of Imaging Device Having Cy, Mg, Y, W Pixel Units]
Next, although there are overlapping portions, the operation in the case of using the second pixel unit formed of cyan (Cy), magenta (Mg), yellow (Y), and white (W) in FIG. 10 from FIG. The association will be described with reference to FIG.

  FIGS. 11 to 14 are flowcharts for explaining the operation of the imaging apparatus having the second pixel unit of Cy, Mg, Y, and W according to the first embodiment.

<Step ST41>
In step ST41, the weighting unit 130 weights the signal extracted from the Cy, Mg, Y RAW pixel data by the image sensor unit 120 using the weighting coefficient g in consideration of the light attenuation rate by the color filter. .
The weighting unit 130 weights the first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3, which are RAW data from the image sensor unit 120.
The first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3 that have undergone the weighting process are supplied to the interpolation processing unit 141 of the signal interpolation synthesis processing unit 140.
The color interpolation detection signal CDT4 from the image sensor unit 120 is supplied to the interpolation processing unit 141 without passing through the weighting unit 130.

<Step ST42>
In step ST42, the signal interpolation composition processing unit 140 determines whether the magnitude (level) of the color interpolation detection signal CDT4 indicating the white signal W is greater than or equal to the threshold value W1 (W> W1) or less than the threshold value W2 (W <W2). Determine whether or not.

<Step ST43>
If it is determined in step ST42 that the magnitude (level) of the color interpolation detection signal CDT4 is greater than or equal to the threshold value W1 (W> W1) or less than or equal to the threshold value W2 (W <W2), the signal interpolation synthesis processing unit 14 generates in step ST43. The processed color signal is processed as a color signal as it is.

<Step ST44>
If it is determined in step ST42 that the magnitude (level) of the color interpolation detection signal CDT4 is equal to or less than the threshold value W1 and equal to or greater than the threshold value W2 (W2 <W <W1), the interpolation processing unit 141 determines that the target in step ST45. The colors X, Y, and Z are specified.
Then, the interpolation processing unit 141 identifies the target color X and performs the process of FIG.
The interpolation processing unit 141 identifies the target color Y and performs the process of FIG.
The interpolation processing unit 141 identifies the target color Z and performs the process of FIG.
For example, the interpolation processing unit 141 identifies X as Cy, Y as Mg, and Z as Y.

<Step ST45>
When the target color X is specified and the colors other than X are Y and Z, the interpolation processing unit 141 determines whether or not the X signal is greater than or equal to a preset threshold value VTHX in step ST45 of FIG. .

<Step ST46>
If it is determined in step ST45 that the X signal is equal to or greater than the preset threshold value VTHX, the interpolation processing unit 141 uses the weighted X signal in step ST46 as it is as the first output signal X1 and sets its value to gx * X.

<Step ST47>
If it is determined in step ST45 that the X signal is not greater than or equal to the preset threshold value VTHX, the interpolation processing unit 141 sets the first output signal X1 to 0 in step ST47.

<Step ST48>
In step ST48, the interpolation processing unit 141 determines whether the colors Y and Z other than X are greater than or equal to the threshold values VTHY and VTHZ.

<Step ST49>
If it is determined in step ST48 that the colors Y and Z other than X are greater than or equal to the threshold values VTHY and VTHZ, the interpolation processing unit 141 calculates the second output signal X2 by [W−gy * Y−gz * Z] in step ST50. ].

<Step ST50>
If it is determined in step ST48 that the colors Y and Z other than X are not greater than or equal to the threshold values VTHY and VTHZ, the interpolation processing unit 141 sets the second output signal X2 to 0 in step ST50.

  The interpolation processing unit 141 outputs the obtained first output signal X1 and second output signal X2 to the synthesis processing unit 142.

<Step ST51>
In step ST51, the synthesis processing unit 142 receives the first output signal X1 and the second output signal X2 from the interpolation processing unit 141, and both signal levels of the first output signal X1 and the second output signal X2 are set. It is determined whether or not it is greater than zero.

<Step ST52>
If it is determined in step ST52 that the first output signal X1 and the second output signal X2 are greater than 0, the synthesis processing unit 142 performs the following processing.
That is, the synthesis processing unit 142 obtains a synthesized signal X1 + X2 obtained by adding the first output signal X1 and the second output signal X2 in step ST52, and outputs the synthesized signal X1 + X2 as the first color signal CX.

<Step ST53>
If it is determined in step ST51 that the first output signal X1 and the second output signal X2 are not greater than 0, the synthesis processing unit 142 determines that the first output signal X1 and the second output signal X2 are 0 in step ST53. It is determined whether or not there is.

<Step ST54>
If it is determined in step ST53 that only one of the first or second output signals X1 and X2 is 0, the synthesis processing unit 142 performs the following processing.
That is, the synthesis processing unit 142 obtains a synthesized signal [2 * (X1 + X2)] obtained by adding the first output signal X1 and the second output signal X2 in step ST54, and uses this synthesized signal as the first color signal CX. Output.

<Step ST55>
If it is determined in step ST53 that both the first and second output signals X1 and X2 are 0, the synthesis processing unit 142 interpolates or extrapolates the signal from the surrounding signal X in step ST55. X is calculated and output as the first color signal CX.

<Step ST56>
When the target color Y is specified and the colors other than Y are X and Z, the interpolation processing unit 141 determines whether or not the Y signal is equal to or greater than a preset threshold value VTHY in step ST56 of FIG. To do.

<Step ST57>
If it is determined in step ST56 that the Y signal is equal to or greater than the preset threshold value VTHY, the interpolation processing unit 141 uses the weighted Y signal in step ST57 as it is as the third output signal Y1, and sets the value to gy * Y.

<Step ST58>
If it is determined in step ST56 that the Y signal is not equal to or greater than the preset threshold value VTHY, the interpolation processing unit 141 sets the third output signal Y1 to 0 in step ST58.

<Step ST59>
In step 59, the interpolation processing unit 141 determines whether the colors X and Z other than Y are greater than or equal to the threshold values VTHX and VTHZ.

<Step ST60>
If it is determined in step ST59 that the colors X and Z other than Y are greater than or equal to the threshold values VTHX and VTHZ, the interpolation processing unit 141 calculates the fourth output signal Y2 by [W−gx * X−gz * Z] in step ST60. ].

<Step ST61>
If it is determined in step ST59 that the colors X and Z other than Y are not equal to or higher than the threshold values VTHX and VTHZ, the interpolation processing unit 141 sets the fourth output signal Y2 to 0 in step ST61.

  The interpolation processing unit 141 outputs the obtained third output signal Y1 and fourth output signal Y2 to the synthesis processing unit 142.

<Step ST62>
In step ST62, the synthesis processing unit 142 receives the third output signal Y1 and the fourth output signal Y2 from the interpolation processing unit 141, and both signal levels of the third output signal Y1 and the fourth output signal Y2 are detected. It is determined whether or not it is greater than zero.

<Step ST63>
If it is determined in step ST62 that the third output signal Y1 and the fourth output signal Y2 are greater than 0, the synthesis processing unit 142 performs the following processing.
That is, the synthesis processing unit 142 obtains a synthesized signal Y1 + Y2 obtained by adding the third output signal Y1 and the fourth output signal Y2 in step ST63, and outputs the synthesized signal Y1 + Y2 as the second color signal CY.

<Step ST64>
If it is determined in step ST62 that the third output signal Y1 and the fourth output signal Y2 are not greater than 0, the synthesis processing unit 142 determines that the third output signal Y1 and the fourth output signal Y2 are 0 in step ST64. It is determined whether or not there is.

<Step ST65>
If it is determined in step ST64 that only one of the third or fourth output signals Y1 and Y2 is 0, the synthesis processing unit 142 performs the following processing.
That is, the synthesis processing unit 142 obtains a synthesized signal [2 * (Y1 + Y2)] obtained by adding the third output signal Y1 and the fourth output signal Y2 in step ST65, and uses this synthesized signal as the second color signal CY. Output.

<Step ST66>
If it is determined in step ST64 that both the third and fourth output signals Y1 and Y2 are 0, the synthesis processing unit 142 interpolates or extrapolates the signal Y from the surrounding signal Y in step ST66. And is output as the second color signal CY.

<Step ST67>
When the target color Z is specified and the colors other than Z are X and Y, the interpolation processing unit 141 determines whether or not the Z signal is greater than or equal to a preset threshold value VTHZ in step ST67 of FIG.

<Step ST68>
If it is determined in step ST67 that the Z signal is equal to or greater than the preset threshold value VTHZ, the interpolation processing unit 141 uses the Z signal weighted in step ST68 as it is as the fifth output signal Z1 and sets its value to gz * Z.

<Step ST69>
If it is determined in step ST67 that the Z signal is not greater than or equal to the preset threshold value VTHZ, the interpolation processing unit 141 sets the fifth output signal Z1 to 0 in step ST69.

<Step ST70>
In step ST70, the interpolation processing unit 141 determines whether the colors X and Y other than Z are greater than or equal to the threshold values VTHX and VTHY.

<Step ST71>
If it is determined in step ST70 that the colors X and Y other than Z are greater than or equal to the threshold values VTHX and VTHY, the interpolation processing unit 141 calculates the sixth output signal Z2 by [W−gx * X−gy * Y] in step ST71. ].

<Step ST72>
If it is determined in step ST70 that the colors X and Y other than Z are not equal to or higher than the threshold values VTHX and VTHY, the interpolation processing unit 141 sets the sixth output signal Z2 to 0 in step ST72.

  The interpolation processing unit 141 outputs the obtained fifth output signal Z1 and sixth output signal Z2 to the synthesis processing unit 142.

<Step ST73>
In step ST73, the synthesis processing unit 142 receives the fifth output signal Z1 and the sixth output signal Z2 from the interpolation processing unit 141, and both signal levels of the fifth output signal Z1 and the sixth output signal Z2 are set. It is determined whether or not it is greater than zero.

<Step ST74>
If it is determined in step ST73 that the fifth output signal Z1 and the sixth output signal Z2 are greater than 0, the synthesis processing unit 142 performs the following processing.
That is, the composition processing unit 142 obtains a composite signal Z1 + Z2 obtained by adding the fifth output signal Z1 and the sixth output signal Z2 in step ST74, and outputs the composite signal Z1 + Z2 as the third color signal CZ.

<Step ST75>
If it is determined in step ST73 that the fifth output signal Z1 and the sixth output signal Z2 are not greater than 0, the synthesis processing unit 142 determines that the fifth output signal Z1 and the sixth output signal Z2 are 0 in step ST76. It is determined whether or not there is.

<Step ST76>
If it is determined in step ST75 that only one of the fifth or sixth output signals Z1 and Z2 is 0, the synthesis processing unit 142 performs the following processing.
That is, the synthesis processing unit 142 obtains a synthesized signal [2 * (Z1 + Z2)] obtained by adding the fifth output signal Z1 and the sixth output signal Z2 in step ST76, and uses this synthesized signal as the third color signal CZ. Output.

<Step ST77>
If it is determined in step ST75 that both the fifth and sixth output signals Z1 and Z2 are 0, the synthesis processing unit 142 interpolates or extrapolates the signal Z from the surrounding signal Z in step ST77. And is output as the third color signal CZ.

  As described above, in the first embodiment, the first pixel unit formed of red (R), green (G), blue (B), and white (W) or cyan (Cy), When the second pixel unit formed of magenta (Mg), yellow (Y), and white (W) is used, the following becomes possible.

In this case, for example, by obtaining a signal from an X color pixel unit, obtaining a signal obtained by subtracting the Y and Z color signals excluding the X color from the white signal W, and obtaining the pixel signal by two methods. The intensity of the X color signal obtained by one pixel unit is doubled, and the signal intensity is doubled.
When one pixel X in one pixel unit is defective and does not operate due to a manufacturing defect or the like, the pixel signal is not defective and does not operate by subtracting the color pixel signals Y and Z other than X from the white signal. However, it is possible to obtain an image signal and prevent the pixel from becoming a white spot.
However, in the present embodiment, since the signal of white pixels is likely to be saturated, the above processing is not performed when the magnitude of the white signal exceeds a preset threshold value, and therefore the first to third highly accurate. Color signals can be obtained.

In other words, in the first embodiment, half of the unit light receiving elements can be secured as a single color light receiving unit, and about twice the amount of light can be secured per unit area compared to the Bayer array. Sensitivity and dynamic range can be improved.
There are two types of information to obtain signal information: color information that can be obtained directly and color information that is generated from white, so even if elements other than white are used up, color information can be synthesized, thus realizing redundancy. it can.

<2. Second Embodiment>
Next, a second embodiment of the present invention will be described.
In the first embodiment described above, it is possible to achieve an image signal processing apparatus and an imaging apparatus that can improve resolution and have excellent color reproducibility.
In the second embodiment, an image signal that can improve the resolution of the first embodiment and has excellent color reproducibility, has a simple interpolation algorithm, and enables high-accuracy interpolation. A processing device and an imaging device can be realized.
Note that the basic configuration of the image signal processing apparatus and the imaging apparatus is the same as that of the first embodiment shown in FIGS. 1 and 5. Therefore, the description is omitted.
The processing described below is mainly processing of the signal interpolation / synthesis processing unit.

The image signal processing method according to the second embodiment performs calculation at least once, so that a color detection signal is detected at two different pixel positions in a pixel unit composed of four color pixels for each color. Allows extraction.
The signal interpolation synthesis processing unit 140 uses the color interpolation detection signal CDT4 from the image sensor unit 120 to interpolate each of the first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3. Processing is performed to generate a plurality of signals for each color detection signal.

  FIG. 15 is a block diagram illustrating a configuration example of the signal interpolation / synthesis processing unit 140A according to the second embodiment.

  As shown in FIG. 15, the signal interpolation synthesis processing unit 140A includes an interpolation processing unit 141A and a signal space interpolation unit 142A.

The interpolation processing unit 141A uses the color interpolation detection signal CDT4, the first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3 to generate the first color detection signal and the second color detection signal CDT3. Interpolation processing is performed for each of the color detection signal and the third color detection signal.
The interpolation processing unit 141A, for each color detection signal, X1 at a plurality of X color pixel positions, X2 at the color interpolation pixel position, Y1 at the Y color pixel position, Y2, Z color at the color interpolation pixel position Z1 at the pixel position and Z2 at the position of the color interpolation pixel are generated.

The signal space interpolation unit 142A is a simple calculation from each color detection signal output from the interpolation processing unit 141A, and at the X color signal X3 and Z color pixel positions at the first color signal X1, X2, and Y color pixel positions. An X color signal X4 is generated.
The signal space interpolation unit 142A includes second color signals Y1, Y2, Y color signal Y3 at the X color pixel position, Y color signal Y4 at the Z color pixel position, and third color signals Z1, Z2, X color. A ZY color signal Z3 at the pixel position and a Z color signal Z4 at the Y color pixel position are generated.

  As described above, in the signal interpolation / synthesis processing unit 140A according to the second embodiment, color reproducibility is achieved by performing spatial interpolation of colors using more pixel information than in the first embodiment. Excellent color spatial interpolation can be realized.

  The image signal processing method of the second embodiment will be specifically described below, although there are portions that overlap with the first embodiment.

When a pixel unit composed of R, G, B, and W or a pixel unit composed of Cy (cyan), Mg (magenta), Y (yellow), and W is used, the following becomes possible.
The basic algorithm is shown in FIGS.
In the following description, as an example, X1 (X) can be specified as R or Cy, X2 (Y) as G or Mg, and X3 as B or Y.

In this case, as means for obtaining the X1 color pixel signal, the X1 color signal at the coordinates of the X1 color pixel is obtained, and the X2 color signal excluding the X1 color is subtracted from the W (white) signal. To do.
Similarly, by repeatedly calculating the X1 color from the generated X2, X3 and W (white) signals, the value of the X1 color signal can be made closer to the original color signal of the Z1 color in the white pixel W. it can.
As a result, the X1 signal at the coordinates of the X1 color pixel, the X2 signal at the coordinates of the X2 color pixel, the X3 signal at the coordinates of the X3 color pixel, and the X1, X2, and X3 signals at the white pixel coordinates are obtained. .

  As a result, the X1 signal at the coordinates of the X1 color pixel, the X2 signal at the coordinates of the X2 color pixel, the X3 signal at the coordinates of the X3 color pixel, and the X1, X2, and X3 signals at the white pixel coordinates are obtained. .

On the other hand, when one color pixel X1 in one pixel unit is defective and does not operate due to a manufacturing defect or the like, by subtracting the pixel signals X2 and X3 of the color pixels (other than X1) from the white signal, An image signal can be obtained even if the pixel is defective and does not operate. In addition, the pixel can be prevented from becoming a white spot.
However, since the signal for white pixels is likely to be saturated, it is desirable not to perform the above processing when the magnitude of the white signal exceeds a preset threshold value.

In this way, the X1 signal at the X1 pixel coordinates, the X2 signal at the X2 pixel coordinates, the X3 signal at the X3 pixel coordinates, and the X1, X2, and X3 signals at the white pixel coordinates can get.
Among the four pixels constituting the pixel unit, signals of X1, X2, and X3 are obtained at two positions of each color pixel and W pixel.
For this reason, in the case of the X1 signal, the X1 signal at the coordinates of the X2 pixel and the X1 signal of the X3 pixel are either the average of the upper and lower X1 signals, or the average of the left and right X1 signals, Can be calculated.
As a result, color signals of each color at two pixel coordinates among the four pixels of the pixel unit can be obtained, and the color signals at the remaining two pixel coordinates can be interpolated, and the algorithm becomes simple. Since the number of pixels used in the process is reduced, the number of pixels in which false colors are generated is also reduced.

FIG. 20 is a diagram for explaining a pixel data coordinate interpolation method.
FIG. 21 is a diagram showing a coordinate interpolation method shown as a comparative example.

In FIG. 20, the signal obtained substantially from the fact that W can predict R, G, and B by simulation is as shown in the figure.
For the A (X1, B) color, the A signal at the positions B (X2, G) and C (X3, R) is obtained from the average of the A values above and below B and C.
For the B (X2, G) color, the B signal at the positions A and C is obtained from the average of the B values at the top, bottom, left and right of A and C.
For the C (X3, R) color, the C signal at positions A and B is obtained from the average of the C values on the left and right sides of A and B.

In the Bayer array of the comparative example of FIG. 21, for example, when the number of pixels is increased by interpolation, it is necessary to predict the interpolation of X and Y pixel data from R, G, and B data. It is necessary to adopt different Y pixels.
As for the signal of the pixel at the position X, the left and right color signals are predicted on average.
The signal of the pixel at the Y position is predicted by the average of the upper and lower color signals.
The signal of the pixel at the position Z is predicted by a color signal located obliquely.
The signal of the pixel at the position V is predicted from the upper, lower, left and right color signals.

Here, the basic algorithm shown in FIGS. 16 to 19 will be described.
FIG. 16 is a diagram illustrating a first example of a basic algorithm according to the second embodiment.

<Step ST101>
In step ST101, it is determined whether or not two or more pixels per pixel unit are defective.

<Step ST102>
If there is no defect of two or more pixels, it is determined in step ST102 whether the W pixel is defective or the signal is saturated.

<Step ST103>
If the W pixel is not defective or the signal is not saturated, it is determined in step ST103 whether there is one pixel defect per pixel unit.

<Step ST104>
If there is one pixel defect per pixel unit, the signal of the defective pixel A is synthesized by {A = WBC} in step ST104.

<Step ST105>
If there is no defective pixel per pixel unit, or if signals of defective pixels are synthesized, R1, G1, B1, and R2, G2, B3 are set in step ST105.
Here, R2, G2, and B2 indicate R, G, and B signals in the W pixel.

<Steps ST106 and ST107>
In steps ST106 and ST107, {R2 ′ = W−G2−B2}, {G2 ′ = W−R2−B2}, and {B2 ′ = W−R2−G2} reach the upper limit set in advance, Repeat until the variable is below the preset criteria.

<Step ST108>
When the number of iterations reaches the preset upper limit and the variable amount falls below the preset judgment condition, the X signal is synthesized from the signals X1 and X2 (X = R, G, B) and supplied to the image processing circuit. To do.

<Step ST109>
In step ST101, when two or more pixels per pixel unit are defective, pixel data is interpolated from surrounding pixels and supplied to the image processing circuit.

<Step ST110>
In step ST102, when the W pixel is defective or the signal is saturated, pixel data is interpolated from the surrounding pixels and supplied to the image processing circuit.

Next, a second example will be described.
FIG. 17 is a diagram illustrating a second example of the basic algorithm according to the second embodiment.

The second example is different from the first example of FIG. 16 as follows.
In step ST107A, the number of iterations reaches a preset upper limit, and the next determination is performed instead of determining a section whose variable is less than or equal to a preset determination condition.
That is, in step ST107A, it is determined whether or not the absolute values of (R2′−R2), (G2′−G2), and (B2′−B2) have reached a predetermined threshold value or less.

Next, a third example will be described.
FIG. 18 is a diagram illustrating a third example of the basic algorithm according to the second embodiment.

The third example is different from the first example of FIG. 16 as follows.
In step ST108A, instead of synthesizing the X signal from the signals X1 and X2 (X = R, G, B), the following processing is performed.
That is, in step ST108A, pixel data is interpolated from surrounding pixels and supplied to the image processing circuit.

Next, a fourth example will be described.
FIG. 19 is a diagram illustrating a fourth example of the basic algorithm according to the second embodiment.

The fourth example is different from the second example of FIG. 17 as follows.
In step ST108A, instead of synthesizing the X signal from the signals X1 and X2 (X = R, G, B), the following processing is performed.
That is, in step ST108A, pixel data is interpolated from surrounding pixels and supplied to the image processing circuit.

  Next, a specific processing example according to the second embodiment will be specifically described for a case corresponding to the processing described in association with FIGS. 6 to 14 of the first embodiment.

[First processing example]
[Operation of Imaging Device Having R, G, B, W Pixel Units]
FIGS. 22 to 25 are flowcharts for explaining a first processing example of the imaging apparatus having the R, G, B, and W pixel units according to the second embodiment.

<Step ST201>
In step ST201, the weighting unit 130 weights the signal extracted from the R, G, and B RAW pixel data by the image sensor unit 120 using the weighting coefficient g in consideration of the light attenuation rate by the color filter. .
The weighting unit 130 weights the first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3, which are RAW data from the image sensor unit 120.
The first color detection signal CDT1, the second color detection signal CDT2, and the third color detection signal CDT3 that have undergone the weighting process are supplied to the interpolation processing unit 141 of the signal interpolation synthesis processing unit 140.
The color interpolation detection signal CDT4 from the image sensor unit 120 is supplied to the interpolation processing unit 141 without passing through the weighting unit 130.

<Step ST202>
In step ST202, the signal interpolation composition processing unit 140 determines whether the magnitude (level) of the color interpolation detection signal CDT4 indicating the white signal W is greater than or equal to the threshold value W1 (W> W1) or less than the threshold value W2 (W <W2). Determine whether or not.

<Step ST203>
If it is determined in step ST202 that the magnitude (level) of the color interpolation detection signal CDT4 is greater than or equal to the threshold W1 (W> W1) or less than or equal to the threshold W2 (W <W2), the signal interpolation synthesis processing unit 140 determines in step ST3. The generated color signal is processed as a color signal as it is.

<Step ST204>
If it is determined in step ST202 that the magnitude (level) of the color interpolation detection signal CDT4 is equal to or less than the threshold value W1 and equal to or greater than the threshold value W2 (W2 <W <W1), the interpolation processing unit 141 is targeted in step ST4. Specify colors X, Y, and Z.
Then, the interpolation processing unit 141 identifies the target color X and performs the process of FIG.
The interpolation processing unit 141 identifies the target color Y and performs the process of FIG.
The interpolation processing unit 141 identifies the target color Z and performs the process of FIG.
The interpolation processing unit 141 specifies, for example, X as R, Y as G, and Z as B.

<Step ST205>
When the target color X is specified and the colors other than X are Y and Z, the interpolation processing unit 141 determines whether or not the X signal is greater than or equal to a preset threshold value VTHX in step ST205 of FIG. .

<Step ST206>
If it is determined in step ST205 that the X signal is equal to or higher than the preset threshold value VTHX, the interpolation processing unit 141 uses the weighted X signal in step ST6 as it is as the first output signal X1, and sets the value to gx * X.

<Step ST207>
If it is determined in step ST205 that the X signal is not greater than or equal to the preset threshold value VTHX, the interpolation processing unit 141 sets the first output signal X1 to 0 in step ST207.

<Step ST208>
In step ST208, the interpolation processing unit 141 determines whether the colors Y and Z other than X are greater than or equal to the threshold values VTHY and VTHZ.

<Step ST209>
If it is determined in step ST208 that the colors Y and Z other than X are greater than or equal to the threshold values VTHY and VTHZ, the interpolation processing unit 141 calculates the second output signal X2 = gx * X, Y = gy * Y, by calculation in step ST209. Set Z2 = gz * Z.

<Step ST210>
If it is determined in step ST2088 that the colors Y and Z other than X are not equal to or higher than the threshold values VTHY and VTHZ, the interpolation processing unit 141 sets the second output signal X2 to 0 in step ST10.

  The interpolation processing unit 141 outputs the obtained first output signal X1 and second output signal X2 to the synthesis processing unit 142.

<Steps ST211, ST212>
When colors Y and Z other than X are greater than or equal to threshold values VTHY and VTHZ, {X2 ′ = W−Y2−Z2}, {Y2 ′ = W−X2−Z2}, {Z2 ′ = W after the processing of step ST209. -X2-Y2}, X2 = X2 ′, Y2 = Y2 ′, Z2 = Z2 ′ are repeated a plurality of times (N times, N = 1, 2, 3,...) Or more.
Since there is no big difference between X2 ′ and X2, it is considered that it converges to a steady state by repeating several times.
For this reason, as described above, if the iteration is repeated N times in advance, the iteration is stopped, and if the absolute value of X2′−X2 falls below a threshold value ε (for example, ε = 1), it is assumed that the iteration has converged, and the iteration is stopped. This method can be adopted.

<Step ST213>
In step ST213, the synthesis processing unit 142 receives the first output signal X1 and the second output signal X2 from the interpolation processing unit 141, and both signal levels of the first output signal X1 and the second output signal X2 are detected. It is determined whether or not it is greater than zero.

<Step ST214>
If it is determined in step ST213 that the first output signal X1 and the second output signal X2 are greater than 0, the synthesis processing unit 142 performs the following processing.
That is, the synthesis processing unit 142 obtains a synthesized signal X1 + X2 obtained by adding the first output signal X1 and the second output signal X2 in step ST214, and outputs the synthesized signal X1 + X2 as the first color signal CX.

<Step ST215>
If it is determined in step ST213 that the first output signal X1 and the second output signal X2 are not greater than 0, the synthesis processing unit 142 determines that the first output signal X1 and the second output signal X2 are 0 in step ST215. It is determined whether or not there is.

<Step ST216>
If it is determined in step ST215 that only one of the first or second output signals X1 and X2 is 0, the synthesis processing unit 142 performs the following process.
That is, the synthesis processing unit 142 obtains a synthesized signal [2 * (X1 + X2)] obtained by adding the first output signal X1 and the second output signal X2 in step ST216, and uses this synthesized signal as the first color signal CX. Output.

<Step ST217>
If it is determined in step ST215 that both the first and second output signals X1 and X2 are 0, the synthesis processing unit 142 performs interpolation or extrapolation from the peripheral signal X in step ST217 to generate a signal. X is calculated and output as the first color signal CX.

<Step ST218>
When the target color Y is specified and the colors other than Y are X and Z, the interpolation processing unit 141 determines whether or not the Y signal is greater than or equal to a preset threshold value VTHY in step ST218 of FIG. .

<Step ST219>
If it is determined in step ST218 that the Y signal is greater than or equal to the preset threshold value VTHY, the interpolation processing unit 141 uses the weighted Y signal in step ST219 as it is as the third output signal Y1, and sets its value to gy * Y.

<Step ST220>
If it is determined in step ST218 that the Y signal is not equal to or greater than the preset threshold value VTHY, the interpolation processing unit 141 sets the third output signal Y1 to 0 in step ST220.

<Step ST221>
In step ST221, the interpolation processing unit 141 determines whether the colors X and Z other than Y are greater than or equal to the threshold values VTHX and VTHZ.

<Step ST222>
If it is determined in step ST221 that the colors X and Z other than Y are greater than or equal to the threshold values VTHX and VTHZ, the interpolation processing unit 141 calculates the second output signal X2 = gx * X, Y = gy * Y, Set Z2 = gz * Z.

<Step ST223>
If it is determined in step ST222 that the colors X and Z other than Y are not greater than or equal to the threshold values VTHX and VTHZ, the interpolation processing unit 141 sets the second output signal X2 to 0 in step ST223.

  The interpolation processing unit 141 outputs the obtained first output signal X1 and second output signal X2 to the synthesis processing unit 142.

<Steps ST224 and ST225>
When the colors X and Z other than Y are greater than or equal to the threshold values VTHX and VTHZ, {Y2 ′ = W−X2−Z2}, {Z2 ′ = W−X2−Y2}, {X2 ′ = W after the processing of step ST222. -Y2-Z2}, Y2 = Y2 ′, Z2 = Z2 ′, X2 = X2 ′ are repeated a plurality of times (N times, N = 1, 2, 3,...) Or more.
Since Y2 ′ and Y2 are not significantly different, it is considered that they converge to a steady state by repeating several times.
For this reason, as described above, when the iteration is repeated N times in advance, the iteration is stopped, and when the absolute value of Y2′−Y2 becomes equal to or less than the threshold value ε (for example, ε = 1), it is regarded that it has converged, and the iteration is stopped. This method can be adopted.

<Step ST226>
In step ST226, the synthesis processing unit 142 receives the third output signal Y1 and the fourth output signal Y2 from the interpolation processing unit 141, and both signal levels of the third output signal Y1 and the fourth output signal Y2 are set. It is determined whether or not it is greater than zero.

<Step ST227>
If it is determined in step ST226 that the third output signal Y1 and the fourth output signal Y2 are greater than 0, the synthesis processing unit 142 performs the following processing.
That is, the composition processing unit 142 obtains a composite signal Y1 + Y2 obtained by adding the third output signal Y1 and the fourth output signal Y2 in step ST227, and outputs the composite signal Y1 + Y2 as the second color signal CY.

<Step ST228>
If it is determined in step ST226 that the third output signal Y1 and the fourth output signal Y2 are not greater than 0, the synthesis processing unit 142 determines that the third output signal Y1 and the fourth output signal Y2 are 0 in step ST228. It is determined whether or not there is.

<Step ST229>
If it is determined in step ST228 that only one of the third or fourth output signals Y1 and Y2 is 0, the synthesis processing unit 142 performs the following process.
That is, the synthesis processing unit 142 obtains a synthesized signal [2 * (Y1 + Y2)] obtained by adding the third output signal Y1 and the fourth output signal Y2 in step ST229, and uses this synthesized signal as the second color signal CY. Output.

<Step ST230>
If it is determined in step ST228 that both the third and fourth output signals Y1 and Y2 are 0, the synthesis processing unit 142 interpolates or extrapolates the signal Y from the surrounding signal Y in step ST230. And is output as the second color signal CY.

<Step ST231>
The interpolation processing unit 141 specifies the target color Z and determines whether the Z signal is equal to or higher than the preset threshold value VTHZ in step ST231 in FIG. 25 when the colors other than Z are X and Y. .

<Step ST232>
If it is determined in step ST231 that the Z signal is greater than or equal to the preset threshold value VTHZ, the interpolation processing unit 141 uses the Z signal weighted in step ST232 as it is as the fifth output signal Z1 and sets its value to gz * Z.

<Step ST233>
If it is determined in step ST231 that the Z signal is not greater than or equal to the preset threshold value VTHZ, the interpolation processing unit 141 sets the fifth output signal Z1 to 0 in step ST234.

<Step ST234>
In step ST234, the interpolation processing unit 141 determines whether the colors X and Y other than Z are greater than or equal to the threshold values VTHX and VTHY.

<Step ST235>
If it is determined in step ST234 that the colors X and Y other than Z are greater than or equal to the threshold values VTHX and VTHY, the interpolation processing unit 141 calculates the second output signals X2 = gx * X, Y = gy * Y, by calculation in step ST235. Set Z2 = gz * Z.

<Step ST236>
If it is determined in step ST234 that the colors X and Y other than Z are not greater than or equal to the threshold values VTHX and VTHY, the interpolation processing unit 141 sets the second output signal X2 to 0 in step ST236.

  The interpolation processing unit 141 outputs the obtained first output signal X1 and second output signal X2 to the synthesis processing unit 142.

<Steps ST237 and ST238>
When the colors X and Y other than Z are equal to or greater than the threshold values VTHX and VTHY, {Z2 ′ = W−X2−Y2}, {X2 ′ = W−Y2−Z2}, {Y2 ′ = W after the processing of step ST222. -X2-Z2}, Z2 = Z2 ′, X2 = X2 ′, Y2 = Y2 ′ are repeated a plurality of times (N times, N = 1, 2, 3,...) Or more.
Since there is no big difference between Z2 ′ and Z2, it is considered that it converges to a steady state by repeating several times.
For this reason, as described above, if the iteration is repeated N times in advance, the iteration is stopped, and if the absolute value of Z2′−Z2 falls below a threshold value ε (for example, ε = 1), it is assumed that the iteration has converged, and the iteration is stopped. This method can be adopted.

<Step ST239>
In step ST239, the synthesis processing unit 142 receives the fifth output signal Z1 and the sixth output signal Z2 from the interpolation processing unit 141, and both signal levels of the fifth output signal Z1 and the sixth output signal Z2 are set. It is determined whether or not it is greater than zero.

<Step ST240>
If it is determined in step ST239 that the fifth output signal Z1 and the sixth output signal Z2 are greater than 0, the synthesis processing unit 142 performs the following processing.
That is, the composition processing unit 142 obtains a composite signal Z1 + Z2 obtained by adding the fifth output signal Z1 and the sixth output signal Z2 in step ST240, and outputs the composite signal Z1 + Z2 as the third color signal CZ.

<Step ST241>
If it is determined in step ST239 that the fifth output signal Z1 and the sixth output signal Z2 are not greater than 0, the synthesis processing unit 142 determines that the fifth output signal Z1 and the sixth output signal Z2 are 0 in step ST241. It is determined whether or not there is.

<Step ST242>
If it is determined in step ST241 that only one of the fifth and sixth output signals Z1 and Z2 is 0, the synthesis processing unit 142 performs the following process.
That is, the synthesis processing unit 142 obtains a synthesized signal [2 * (Z1 + Z2)] obtained by adding the fifth output signal Z1 and the sixth output signal Z2 in step ST242, and uses this synthesized signal as the third color signal CZ. Output.

<Step ST243>
If it is determined in step ST241 that both the fifth and sixth output signals Z1 and Z2 are 0, the synthesis processing unit 142 interpolates or extrapolates the signal from the surrounding signal Z in step ST243. Z is calculated and output as the third color signal CZ.

  The signals of the pixels X, Y, and Z obtained in this way can be obtained by falsely predicting the R, G, and B signals in the white pixel coordinates more accurately when there is no defect in the pixels of the pixel unit. The signal intensity can be approximately doubled in a state where there is little. In the case of up to one pixel defect, it is possible to synthesize an alternative signal for the defective pixel.

  In the above description, the case where the threshold values VTHX, VTHY, and VTHZ corresponding to the respective color pixels are used has been described as an example. However, these threshold values can be used as the same value.

The above description is an example in which the first pixel unit is formed of R, G, B, and W. For example, as shown in FIG. 10, cyan (Cy), magenta (Mg), yellow ( It is also possible to use pixel units formed of Y) and white (W).
Since the processing is performed in the same manner, the description thereof is omitted here.
This first processing example is applicable to the following cases regardless of whether the pixel unit is R, G, B, W, and cyan (Cy), magenta (Mg), yellow (Y), or white (W). .
The present invention can be applied to a signal extracted from a unit pixel of an image sensor composed of four pixels, or a signal extracted from a unit pixel in an image pickup apparatus provided with four single-size image sensors to obtain the signal.

[Second processing example]
[Operation of Imaging Device Having R, G, B, W Pixel Units]
FIGS. 26 to 29 are flowcharts for explaining a second processing example of the imaging apparatus having the R, G, B, and W pixel units according to the second embodiment.

In FIG. 26 to FIG. 29, the same processing as the first processing in FIG. 22 to FIG. 25 is shown with the same steps in order to avoid overlapping explanation and facilitate understanding.
The process of FIG. 26 is the same as the process of FIG.
The second process is different from the first process as follows.

  Instead of the processing in steps ST213 to ST217 in FIG. 23, the processing in the next steps ST251 and ST252 is performed in FIG.

<Step ST251>
The X color signal in the X pixel is given by X1, and the X color signal in the white pixel W is given by Z2.

<Step ST252>
When the X pixel signal is interpolated with respect to the coordinates, among the eight pixels surrounding the pixel to which the X color signal is to be synthesized, the X signal is the average of the X signals that have a non-zero X signal. Synthesize.

  Instead of the processing in steps ST226 to ST230 in FIG. 24, the processing in the next steps ST253 and ST254 is performed in FIG.

<Step ST253>
The Y color signal in the Y pixel is given by Y1, and the Y color signal in the white pixel W is given by Y2.

<Step ST254>
When the Y pixel signal is interpolated with respect to the coordinates, the Y signal is the average of the Y signals of the eight pixels surrounding the pixel to which the Y color signal is to be synthesized but has a non-zero Y signal. Synthesize.

  Instead of the processing in steps ST239 to ST243 in FIG. 25, the processing in the next steps ST255 and ST256 is performed in FIG.

<Step ST255>
The Z color signal in the Z pixel is given by Z1, and the Z color signal in the white pixel W is given by Z2.

<Step ST256>
When interpolation processing is performed on the Z-color pixel signal with respect to the coordinates, among the eight pixels surrounding the pixel for which the Z-color signal is to be synthesized, the Z signal is the average of the Z signals that have a non-zero Z signal. Synthesize.

In the case of the second processing, the X color, the Y color, and the Z color are different algorithms, but interpolation can be performed with a single algorithm within the Z color, the Y color, and the Z color.
The signals of the pixels X, Y, and Z obtained in this way can interpolate the color signals of the remaining two pixels based on the information of the two pixels if the pixels of the pixel unit are not defective. .

This second processing example is also applicable to the following cases regardless of whether the pixel unit is R, G, B, W, cyan (Cy), magenta (Mg), yellow (Y), or white (W). .
The present invention can be applied to a signal extracted from a unit pixel of an image sensor composed of four pixels, or a signal extracted from a unit pixel in an image pickup apparatus provided with four single-size image sensors to obtain the signal.

The imaging device of the present invention may be an imaging device formed as a single chip or a module type imaging device formed as an aggregate of a plurality of chips.
In the case of an imaging device formed as an assembly of a plurality of chips, the imaging device is divided into a sensor chip that performs imaging, a signal processing chip that performs digital signal processing, and the like, and may further include an optical system.

The imaging device can function as a digital camera, a video camera, or the like.
If necessary, the imaging device 300 can be added with the following functions and devices as shown in FIG. 30 in addition to the optical lens of FIG.
One of the recording device 301, the signal reception device 302, the signal transmission device 303, the focus adjustment device 304, the image recognition device 305, the wireless reception, the transmitters 306 and 307, the position control device 308, and the signal control device 309 is included in the imaging device 300. Alternatively, all functions can be added.

Note that the method described above in detail can be formed as a program according to the above-described procedure and executed by a computer such as a CPU.
Further, such a program can be configured to be accessed by a recording medium such as a semiconductor memory, a magnetic disk, an optical disk, a floppy (registered trademark) disk, or the like, and to execute the program by a computer in which the recording medium is set.

  DESCRIPTION OF SYMBOLS 100 ... Imaging device, 110 ... Optical lens system, 120 ... Image sensor part, 130 ... Weighting part, 140,140A ... Signal interpolation synthetic | combination processing part, 141, 141A ... Interpolation process , 142... Synthesis processing unit, 142 A... Signal space interpolation unit, 150.

Claims (19)

An interpolation processing unit including a function of performing interpolation processing of a plurality of input color detection signals and generating a plurality of signals for each color detection signal;
Combining each output signal of the interpolation processing unit to generate each color signal, and
The interpolation processing unit
A first color detection signal from a first color detection element having transparency in the wavelength range of the first electromagnetic wave;
A second color detection signal from a second color detection element having transparency in the wavelength range of the second electromagnetic wave,
A third color detection signal from a third color detection element having transparency in the wavelength range of the third electromagnetic wave, and a wavelength range including at least a part of the first, second, and third wavelength ranges; The color interpolation detection signal from the transparent color interpolation element is input,
Using the color interpolation detection signal, the first color detection signal, the second color detection signal, and the third color detection signal, the first color detection signal, the second color detection signal, And interpolating each of the third color detection signals to generate a plurality of signals for each color detection signal,
The synthesis processing unit
An image signal processing apparatus that generates a first color signal, a second color signal, and a third color signal by combining a plurality of output signals of each color detection signal output from the interpolation processing unit. The interpolation processing unit
For one interpolation target color detection signal, the interpolation target color detection signal is used as a first output signal, and two color detection signals excluding the interpolation target color detection signal are subtracted from the color interpolation detection signal. Generating a second output signal;
The synthesis processing unit
The image signal processing apparatus according to claim 1, wherein the first output signal output from the interpolation processing unit and the second output signal are combined to generate a color signal. The interpolation processing unit
For the first color detection signal, two second and third color detections that exclude the first color detection signal from the color interpolation detection signal by using the first color detection signal as a first output signal. If both of the signals are greater than or equal to a predetermined value, they are subtracted to generate a second output signal, and if either of the two color detection signals is less than or equal to the predetermined value, a second output signal of value 0 is generated,
For the second color detection signal, two first and third color detections that exclude the second color detection signal from the color interpolation detection signal by using the second color detection signal as a third output signal. If both of the signals are greater than or equal to a predetermined value, they are subtracted to generate a fourth output signal, and if either of the two color detection signals is less than or equal to the predetermined value, a fourth output signal of value 0 is generated,
For the third color detection signal, two first and second color detections that exclude the third color detection signal from the color interpolation detection signal by using the third color detection signal as a fifth output signal. If both of the signals are greater than or equal to a predetermined value, these are subtracted to generate a sixth output signal, and if either of the two color detection signals is less than or equal to the predetermined value, a sixth output signal of value 0 is generated,
The synthesis processing unit
Combining the first output signal and the second output signal output from the interpolation processing unit to generate a first color signal;
Combining the third output signal output from the interpolation processor and the fourth output signal to generate a second color signal;
The image signal processing apparatus according to claim 1, wherein the fifth output signal output from the interpolation processing unit and the sixth output signal are combined to generate a third color signal. The interpolation processing unit
Calculates three color detection signals for one interpolation target color detection signal, using the interpolation target color detection signal as the first output signal, including two color detection signals excluding the interpolation target color detection signal. For the three color detection signals, a process of subtracting two color detection signals excluding the color detection signal to be calculated from the color interpolation detection signal is repeated a plurality of times to generate a second output signal,
The synthesis processing unit
The image signal processing apparatus according to claim 1, wherein the first output signal output from the interpolation processing unit and the second output signal are combined to generate a color signal. The interpolation processing unit
With respect to the first color detection signal, two second and third color detection signals excluding the first color detection signal from the color interpolation detection signal using the first color detection signal as a first output signal. Are both equal to or greater than a predetermined value, the first color detection signal, the second color detection signal, and the third color detection signal are subject to computation, the three color detection signals are subject to computation, and the three For the color detection signal, a process for subtracting two color detection signals excluding the color detection signal to be calculated from the color interpolation detection signal is repeated a plurality of times to generate a second output signal, and any of the two color detection signals If is less than or equal to a predetermined value, a second output signal of value 0 is generated,
For the second color detection signal, two first and third color detections that exclude the second color detection signal from the color interpolation detection signal by using the second color detection signal as a third output signal. When both of the signals are equal to or greater than a predetermined value, the first color detection signal, the second color detection signal, and the third color detection signal are set as calculation targets, the three color detection signals are set as calculation targets, and the 3 For one color detection signal, a process of subtracting two color detection signals excluding the color detection signal to be calculated from the color interpolation detection signal is repeated a plurality of times to generate a fourth output signal, and the two color detection signals If either is below a predetermined value, a fourth output signal with a value of 0 is generated,
For the third color detection signal, two first and second color detections that exclude the third color detection signal from the color interpolation detection signal by using the third color detection signal as a fifth output signal. When both of the signals are equal to or greater than a predetermined value, the first color detection signal, the second color detection signal, and the third color detection signal are set as calculation targets, the three color detection signals are set as calculation targets, and the 3 For one color detection signal, a process of subtracting two color detection signals excluding the color detection signal to be calculated from the color interpolation detection signal is repeated a plurality of times to generate a sixth output signal, and the two color detection signals If either is below a predetermined value, a sixth output signal with a value of 0 is generated,
The synthesis processing unit
Combining the first output signal and the second output signal output from the interpolation processing unit to generate a first color signal;
Combining the third output signal output from the interpolation processor and the fourth output signal to generate a second color signal;
The image signal processing apparatus according to claim 1, wherein the fifth output signal output from the interpolation processing unit and the sixth output signal are combined to generate a third color signal. The interpolation processing unit
The image signal processing apparatus according to claim 4 or 5, wherein when the number of times set in advance is reached, the above-described repetitive calculation processing is terminated. The interpolation processing unit
The image signal processing apparatus according to claim 4 or 5, wherein when the absolute value of the difference obtained by the repeated calculation process is equal to or less than a preset threshold value, the repeated calculation process is terminated. The synthesis processing unit
When the first output signal and the second output signal have a predetermined level, a color signal is generated by adding the first output signal and the second output signal,
When only one of the first output signal and the second output signal has a predetermined level, a color obtained by multiplying a signal obtained by adding the first output signal and the second output signal by a value greater than 1 Generate a signal,
The image according to claim 2, wherein when both of the first output signal and the second output signal have not reached a predetermined level, a color signal is generated using a color detection signal around the corresponding color detection signal. Signal processing device. The synthesis processing unit
In generating the first color signal,
When the first output signal and the second output signal have a predetermined level, the first output signal and the second output signal are added to generate a first color signal,
When only one of the first output signal and the second output signal has a predetermined level, the signal obtained by adding the first output signal and the second output signal is multiplied by a value greater than 1 to 1 color signal is generated,
If both of the first output signal and the second output signal have not reached a predetermined level, a first color signal is generated using a color detection signal around the corresponding color detection signal,
In generating the second color signal,
When the third output signal and the fourth output signal have predetermined levels, the third output signal and the fourth output signal are added to generate a second color signal,
When only one of the third output signal and the fourth output signal has a predetermined level, the signal obtained by adding the third output signal and the fourth output signal is multiplied by a value greater than 1, and 2 color signals are generated,
When both the third output signal and the fourth output signal have not reached a predetermined level, a third color signal is generated using a color detection signal around the corresponding color detection signal,
In generating the third color signal,
When the fifth output signal and the sixth output signal have predetermined levels, the fifth output signal and the sixth output signal are added to generate a third color signal,
When only one of the fifth output signal and the sixth output signal has a predetermined level, the signal obtained by adding the fifth output signal and the sixth output signal is multiplied by a value greater than 1 to 3 color signals are generated,
The third color signal is generated using a color detection signal around the corresponding color detection signal when both the fifth output signal and the sixth output signal have not reached a predetermined level. The image signal processing apparatus described. A weighting unit that weights the first color detection signal, the second color detection signal, and the third color detection signal according to a light attenuation rate and supplies the weighted signal to the interpolation processing unit; The image signal processing apparatus according to claim 1. The interpolation processing unit and the synthesis processing unit are
11. The first color detection signal, the second color detection signal, and the third color detection signal are directly processed as color signals when the color interpolation detection signal is outside an allowable range. Image signal processing apparatus. A sensor comprising a plurality of pixel units in which a plurality of pixels having different wavelength ranges to be transmitted are arranged in an array, the light transmitted through the pixels is converted into an electrical signal, and a plurality of color detection signals and a color interpolation detection signal are output. And
An image signal processing device having a function of performing interpolation and synthesis processing on a plurality of color detection signals and color interpolation detection signals by the sensor unit;
The image signal processing apparatus is
An interpolation processing unit including a function of performing interpolation processing of a plurality of color detection signals output from the sensor unit and generating a plurality of signals for each color detection signal;
Combining each output signal of the interpolation processing unit to generate each color signal, and
The pixel unit of the sensor unit is
A first color detection element having transparency in the wavelength range of the first electromagnetic wave, and detecting the first electromagnetic wave to generate a first color detection signal;
A second color detection element that is transparent in the wavelength range of the second electromagnetic wave, detects the second electromagnetic wave, and generates a second color detection signal;
A third color detection element that is transparent in the wavelength range of the third electromagnetic wave, detects the third electromagnetic wave, and generates a third color detection signal;
It has transparency in a fourth wavelength range including at least a part of the first, second, and third wavelength ranges, and detects a color interpolation detection signal by detecting electromagnetic waves in the fourth wavelength range. A color interpolation element to be generated,
The interpolation processing unit
Using the color interpolation detection signal, the first color detection signal, the second color detection signal, and the third color detection signal by the sensor unit, the first color detection signal and the second color detection signal are used. Interpolating each of the color detection signal and the third color detection signal to generate a plurality of signals for each color detection signal,
The synthesis processing unit
An imaging apparatus that generates a first color signal, a second color signal, and a third color signal by combining a plurality of output signals of each color detection signal output from the interpolation processing unit. The interpolation processing unit
For one interpolation target color detection signal, the interpolation target color detection signal is used as a first output signal, and two color detection signals excluding the interpolation target color detection signal are subtracted from the color interpolation detection signal. Generating a second output signal;
The synthesis processing unit
The imaging apparatus according to claim 12, wherein a color signal is generated by combining the first output signal and the second output signal output from the interpolation processing unit. The interpolation processing unit
Calculates three color detection signals for one interpolation target color detection signal, using the interpolation target color detection signal as the first output signal, including two color detection signals excluding the interpolation target color detection signal. For the three color detection signals, a process of subtracting two color detection signals excluding the color detection signal to be calculated from the color interpolation detection signal is repeated a plurality of times to generate a second output signal,
The synthesis processing unit
The imaging apparatus according to claim 12, wherein a color signal is generated by combining the first output signal and the second output signal output from the interpolation processing unit. The synthesis processing unit
When the first output signal and the second output signal have a predetermined level, a color signal is generated by adding the first output signal and the second output signal,
When only one of the first output signal and the second output signal has a predetermined level, a color obtained by multiplying a signal obtained by adding the first output signal and the second output signal by a value greater than 1 Generate a signal,
The imaging signal according to claim 13, wherein when both the first output signal and the second output signal have not reached a predetermined level, a color signal is generated using a color detection signal around the corresponding color detection signal. apparatus. A weighting unit that weights the first color detection signal, the second color detection signal, and the third color detection signal according to a light attenuation rate and supplies the weighted signal to the interpolation processing unit; The imaging device according to any one of claims 12 to 15. The interpolation processing unit and the synthesis processing unit are
17. The first color detection signal, the second color detection signal, and the third color detection signal are directly processed as color signals when the color interpolation detection signal is outside an allowable range. Imaging device. A first color detection signal from a first color detection element having transparency in the wavelength range of the first electromagnetic wave;
A second color detection signal from a second color detection element having transparency in the wavelength range of the second electromagnetic wave,
A third color detection signal from a third color detection element having transparency in the wavelength range of the third electromagnetic wave, and a wavelength range including at least a part of the first, second, and third wavelength ranges; In the above, interpolation processing for each of the first color detection signal, the second color detection signal, and the third color detection signal is performed using a color interpolation detection signal from a transparent color interpolation element. And generate multiple signals for each color detection signal,
An image signal processing method for generating a first color signal, a second color signal, and a third color signal by synthesizing a plurality of signals of the generated color detection signals. A first color detection signal from a first color detection element having transparency in the wavelength range of the first electromagnetic wave;
A second color detection signal from a second color detection element having transparency in the wavelength range of the second electromagnetic wave,
A third color detection signal from a third color detection element having transparency in the wavelength range of the third electromagnetic wave, and a wavelength range including at least a part of the first, second, and third wavelength ranges; In the above, interpolation processing for each of the first color detection signal, the second color detection signal, and the third color detection signal is performed using a color interpolation detection signal from a transparent color interpolation element. Processing to generate a plurality of signals for each color detection signal,
A computer is caused to perform image signal processing including a process of generating a first color signal, a second color signal, and a third color signal by synthesizing a plurality of signals of the generated color detection signals. program.

Images