JP6837652B2 - Imaging device and signal processing method - Google Patents

Description

The present invention relates to an image pickup apparatus and a signal processing method capable of appropriately correcting the output signal of a low-sensitivity pixel and reducing the amount of data to be image-processed in a subsequent stage.

Dynamic range is one of the characteristics of image sensors such as CCD and CMOS. The dynamic range means the ratio of the maximum value and the minimum value of the accumulated amount of electric charge generated by photoelectric conversion. If the amount of charge accumulated is within the range of this dynamic range, the image sensor can output an appropriate signal, and image data with gradation can be obtained.

On the other hand, all the subject areas having a brightness equal to or higher than the saturation level of the image sensor are converted at the saturation level, and a phenomenon called overexposure appears. Further, in the case of a subject having a large difference in brightness, not only overexposure but also underexposure that deteriorates the gradation of dark parts may occur, and the image quality may be further deteriorated.

As described above, a technique for expanding the dynamic range has been conventionally proposed as a method for obtaining an image having good gradation for a subject having a large brightness difference.

For example, in the invention disclosed in Patent Document 1, the pixel sensor array includes a plurality of pixel sensors having a first gain and a plurality of pixel sensors having a second gain less than the first gain. In the highlight pixel where the pixel value of the pixel having a gain of 1 is saturated, the correction value calculated as the average value or the weighted average value of the unsaturated normal light sensitivity pixels available in the vicinity and the highlight pixel value. It is configured to be corrected. A light shield is used to manufacture a photodiode whose optical sensitivity is lower than that of a general pixel, and blocks some of the light entering the pixel in order to reduce the sensitivity. Desensitized pixels have a reduced aperture size produced by a layer of dark shielding material 32.

According to the present invention, unlike conventional image sensors, which all use optical sensors with the same optical sensitivity, a sensor using photodiodes with different optical sensitivities is presented, in particular, the sensor cells are usually saturated and bright. It aims to improve both image quality and dynamic range in the image area.

Here, the structure of the image sensor will be described. The image sensor disclosed in Patent Document 2 is a vertical color separation type image sensor that utilizes the characteristics of silicon, unlike the widely known Bayer type image sensor, and further, the uppermost layer (blue sensor) is in the middle. It is an image sensor read out as an array of "114" structure having a resolution four times higher than the resolution of the layer (green sensor) and the third layer (red sensor).

According to the present invention, the uppermost layer is read out at a sufficient resolution, and the intermediate layer and the third layer are read out at a resolution smaller than a sufficient resolution, so that the intermediate layer and the third layer are read out while maintaining a high spatial resolution in the blue channel of the uppermost layer. It is said that the signal-to-noise ratio (SN ratio) in the red and green channels of the three layers can be increased.

Japanese Unexamined Patent Publication No. 2013-081154 Japanese Patent No. 5033808

Here, consider a case where the method disclosed in Patent Document 1 is applied to an imaging device having a structure disclosed in Patent Document 2. In this case, in order to reduce the sensitivity of the vertical color separation type sensor having the 114 structure, it is necessary to block all of the plurality of blue sensors on the uppermost layer. This is to obtain the effect of reducing the sensitivity by shading even in the green and red sensors in the middle and bottom layers.

As the number of low-sensitivity color sensors increases in this way, the amount of data that needs to be corrected when generating final image data also increases. That is, in the case of the 111 structure which is a normal vertical color separation type sensor, the amount of data to be corrected in the subsequent processing is a total of three in the uppermost layer 1, the intermediate layer 1, and the third layer 1. In the case of the 114 structure disclosed in the above, the number of the uppermost layer 4, the intermediate layer 1, and the third layer 1 is six in total, and the amount of data requiring correction processing increases.

The present invention has been made in view of such a situation, and image processing is performed in a subsequent stage while appropriately correcting the output signal of a vertically color-separated low-sensitivity pixel in which a part of the layers is a high-resolution layer. It is an object of the present invention to provide an image pickup apparatus and a signal processing method capable of reducing the amount of data to be output.

In order to achieve the above object, in the image pickup apparatus according to the present invention, a plurality of pixels having a vertical color separation structure in which a plurality of photoelectric conversion layers are laminated in the direction perpendicular to the optical axis are arranged in a two-dimensional direction. The pixels are an image pickup element, which is a high-resolution photoelectric conversion layer in which some of the photoelectric conversion layers have a plurality of photoelectric conversion units in the two-dimensional direction, and an output signal is read from the image pickup element. A read control unit, a signal processing unit that performs predetermined signal processing on the read output signal, and an image processing unit that performs predetermined image processing on the output signal sent from the signal processing unit. The plurality of pixels include a normal pixel that generates an output signal for image processing, and a functional pixel having a plurality of photoelectric conversion layers that are each less sensitive than the sensitivity of the plurality of photoelectric conversion layers of the normal pixel. The processing unit calculates the output signal of the first photoelectric conversion unit among the plurality of photoelectric conversion units of the high-resolution photoelectric conversion layer in the functional pixel based on the output signals of all the photoelectric conversion units of the high-resolution photoelectric conversion layer. The output signal of the replacement processing unit that performs the replacement processing to replace the calculated value and the photoelectric conversion unit excluding the first photoelectric conversion unit of the high-resolution photoelectric conversion layer is subjected to the first correction processing and the output for imaging. It has a first correction processing unit that corrects the signal so that it can be used as a signal, and the image processing unit performs a second correction processing on the output signal of the first photoelectric conversion unit that has undergone replacement processing, and outputs an image for imaging. It is characterized by having a second correction processing unit that corrects the signal so that it can be used.

Further, in the above invention, the imaging apparatus according to the present invention is characterized in that the signal processing unit performs processing by hardware processing and the image processing unit performs processing by software processing.

Further, the imaging apparatus according to the present invention is characterized in that, in the above invention, the calculated value is a simple average value or a weighted average value of the output signals of all the photoelectric conversion units of the high resolution photoelectric conversion layer.

Further, in the imaging apparatus according to the present invention, in the above invention, the first correction processing unit performs the output signal of the photoelectric conversion unit other than the first photoelectric conversion unit in the high resolution photoelectric conversion layer as the first correction processing. The gain correction is performed on the calculated value, and the second correction processing unit performs interpolation correction on the calculated value as the second correction processing.

Further, in the imaging apparatus according to the present invention, in the above invention, the second correction processing unit uses the output signal of the photoelectric conversion layer excluding the high resolution photoelectric conversion layer among the plurality of photoelectric conversion layers as the second correction processing. On the other hand, it is characterized in that gain correction or interpolation correction is performed.

Further, in the imaging apparatus according to the present invention, in the above invention, the signal processing unit further has a defective pixel discriminating unit, and in the defective pixel discriminating unit, whether a plurality of photoelectric conversion units of the high resolution photoelectric conversion layer are defective pixels. When any of the plurality of photoelectric conversion units is a defective pixel, the replacement processing unit excludes the output signal of the defective pixel and performs the replacement processing.

Further, in the imaging apparatus according to the present invention, in the above invention, a plurality of pixels having a vertical color separation structure are formed by stacking three photoelectric conversion layers, and the high resolution photoelectric conversion layer is provided on the uppermost photoelectric conversion layer. It is characterized by being able to be.

Further, the imaging apparatus according to the present invention is characterized in that, in the above invention, the functional pixel has a light-shielding portion above the uppermost photoelectric conversion layer, which limits the amount of light incident inside to a predetermined ratio.

Further, in the signal processing method according to the present invention, a high-resolution photoelectric conversion layer in which three photoelectric conversion layers are laminated in the direction perpendicular to the optical axis and the uppermost photoelectric conversion layer has a plurality of photoelectric conversion units in the two-dimensional direction. This is a signal processing method for an image pickup device composed of a plurality of pixels having a vertical color separation structure, wherein the output signal of the first photoelectric conversion unit among the plurality of photoelectric conversion units of the high resolution photoelectric conversion layer is converted into high resolution photoelectric. For imaging, a replacement processing step of performing a replacement process of replacing the output signals of all the photoelectric conversion units of the conversion layer with the average value, and an image pickup of the output signal of the photoelectric conversion unit other than the first photoelectric conversion unit of the high resolution photoelectric conversion layer. A first correction processing step that corrects the output signal of the first photoelectric conversion unit that has undergone replacement processing so that it can be used as an output signal of the above, and a second correction processing step that corrects the output signal of the first photoelectric conversion unit that has undergone replacement processing so that it can be used as an output signal for imaging. It has a substitution process steps and the first correction processing steps performed by a hardware processing circuit, the second correction processing step and performing the software processing circuit.

According to the image pickup apparatus and the signal processing method according to the present invention, it is possible to reduce the amount of data to be image-processed in the subsequent stage while appropriately correcting the output signal of the low-sensitivity pixel.

It is a block diagram which showed the main structure of the image pickup apparatus which is one Embodiment of this invention. It is a figure explaining the image pickup element, (a) is the sectional view explaining the incident light and its absorption relation, and (b) is the schematic diagram explaining 114 structure. It is a top view for demonstrating the distribution of low-sensitivity pixels in an image sensor. It is a block diagram for demonstrating the processing performed by a signal processing unit and an image processing unit. It is a flowchart which showed the flow of the main processing in a signal processing part.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the accompanying drawings. The present invention is not limited to this embodiment.

The block diagram shown in FIG. 1 shows a main configuration of an image pickup apparatus according to an embodiment of the present invention. The image pickup device 100 shown in this figure includes a photographing optical system 110, an image pickup element 120, a read control unit 130, a signal processing unit 140, an image processing unit 150, a camera CPU 160, and a user interface (I / F) 161. A recording medium interface (I / F) 163 and an image display unit 170 are provided. The photographing optical system 110 is composed of a plurality of lens groups (not shown) including a focus lens group and a zoom lens group. In this figure, only one lens is shown for the sake of simplicity.

The image sensor 120 receives the light beam focused by the photographing optical system 110, performs photoelectric conversion, and outputs an image signal. A CMOS image sensor is used as the image sensor 120 of this embodiment. The light receiving surface of the image sensor 120 is composed of a large number of pixels. These pixels are vertical color separation type image sensors capable of outputting each color component signal of RGB from a single pixel by using the difference in depth that is photoelectrically converted by the wavelength of incident light inside. The vertical color separation type image sensor will be described in detail later.

Further, the image pickup device 120 includes a plurality of normal pixels and a plurality of low-sensitivity pixels, each of which constitutes the image pickup device 120 with a predetermined arrangement. The image sensor 120 includes a gain variable amplifier, a gain correction circuit, and an A / D converter (not shown) inside, and the image signal is output as digital data.

The low-sensitivity pixel is a pixel used for the purpose of obtaining a photographed image (so-called HDR image) having an expanded dynamic range, for example. In the present embodiment, this low-sensitivity pixel is adopted as one type of functional pixel, but the present invention is not limited to this, and the phase difference detection pixel for realizing so-called imaging surface phase difference AF as a functional pixel is used. May be adopted. The low-sensitivity pixels will be described in detail later.

The read-out control unit 130 outputs a signal for determining the drive timing of the image sensor 120. As a result, the horizontal drive and the vertical drive for each pixel are controlled, and each RGB color component signal is read out from each pixel. The color component signal output from the image sensor 120 is recorded in the memory unit 141 provided in the signal processing unit 140. The memory unit 141 also functions as a buffer for various data.

The signal processing unit 140 performs various signal processing on the color component signal read from the image sensor 120 and recorded in the memory unit 141. As the signal processing, the color component signal is subjected to the first correction processing and replacement processing. In this first correction process and replacement process, correction is performed on the output signal from the low-sensitivity pixel. The first correction process and replacement process will be described in detail later.

As another process, for example, there is an amplification process for amplifying the read color component signal.

The signal processing unit 140 is a front-end device that performs pre-stage processing on the output of the image sensor 120. For example, it is a logic circuit composed of ICs (semiconductors) that execute image processing such as FPGA and ASIC, and performs predetermined hardware processing (specific data processing by dedicated hardware).

The image processing unit 150 performs various image processing on the color component signal sent from the signal processing unit 140. As image processing, a second correction process, RAW data generation, and HDR image generation are performed on the color component signal. In this second correction process, the output signals of the low-sensitivity pixels and the normal pixels are corrected. The second correction process will be described in detail later.

Other processes include, for example, white balance processing, color reproduction processing, development processing for JPEG format and TIFF format image data, and the like.

The image processing unit 150 is a back-end device that performs post-stage processing on the signal output from the signal processing unit 140. A processor that performs certain software-programmed data processing, such as a DSP.

The camera CPU 160 controls the entire image pickup apparatus 100 comprehensively.

The user I / F161 has, for example, operating members such as a release button, a power button, a command dial, and a cross key. When the user operates these operating members, the camera CPU 160 issues an instruction to perform a predetermined operation. ..

The recording medium I / F163 records or reads RAW data and developed image data with a recording medium (not shown). This recording medium is a removable recording medium such as a semiconductor memory.

The image display unit 170 displays a captured image, an image read from a recording medium (not shown), or the like.

When the image sensor 120 that does not have the above-mentioned gain variable amplifier, gain correction circuit, and A / D converter built-in is adopted, these devices may be mounted individually.

FIG. 2A is a simplified cross-sectional view showing a three-layer photodiode of a vertical color-separated image sensor 120 mounted on the image pickup device 100. As described above, the image sensor 120 of the present embodiment is a so-called vertical color separation type image sensor, and is simply configured by stacking three photodiodes, which are photoelectric conversion layers, in the depth direction. There is.

When light is incident on a certain pixel, the blue (B) component in the incident light is photoelectrically converted mainly by the B photodiode 121 located in the uppermost layer. Similarly, the green (G) component in the incident light is photoelectrically converted mainly by the G photodiode 123 located in the intermediate layer, and the red (R) component is photoelectrically converted mainly by the R photodiode 125 located in the bottom layer. The diode. These vertical color separations utilize the characteristics of silicon (Si) used as a material for the image sensor 120. The three-layer photodiode is formed by performing predetermined doping treatments at different depths inside the Si substrate. Specifically, the B photodiode 121 is formed to a depth between about 0.2 and 0.5 μm, the G photodiode 123 is formed to a depth between about 0.5 and 1.5 μm, and the R The photodiode 125 is formed to a depth between about 1.5 and 3.0 μm.

Therefore, the image sensor 120 of the present embodiment can acquire the color component signals of three RGB colors with one pixel, without the need for the color filter essential for the Bayer type image sensor. Since each pixel can photoelectrically convert the wavelength components of all three colors, there is also an advantage that it is not necessary to perform pixel interpolation, which is indispensable for a Bayer type image sensor.

Further, as shown in FIG. 3B, the image pickup device 120 of the present embodiment has the “114” structure disclosed in Patent Document 2 described above. That is, the number of B photodiodes 121 is four times that of the corresponding number of G photodiodes 123 and R photodiodes 125, and the area of the B photodiodes 121 itself is smaller than that of GR photodiodes. ..

A microlens (not shown) is provided above the B photodiode 121 so as to correspond to each B photodiode 121. The description of B0, B1, B2, and B3 indicates the order of signal reading of the B photodiode 121, which will be described later, and each pixel is referred to as such as it is.

When the term "pixel" is used in this embodiment, it mainly means the minimum configuration of the 114 structure including the four B photodiodes 121, one G photodiode 123, and the R photodiode as described above. To do. According to this, a total of 6 types of color component signals can be obtained from one "pixel".

Instead of the photodiode, a plurality of photoelectric conversion films formed of an organic substance, an inorganic substance, or the like and having specific absorption characteristics may be laminated as a photoelectric conversion layer.

As described above, the image sensor 120 of the present embodiment has a normal pixel and a low-sensitivity pixel, both of which have the 114 structure described above. The low-sensitivity pixel is a pixel configured so that each photodiode has a lower sensitivity than a normal pixel, and other structures are the same as those of a normal pixel.

There are multiple possible structures for obtaining low-sensitivity pixels. For example, a light-shielding portion that serves as a mask is formed by etching metal or the like on the upper surface of the pixel, thereby limiting the opening area and reaching the photodiode in the pixel. There is a way to reduce the amount of light. The low-sensitivity pixel of the present embodiment has a configuration in which a metal mask (not shown) corresponding to each of the four B photodiodes 121 on the uppermost layer is provided above each photodiode.

With this configuration, not only the amount of incident light of the B photodiode 121 as a whole can be reduced, but also the amount of incident light of the G photodiode 123 and the R photodiode 125 located in the lower layer can be reduced in the same manner. Is possible. As a result, each photodiode of the low-sensitivity pixel has a lower sensitivity than each photodiode of the normal pixel.

FIG. 3 is a diagram schematically showing the arrangement of each pixel when viewed from above the image sensor 120, and each rectangle represents the pixel of the 114 structure described above. White rectangles represent normal pixels, and dark rectangles represent low-sensitivity pixels. As shown in this figure, the low-sensitivity pixels are discretely distributed among the normal pixels arranged in a two-dimensional manner.

The read-out control unit 130 outputs a signal for determining the drive timing of the image pickup device 120 in order to read out for each pixel sequence. As a result, the horizontal drive and the vertical drive for each pixel are controlled, and each RGB color component signal is read out from each pixel. Specifically, in the row selected by the horizontal drive signal, first, the four B photodiodes 121 on the uppermost layer are read out in a predetermined order. Each color component signal read from the image sensor 120 is sequentially sent to the signal processing unit 140.

In this embodiment, as described above, the color component signals stored in each B photodiode 121 are read out in the order of B0, B1, B2, and B3 as a predetermined reading order. That is, when a certain row is selected and reading is started, B0 of all the pixels in the row is first read in order from the end, and when the reading of B0 is completed, B1 is read from all the pixels in the same row. It is read, and then B2 and then B3 are read. When the reading from the four types of B photodiodes 121 is completed, the signal of the intermediate layer G photodiode 123 is subsequently read from the same row, and finally the signal of the lowest layer R photodiode 125 is read.

As a result, six types of color component signals of B0, B1, B2, B3, G, and R are read out from all the pixels in one line. Then, by performing the above reading for all the rows on the image sensor 120, the reading of each color component signal for one frame of the image is completed.

As described above, since the low-sensitivity pixels are discretely distributed in the image sensor 120, the signal from the low-sensitivity pixel is included in the color component signal read out depending on the selected row. Processing for appropriately handling the output signal from the low-sensitivity pixel is performed by the signal processing unit 140, which is responsible for the pre-stage processing, and the image processing unit 150, which is responsible for the post-stage processing. FIG. 4 is a schematic view showing the processing according to the present invention performed by the signal processing unit 140 and the image processing unit 150.

As described above, since the signal processing unit 140 is a logic circuit that performs hardware processing, it is characterized in that it can perform high-speed processing as compared with the image processing unit 150 in the subsequent stage. On the other hand, since the image processing unit 150 is a processor that performs software processing, it has a feature that the processing program can be easily changed as compared with the signal processing unit 140 in the previous stage.

First, the signal processing unit 140 will be described. The color component signal input from the image sensor 120 to the signal processing unit 140 is first determined by the signal discrimination unit 143 whether or not the predetermined condition is satisfied, and the signal satisfying the condition is appropriately replaced with the first correction processing unit 145 or the replacement. It is sent to the processing unit 147.

The first correction processing unit 145 determines the B component signals read from the three B photodiodes 121 other than the representative photodiode Br, which will be described later, among the B component signals read from the low-sensitivity pixels. Perform correction processing. The three B component signals subjected to these processes are sent to the image processing unit 150.

On the other hand, in the replacement processing unit 147, among the B component signals read from the low-sensitivity pixels, for example, the B component signal read from the B photodiode 121 located at B0 described above is subjected to predetermined replacement processing. .. This B0 photodiode, which is the target of the replacement process in the replacement process unit 147, is referred to as a representative photodiode Br.

The representative photodiode Br is not limited to B0. Any of the four B photodiodes 121 can be set as the representative photodiode Br.

The signal value of the representative photodiode Br is replaced by the signal processing unit 140 and then sent to the image processing unit 150 as a replacement signal value.

The flowchart of FIG. 5 shows the main processing flow in the signal processing unit 140. As described above, the signal processing unit 140 performs different processing depending on the type of pixel or photodiode from which the color component signal is read.

First, in step S101, it is determined whether or not the color component signal read from the pixel is a signal read from the low-sensitivity pixel. If the signal is not read from the low-sensitivity pixel, the flowchart is terminated and the signal is sent to the image processing unit 150 in the subsequent stage without performing any processing. On the other hand, if the signal is read from the low-sensitivity pixel, the process proceeds to step S102.

Next, in step S102, it is determined whether or not the read color component signal is a signal read from the B photodiode 121. If the signal is not read from the B photodiode 121, the flowchart is terminated and the signal is sent to the subsequent image processing unit 150 without any processing. On the other hand, if the signal is read from the B photodiode 121, the process proceeds to step S103.

Next, in step S103, it is determined whether or not the read color component signal is a signal read from the representative photodiode Br. If the signal is not read from the representative photodiode Br, the process proceeds to step S104, while if the signal is read from the representative photodiode Br, the process proceeds to step S105.

These determinations in the flowchart are performed by the signal determination unit 143 provided in the signal processing unit 140. At the time of discrimination, the signal discrimination unit 143 counts, for example, the number of read B photodiodes 121 from the end of the same row, and the counted number and the position of the low-sensitivity pixel stored in advance (the number from the end). Information) is compared to make a judgment.

In step S104, the first correction process is applied to the read color component signal. Specifically, the first correction processing unit 145 in the signal processing unit 140 refers to the signal values read from the three types of B photodiodes 121 other than the representative photodiode Br, and the signal values lost due to the decrease in sensitivity. The first correction process for calculating the correction signal values B1', B2', and B3' corrected by the above is performed. This correction may be either a gain correction using a gain value stored in advance or an interpolation correction that interpolates the signal value at the low-sensitivity pixel position from the signal values of the surrounding normal pixels.

Further, in step S105, the read color component signal is subjected to replacement processing. Specifically, the replacement processing unit 147 in the signal processing unit 140 calculates the average value Bav of the signal values before correction of the four types of B photodiodes 121 including the representative photodiode Br, and the average signal value Bav is used as the representative photo. A process of replacing the signal value of the diode Br is performed.

The representative photodiode Br described above is the average value of the signal values of the four types of B photodiode 121, but this may be a simple average value or a weighted average value. The weighting coefficient is arbitrarily determined, but can be determined in consideration of a minute difference in sensitivity between the B photodiodes based on the internal structure of the pixel such as the wiring layer. When it is clear that the sensitivity of B0 is relatively low due to the influence of the wiring layer, the influence of unnecessary noise on the average value Bav can be reduced by setting the weighting coefficient for B0 low.

By the above two types of processing, the signal values of the four types of B photodiodes 121 that were in the low-sensitivity state are corrected to one signal value (Bav) in the low-sensitivity state so that they can be used in the image data. It means that three values (B1', B2', B3') have been calculated. These signals are output to the image processing unit 150. This ends this flowchart.

Returning to the description of FIG. 4, the image processing unit 150 that is responsible for the post-stage processing will be described. The average signal value Bav, the G component signal, and the R component signal input from the signal processing unit 140 are sent to the second correction processing unit 151 provided inside the image processing unit 150. In addition, these signal values are also sent to the HDR image generation unit 155, which will be described later.

The second correction processing unit 151 performs a predetermined correction processing on the average signal value Bav input from the signal processing unit 140. Similarly, the second correction processing unit 151 also performs predetermined correction processing on the G component signal and the R component signal read from the low-sensitivity pixels.

These correction processes performed by the second correction processing unit 151 are processes for correcting the signal values lost due to the decrease in sensitivity with respect to the signal values obtained by the low-sensitivity pixels.

Specifically, the second correction processing unit 151 performs known pixel interpolation processing on the average signal value Bav to generate the corrected signal value Bav'. This pixel interpolation process is a process of estimating the signal value of the pixel position from the signal value of the surrounding normal pixels and correcting it.

This pixel interpolation processing can be prepared as a dedicated interpolation processing for the average signal value Bav, but it is also possible to apply a so-called defect pixel correction processing. The defective pixel correction process is, for example, registering a defective pixel in a camera in an adjustment process before factory shipment, and interpolating and correcting the signal value of the defective pixel position at the time of shooting using the surrounding normal pixel signal value. It is a process to do.

Since this defect pixel correction process is generally performed in the image processing unit in the subsequent stage, applying this process to the correction of the average signal value Bav also has merits such as reduction of development resources and compression of program capacity.

Further, the G component signal and the R component signal read from the low-sensitivity pixels may be subjected to the pixel interpolation processing performed on the replacement signal value in the second correction processing unit 151 in advance. Gain correction may be performed to correct the signal value by multiplying the stored gain value. As a result, the corrected signal values G'and R'are generated.

Each signal value that has undergone correction processing by the second correction processing unit 151 is sent to the RAW data generation unit 153. The six types of signal values (Bav', B1', B2', B3', G', R') input to the RAW data generation unit 153 are initially low-sensitivity signals read from the low-sensitivity pixels. Although it was a value, the signal value was corrected by the above-mentioned first correction process and the second correction process. The RAW data generation unit 153 generates RAW data from these signal values and the signal values read from the normal pixels.

In the HDR image generation unit 155, if necessary, HDR (high) is used with the generated RAW data and three types of signal values (Bav, G, R) in the low sensitivity state read from the same low sensitivity pixel. Dynamic range) An image is generated. For the generation of the HRD image, a known technique such as Patent Document 1 described above can be used. Whether or not to perform this HDR image generation processing can be arbitrarily set by the user, for example, via the user I / F161.

The generated RAW data and HDR image are sent to the storage medium I / F via the camera CPU 160 and recorded. In addition, an image for confirming the shooting result is displayed on the image display unit 170.

As described above, in the image sensor 120 in which a plurality of low-sensitivity signal values are included in the B color component due to the 114 structure, the signal processing unit 140 in the previous stage leaves only one low-sensitivity signal of the B component. It was decided that the other three low-sensitivity signals were corrected in advance, and the remaining one B component low-sensitivity signal was corrected by the image processing unit 150 in the subsequent stage. With such a configuration, it is possible to increase the division of processing in the signal processing unit having excellent processing speed, reduce the load on the image processing unit that also performs other general processing, and prevent the occurrence of processing delay. Is possible.

Further, instead of using the signal value of the representative photodiode Br as it is as one remaining B component low-sensitivity signal, the average value Bav of four signals including the signal values of the other B photodiodes constituting the pixel is replaced. By using the 114 structure, it is possible to average and reduce the noise of each B photodiode that has increased relatively.

On the other hand, it is also possible to use the signal value B0 as it is without replacing the signal value of the representative photodiode Br with the average value Bav.

Next, consider the case where the photodiodes that make up the pixels are defective. For example, when B0, which is a representative photodiode Br, is a defective pixel, the information is registered in the camera in the adjustment process before shipment from the factory of the image pickup apparatus 100, so that the information is not shown inside the signal processing unit 140. The defective pixel discriminating unit of the above discriminates and identifies defective pixels with reference to the registration information, and calculates the average signal value Bav from the signal values of the three B photodiodes 121 of B1 to B3 excluding the defective pixels. This makes it possible to prevent the average signal value Bav from being affected by the abnormal signal value output by the defective pixel B0.

Similarly, even when defective pixels are included in B1 to B3 located in the same pixel as B0 which is the representative photodiode Br, the signal processing unit 140 excludes the defective pixels by referring to the registration information. The average signal value Bav can be calculated.

In addition, when the signal value of the representative photodiode Br is used as it is without being replaced by the average value, if the representative photodiode Br becomes a defective pixel, the accuracy of the information required for generating the HDR image is lowered. Resulting in. Therefore, only when the representative photodiode Br is a defective pixel, it is possible to replace it with the average signal value Bav of the signal values of B1 to B3 in the same pixel.

In the image pickup apparatus of the above-described embodiment, an example in which the present invention is applied to an image pickup device having a so-called 114 structure is shown, but the present invention is not limited thereto.

For example, even an image sensor having a 112 structure in which the uppermost layer is bisected or a 118 structure in which the uppermost layer is bisected can exhibit the effects of the present invention. That is, even in the case of these image sensors, the signal value of the representative photodiode Br is replaced with the average signal value Bav in the signal processing unit, and the signal values of the other B photodiodes are interpolated, as in the above-described embodiment. Just do it.

Further, in the above-described embodiment, the B photosensitive layer, which is the uppermost layer, is divided into four to form a high-resolution layer, but the present invention is not limited to this. That is, the photodiode may be divided into a plurality of layers with the G photosensitive layer, which is an intermediate layer other than the uppermost layer, and the R photosensitive layer, which is the lowest layer, as a high resolution layer. In this case as well, as in the above-described embodiment, the signal processing unit may perform a process of replacing one of the plurality of photoelectric conversion units with an average value.

As described above, according to the image pickup apparatus described in the present invention, in an image pickup device having a vertical color separation type low-sensitivity pixel in which a part of the layers is a high-resolution layer, the signal processing unit in the previous stage is high. It was decided that only one of the low-sensitivity signals in the resolution layer was left and the other low-sensitivity signals were corrected in advance, and the remaining one low-sensitivity signal was corrected in the image processing unit in the subsequent stage.

With such a configuration, it is possible to increase the division of processing in the signal processing unit having excellent processing speed, reduce the load on the image processing unit that also performs other general processing, and prevent the occurrence of processing delay. Is possible.

100 image pickup device, 110 imaging optical system, 120 image sensor, 121 B photodiode, 123 G photodiode, 125 R photodiode, 130 read control unit, 140 signal processing unit, 141 memory unit, 143 signal discriminator unit, 145 first Correction processing unit, 147 replacement processing unit, 150 image processing unit, 151 second correction processing unit, 153 RAW data generation unit, 155 HDR image generation unit, 160 camera CPU, 161 user interface, 163 recording medium interface, 170 images. Display

Claims (9)

A plurality of pixels having a vertical color separation structure formed by stacking a plurality of photoelectric conversion layers in the vertical direction of the optical axis are arranged in a two-dimensional direction, and the plurality of pixels are a part of the photoelectrics of the plurality of photoelectric conversion layers. An image pickup element, which is a high-resolution photoelectric conversion layer in which the conversion layer has a plurality of photoelectric conversion units in the two-dimensional direction,
A read control unit that reads an output signal from the image sensor,
A signal processing unit that performs predetermined signal processing on the read output signal, and
An image processing unit that performs predetermined image processing on the output signal sent from the signal processing unit, and
Have,
The plurality of pixels include a normal pixel that generates an output signal for imaging, and a functional pixel having a plurality of the photoelectric conversion layers having a sensitivity lower than the sensitivity of the plurality of photoelectric conversion layers of the normal pixel.
The signal processing unit in the functional pixel
A calculated value calculated based on the output signals of the first photoelectric conversion unit among the plurality of photoelectric conversion units of the high-resolution photoelectric conversion layer, based on the output signals of all the photoelectric conversion units of the high-resolution photoelectric conversion layer. A replacement processing unit that performs replacement processing to replace with
A first correction processing unit that performs a first correction process on the output signal of the photoelectric conversion unit excluding the first photoelectric conversion unit in the high-resolution photoelectric conversion layer so that it can be used as an output signal for imaging. When,
Have,
The image processing unit has a second correction processing unit that performs a second correction process on the output signal of the first photoelectric conversion unit that has undergone the replacement process so that it can be used as an output signal for imaging. imaging device, characterized in that. The image pickup apparatus according to claim 1, wherein the signal processing unit performs processing by hardware processing, and the image processing unit performs processing by software processing. The imaging device according to claim 1 or 2, wherein the calculated value is a simple average value or a weighted average value of the output signals of all the photoelectric conversion units of the high-resolution photoelectric conversion layer. As the first correction process, the first correction processing unit performs gain correction on the output signal of the photoelectric conversion unit excluding the first photoelectric conversion unit in the high resolution photoelectric conversion layer.
The imaging apparatus according to any one of claims 1 to 3, wherein the second correction processing unit performs interpolation correction on the calculated value as the second correction processing. As the second correction process, the second correction processing unit performs gain correction or interpolation correction on the output signal of the photoelectric conversion layer excluding the high resolution photoelectric conversion layer among the plurality of photoelectric conversion layers. The imaging apparatus according to any one of claims 1 to 4. The signal processing unit further has a defective pixel discriminating unit.
The defective pixel determination unit determines whether or not the plurality of photoelectric conversion units of the high-resolution photoelectric conversion layer are defective pixels.
Any of claims 1 to 5, wherein when any of the plurality of photoelectric conversion units is a defective pixel, the replacement processing unit performs the replacement processing by excluding the output signal of the defective pixel. The imaging device described in Crab. The plurality of pixels having the vertical color separation structure are formed by stacking three layers of the photoelectric conversion layer.
The imaging device according to claim 1 to 6, wherein the high-resolution photoelectric conversion layer is provided on the uppermost photoelectric conversion layer. The imaging device according to claim 1 to 7, wherein the functional pixel has a light-shielding portion that limits the amount of light incident inside to a predetermined ratio above the photoelectric conversion layer on the uppermost layer. A plurality of pixels having a vertical color separation structure in which three photoelectric conversion layers are laminated in the vertical direction of the optical axis, and the uppermost photoelectric conversion layer is a high-resolution photoelectric conversion layer having a plurality of photoelectric conversion units in a two-dimensional direction. It is a signal processing method of an image sensor consisting of
Substitution processing that replaces the output signal of the first photoelectric conversion unit among the plurality of photoelectric conversion units of the high-resolution photoelectric conversion layer with the average value of the output signals of all the photoelectric conversion units of the high-resolution photoelectric conversion layer. And the replacement process
A first correction processing step of correcting the output signal of the photoelectric conversion unit excluding the first photoelectric conversion unit of the high-resolution photoelectric conversion layer so that it can be used as an output signal for imaging.
A second correction processing step of correcting the output signal of the first photoelectric conversion unit that has undergone the replacement processing so that it can be used as an output signal for imaging, and
Have,
Signal processing method, which comprises carrying out the replacement processing step and the first correction processing steps performed by a hardware processing circuit, the second correction processing steps by software processing circuit.

Images