JP2017126898A - Imaging device and video processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device and a video processing apparatus capable of acquiring an image corresponding to the intension of a user by correcting a pixel value of a defective pixel, when the defective pixel exists in an image pick-up device having a pixel including multiple photoelectric conversion parts of different light reception efficiency.SOLUTION: An imaging device includes: an image pick-up device having multiple pixels each including a first pixel element having a first photoelectric conversion part, and a second pixel element having a second photoelectric conversion part of different sensitivity from that of the first photoelectric conversion part, and acquiring a first video signal based on a signal charge generated in the first photoelectric conversion part, and a second video signal synchronized with the first video signal based on the signal charge generated in the second photoelectric conversion part; and a signal processing part for calculating a pixel value of a defective pixel element in the first video signal, on the basis of a first pixel value of the second pixel element of a defective pixel, and a second pixel value of the first pixel element of other pixel adjacent to the defective pixel when the defective pixel, where the first pixel element is a defective pixel element, is included.SELECTED DRAWING: Figure 34

Description

  The present invention relates to an imaging apparatus and a video processing apparatus.

  If you can shoot movies and still images simultaneously with a single camera, you can watch the shooting scene as a movie and enjoy the decisive scene in the movie as a still image, greatly increasing the value of the shot image. Can be increased. Also, if you can shoot a normal frame rate video and a high frame rate video at the same time with a single camera, you can enjoy a high-quality work while switching a specific scene to a slow motion video. It can convey a dynamic feeling. In addition, as a technique for making movies and home television images more realistic, there is an HDR (high dynamic range) technique for moving images. This combines the output from two pixel groups with different sensitivities to expand the brightness reproduction range of the display screen, and provides a more realistic sensation than before, mainly by momentary or partial brightness increase. To do.

  As an imaging apparatus for capturing such an image, an imaging apparatus using an imaging element having pixels including a plurality of photoelectric conversion units having different light receiving efficiency has been proposed. By the way, in an image sensor, it is difficult to completely eliminate pixels that cannot output signals normally (so-called defective pixels). Therefore, a technique for correcting pixel values when defective pixels exist has been proposed. Yes.

  In Patent Document 1, in an imaging device in which each unit pixel has a plurality of photoelectric conversion units, the pixel value of a defective photoelectric conversion unit is changed to the pixel value of another photoelectric conversion unit included in the same pixel or another pixel. The correction is described based on the pixel value of the photoelectric conversion unit included. In Patent Document 2, in a solid-state imaging device in which each pixel has a main photosensitive pixel and a secondary photosensitive pixel, when the secondary photosensitive pixel is defective, the pixel value of the secondary photosensitive pixel is set as the pixel value of the primary photosensitive pixel. It is described that the correction is made based on the above.

Japanese Patent Laying-Open No. 2015-026710 JP 2004-222177 A

  However, there are various requests that the user requests for video, and when the pixel values of defective pixels are corrected by the methods described in Patent Document 1 and Patent Document 2, a video that reflects the user's intention is not necessarily obtained. Not always.

  An object of the present invention is to appropriately correct a pixel value of a defective pixel and display an image according to a user's intention when an image sensor having pixels including a plurality of photoelectric conversion units having different light receiving efficiencies exists. An object of the present invention is to provide an image pickup apparatus and a video processing apparatus that can be acquired.

  According to one aspect of the present invention, a first pixel element including a first photoelectric conversion unit, and a second pixel element including a second photoelectric conversion unit having a sensitivity different from that of the first photoelectric conversion unit, The first video signal based on the signal charge generated in the first photoelectric conversion unit and the video signal based on the signal charge generated in the second photoelectric conversion unit When the imaging device that acquires the second video signal synchronized with the first video signal and the plurality of pixels include a defective pixel in which the first pixel element is a defective pixel element. In addition, the pixel value of the defective pixel element in the first video signal is set to the first pixel value of the second pixel element of the defective pixel and the first pixel value of another pixel adjacent to the defective pixel. A signal processing unit that calculates based on the second pixel value of the pixel element; Imaging device that is provided.

  According to another aspect of the present invention, a second pixel including a first pixel element including a first photoelectric conversion unit and a second photoelectric conversion unit having a sensitivity different from that of the first photoelectric conversion unit. Image data acquired by an image sensor having a plurality of pixels each including a first pixel data based on the signal charge generated in the first photoelectric conversion unit, and the second A video data acquisition unit that acquires video data based on the signal charges generated in the photoelectric conversion unit of the second video data synchronized with the first video data; and the plurality of pixels of the imaging device When a defective pixel in which the first pixel element is a defective pixel element is included, the pixel value of the defective pixel element in the first video data is calculated as the second pixel element of the defective pixel. 1 pixel value and the defective pixel Video processing apparatus and a correction processing unit for correcting, based on the second pixel value of the first pixel element of the other adjacent pixels is provided.

  According to another aspect of the present invention, a second pixel including a first pixel element including a first photoelectric conversion unit and a second photoelectric conversion unit having a sensitivity different from that of the first photoelectric conversion unit. Image data acquired by an image sensor having a plurality of pixels each including a first pixel data based on the signal charge generated in the first photoelectric conversion unit, and the second Image data based on the signal charges generated in the photoelectric conversion unit of the second image data, the second image data synchronized with the first image data, and acquiring the first image data in the plurality of pixels of the image sensor. When a defective pixel whose pixel element is a defective pixel element is included, a pixel value of the defective pixel element in the first video data is set to a first pixel value of the second pixel element of the defective pixel. , Of other pixels adjacent to the defective pixel Image processing method of correcting, based on the second pixel values of the serial first pixel element is provided.

  According to the present invention, when a defective pixel exists in an imaging device having pixels including a plurality of photoelectric conversion units having different light receiving efficiencies, an image according to the user's intention is obtained by appropriately correcting the pixel value of the defective pixel. Can be acquired.

1 is an external view showing an imaging apparatus according to an embodiment of the present invention. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to an embodiment of the present invention. It is a block diagram which shows the structural example of the image pick-up element of the imaging device by one Embodiment of this invention. It is sectional drawing which shows the internal structure of the image pick-up element in the imaging device by one Embodiment of this invention. It is a graph which shows the relationship between the angle of the light ray which injects into a pixel, and the output from a photodiode. It is a figure which shows the relationship between the imaging optical system and imaging device in the imaging device by one Embodiment of this invention. It is the schematic explaining the video signal output from an image sensor. It is a circuit diagram showing an example of composition of a pixel of an image sensor of an imaging device by one embodiment of the present invention. FIG. 3 is a first layout diagram illustrating a main part of a pixel of an imaging element of the imaging apparatus according to the embodiment of the present invention (part 1); It is a plane layout figure (the 2) showing an important section of a pixel of an image sensor of an imaging device by one embodiment of the present invention. It is a circuit diagram which shows the block diagram of the read-out circuit of the image pick-up element of the imaging device by one Embodiment of this invention. It is a timing chart which shows the drive sequence of an image sensor. It is a graph which shows the time change of the signal charge in a photodiode. FIG. 10 is a potential diagram of a pixel along the line AB in FIG. 9. It is sectional drawing which shows the behavior of the signal charge which generate | occur | produced by propagation of the light in an inside of an image pick-up element, and photoelectric conversion. It is a timing chart for demonstrating the imaging sequence in the imaging device by one Embodiment of this invention. It is a figure which shows an example of the value of the time code added to each frame of moving image data. It is a figure which shows an example of the file structure of "picture A" and "picture B". It is a figure explaining the setting screen of the imaging conditions of "picture A" and "picture B". It is a figure which shows the relationship of the ISO sensitivity range of "picture A" and "picture B". It is a program AE diagram in the dual video mode of the imaging device according to the embodiment of the present invention. It is a figure explaining the difference of the shutter speed of "picture A" and "picture B" on an imaging sequence. It is a figure showing the mode of the display part in the live view display after powering on an imaging device. It is a figure which shows 1 frame of the images | videos acquired by operating switch ST and switch MV. It is a figure which shows an example of the image | video after performing crosstalk correction | amendment. It is a figure which shows a mode that "picture A" and "picture B" were displayed side by side on the display part. It is a figure for demonstrating the utilization example of "picture A" and "picture B" stored in the storage. It is a figure for demonstrating the accumulation | storage operation | movement of "picture A" and "picture B" on an imaging sequence. It is a figure explaining the signal level and composition ratio of "picture A" and "picture B" when performing HDR imaging | photography. It is a figure which shows the example of "picture A" and "picture B" for HDR synthetic | combination. It is a figure which shows the state which displayed the result which synthesize | combined the HDR image | video after the crosstalk correction process on the display part. It is a figure explaining a defective pixel element and a pixel element used for correction of the pixel value. It is a figure which shows an example of a to-be-photographed object. 3 is a flowchart illustrating a video processing method according to an embodiment of the present invention. It is a figure which shows an example of the process in information processing apparatus.

  An imaging apparatus according to an embodiment of the present invention will be described with reference to FIGS. In the present embodiment, an image pickup apparatus including an image pickup device for image pickup, a shooting optical system, and the like and a video processing apparatus will be described as an example of a preferred embodiment of the present invention. However, the video processing apparatus is not necessarily configured as a part of the imaging apparatus, and may be configured by hardware different from the imaging element and the imaging optical system. Moreover, you may make it mount all or one part of the function of a video processing apparatus in an image pick-up element.

  FIG. 1 is an external view of a digital still motion camera as an example of the imaging apparatus according to the present embodiment. FIG. 1A shows a front view thereof, and FIG. 1B shows a rear view thereof.

  The imaging apparatus 100 according to the present embodiment includes a housing 151, a photographing optical system 152 provided on the front portion of the housing 151, and a switch ST <b> 154 and a propeller 162 provided on the upper surface of the housing 151. Yes. In addition, the imaging apparatus 100 includes a display unit 153, a switch MV155, a shooting mode selection lever 156, a menu button 157, up / down switches 158 and 159, a dial 160, and a playback button on the rear surface of the casing 151. 161.

  The casing 151 is a container that houses various functional components that constitute the imaging device 100 such as an imaging element and a shutter device. The photographing optical system 152 is an optical system for forming an optical image of a subject. The display unit 153 includes a display device for displaying shooting information and video. The display unit 153 may be provided with a movable mechanism for changing the orientation of the screen as necessary. The display unit 153 has a display luminance range that can display an image with a wide dynamic range without suppressing the luminance range. The switch ST154 is a shutter button used mainly for taking a still image. The switch MV155 is a button for starting and stopping moving image shooting. The shooting mode selection lever 156 is a changeover switch for selecting a shooting mode. The menu button 157 is a button for shifting to a function setting mode for setting a function of the imaging apparatus 100. Up / down switches 158 and 159 are buttons used when changing various setting values. The dial 160 is a dial for changing various setting values. The playback button 161 is a button for shifting to a playback mode in which video recorded on a recording medium stored in the imaging apparatus 100 is played on the display unit 153. The propeller 162 is for floating the imaging device 100 in the air in order to perform shooting from the air.

  FIG. 2 is a block diagram illustrating a schematic configuration of the imaging apparatus 100 according to the present embodiment. As shown in FIG. 2, the imaging apparatus 100 includes an aperture 181, an aperture control unit 182, an optical filter 183, an image sensor 184, analog front ends 185 and 186, digital signal processing units 187 and 188, and a timing generation unit 189. ing. Further, the imaging apparatus 100 includes a system control CPU 178, a switch input unit 179, a video memory 190, and a flight control apparatus 200. Further, the imaging apparatus 100 includes a display interface unit 191, a recording interface unit 192, a recording medium 193, a print interface unit 194, an external interface unit 196, and a wireless interface unit 198.

  The image sensor 184 is for converting an optical image of a subject formed through the photographing optical system 152 into an electrical video signal. The image sensor 184 is not particularly limited, and has, for example, a sufficient number of pixels, a signal readout speed, a color gamut, and a dynamic range to satisfy the UHDTV (Ultra High Definition Television) standard. The diaphragm 181 is for adjusting the amount of light passing through the photographing optical system 152. The aperture control unit 182 is for controlling the aperture 181. The optical filter 183 is for limiting the wavelength of light incident on the image sensor 184 and the spatial frequency transmitted to the image sensor 184. The photographing optical system 152, the diaphragm 181, the optical filter 183, and the image sensor 184 are disposed on the optical axis 180 of the photographing optical system 152.

  The analog front ends 185 and 186 are for performing analog signal processing and analog-digital conversion processing of the video signal output from the image sensor 184. The analog front ends 185 and 186 include, for example, a correlated double sampling (CDS) circuit that removes noise, an amplifier that adjusts a signal gain, an A / D converter that converts an analog signal into a digital signal, and the like. The digital signal processing units 187 and 188 are for performing various corrections on the digital video data output from the analog front ends 185 and 186 and then compressing the video data. In this sense, the digital signal processing units 187 and 188 are also correction processing units. The corrections performed by the digital signal processing units 187 and 188 include correction of pixel values of defective pixels, which will be described later, and crosstalk correction. The timing generator 189 is for outputting various timing signals to the image sensor 184, the analog front ends 185 and 186, and the digital signal processors 187 and 188. The system control CPU 178 is a control unit that performs various calculations and controls the entire imaging apparatus 100. The video memory 190 is for temporarily storing video data.

  The display interface unit 191 is an interface between the system control CPU 178 and the display unit 153 for displaying captured images on the display unit 153. The recording medium 193 is a recording medium such as a semiconductor memory for recording video data, additional data, and the like, and may be provided in the imaging apparatus 100 or detachable. The recording interface unit 192 is an interface between the system control CPU 178 and the recording medium 193 for recording on or reading from the recording medium 193. The external interface unit 196 is an interface between the system control CPU 178 and an external device for communicating with an external device such as the external computer 197. The print interface unit 194 is an interface between the system control CPU 178 and the printer 195 for outputting a photographed image to a printer 195 such as a small inkjet printer for printing. The wireless interface unit 198 is an interface between the system control CPU 178 and the network 199 for communicating with a network 199 such as the Internet. The switch input unit 179 includes a switch ST154, a switch MV155, and a plurality of switches for switching various modes. The flight control device 200 is a control device for controlling the propeller 162 to fly the imaging device 100 in order to perform shooting from the air. The system control CPU 178 includes a nonvolatile register, can store factory adjustment values, and operates as a storage device of the present invention.

  FIG. 3 is a block diagram illustrating a configuration example of the image sensor 184. As shown in FIG. 3, the image sensor 184 includes a pixel array 302, a vertical scanning circuit 307, readout circuits 308A and 308B, and timing control circuits 309A and 309B.

  In the pixel array 302, a plurality of pixels 303 are arranged in a matrix. Note that the actual number of pixels 303 belonging to the pixel array 302 is generally large, but here, in order to simplify the drawing, 16 pixels 303 arranged in a matrix of 4 rows × 4 columns. Only shows. Each of the plurality of pixels 303 includes a set of a pixel element 303A and a pixel element 303B. In FIG. 3, the upper half area of the pixel 303 is a pixel element 303A, and the lower half area of the pixel 303 is a pixel element 303B. The pixel element 303A and the pixel element 303B each generate a signal by photoelectric conversion.

  Each column of the pixel array 302 is provided with signal output lines 304A and 304B extending in the column direction. The signal output line 304A of each column is connected to the pixel element 303A belonging to the column. A signal from the pixel element 303A is output to the signal output line 304A. The signal output line 304B of each column is connected to the pixel element 303B belonging to the column. A signal from the pixel element 303B is output to the signal output line 304B. Each column of the pixel array 302 is also provided with a power supply line 305 and a ground line 306 extending in the column direction. The power supply line 305 and the ground line 306 in each column are connected to the pixels 303 belonging to the column. The power supply line 305 and the ground line 306 may be signal lines extending in the row direction.

  The vertical scanning circuit 307 is disposed adjacent to the pixel array 302 in the row direction. The vertical scanning circuit 307 controls a readout circuit in the pixel 303 via a control line (not shown) extending in the row direction with respect to the plurality of pixels 303 of the pixel array 302 in the row direction. A predetermined control signal is output. In the figure, reset pulses φRESn and transfer pulses φTXnA, TXnB are shown as control signals (n is an integer corresponding to a row number).

  The readout circuits 308A and 308B are arranged adjacent to the pixel array 302 in the column direction so as to sandwich the pixel array 302. The readout circuit 308A is connected to the signal output line 304A of each column. The readout circuit 308A sequentially activates the signal output lines 304A in each column, thereby sequentially reading out the signals from the signal output lines 304A in each column, and performs predetermined signal processing. Similarly, the readout circuit 308B is connected to the signal output line 304B of each column. The readout circuit 308B sequentially activates the signal output lines 304B in each column sequentially, thereby sequentially reading out the signals from the signal output lines 304B in each column and performing predetermined signal processing. Each of the readout circuits 308A and 308B can include a noise removal circuit, an amplification circuit, an analog-digital conversion circuit, a horizontal scanning circuit, and the like, and sequentially outputs signals that have undergone predetermined signal processing.

  The timing control circuit 309A is connected to the vertical scanning circuit 307 and the readout circuit 308A. The timing control circuit 309A outputs a control signal for controlling the driving timing of the vertical scanning circuit 307 and the readout circuit 308A. The timing control circuit 309B is connected to the vertical scanning circuit 307 and the readout circuit 308B. The timing control circuit 309B outputs a control signal for controlling the driving timing of the vertical scanning circuit 307 and the readout circuit 308B.

  FIG. 4 is a cross-sectional view showing the internal structure of the pixel 303 of the image sensor 184. Each pixel 303 includes two photodiodes 310A and 310B, a light guide 255, and a color filter 256, as shown in FIG. The photodiode 310A constitutes a part of the pixel element 303A, and the photodiode 310B constitutes a part of the pixel element 303B. The photodiodes 310A and 310B are provided in the silicon substrate 251. The light guide 255 is provided in an insulating layer 254 provided on the silicon substrate 251. The insulating layer 254 is made of, for example, silicon oxide, and the light guide 255 is made of a material having a higher refractive index than the insulating layer 254, for example, silicon nitride. A wiring layer 252 is provided on the insulating layer 254 between the light guides 255. On the light guide 255, a color filter 256 having a predetermined spectral transmittance characteristic is provided. FIG. 4 shows an example in which the color filters of two adjacent pixels 303 are configured by color filters 256 and 257 having different spectral transmittance characteristics.

  The light guide 255 has a property of confining light inside due to a difference in refractive index with the insulating layer 254. Thereby, the light incident through the color filter 256 can be guided to the photodiodes 310 </ b> A and 310 </ b> B by the light guide 255. The photodiodes 310A and 310B are arranged asymmetrically with respect to the light guide 255, and the light beam propagated through the light guide 255 is incident on the photodiode 310A with high efficiency and is incident on the photodiode 310B with low efficiency. Further, the light guide 255 is adjusted so that the incident angle characteristics of the light guide 255 are not biased with respect to the incident light beam that can be effectively photoelectrically converted by the photodiodes 310A and 310B.

  FIG. 5 is a graph showing the relationship between the angle of light rays incident on the pixel and the output from the photodiode. In FIG. 5, the horizontal axis represents the angle of light rays incident on the pixel, and the vertical axis represents the output from the photodiode. FIG. 5 shows an output characteristic 261 from the photodiode 310A and an output characteristic 262 from the photodiode 310B.

  As shown in FIG. 5, the characteristic 261 and the characteristic 262 both have a slightly symmetrical shape with a peak when the incident angle of the light beam is zero. Further, the peak intensity PB of the characteristic 262 is about 1/8 of the peak intensity PA of the characteristic 261. This means that the incident angle dependence of the photodiodes 310A and 310B is small, and the light receiving efficiency of the photodiode 310B is 1/8 of that of the photodiode 310A. That is, if the photodiode 310B is replaced with the ISO sensitivity setting value, the sensitivity is lower by three stages than the photodiode 310A.

  Next, the relationship between the photographing optical system 152 and the image sensor 184 will be described in more detail with reference to FIG. FIG. 6 is a diagram for explaining the relationship between the photographing optical system 152 and the image sensor 184. FIG. 6A is a view of the photographing optical system 152 as seen from the direction of the optical axis 180. FIG. 6B is a diagram showing in more detail the part from the photographing optical system 152 to the image sensor 184 in FIG.

  As shown in FIG. 6B, the imaging element 184 includes a pixel 276 located in the center of the imaging area and a pixel 277 located near the outer edge of the imaging area. In this case, the pixel 276 can receive a light flux from a region surrounded by the light rays 272 and 273. Further, the pixel 277 can receive a light beam from a region surrounded by the light beam 274 and the light beam 275. At this time, since the field lens 270 is disposed between the optical filter 183 and the photographing optical system 152, the light flux received by the pixel 276 and the light flux received by the pixel 277 in the vicinity of the photographing optical system 152 are illustrated in FIG. 6 (a) is overlapped as indicated by a region 271. As a result, the light beam emitted from the photographing optical system 152 can be received with high efficiency in any pixel.

  FIG. 7 is a schematic diagram illustrating a video signal output from the image sensor. Here, it is assumed that a color filter having a predetermined light transmittance characteristic is arranged in the pixel array 302 in the color filter array 281 shown in FIG. FIG. 7A schematically shows a pixel array 302 in which pixels 303 are arranged in a matrix of 6 rows × 8 columns, and colors of color filters arranged in each pixel. In the figure, R represents a red color filter, G1 and G2 represent green color filters, and B represents a blue color filter. The color filter array 281 shown in the figure is a color filter array called a so-called Bayer array, and color filters for each color are arranged for each row by repeating G1BG1B... RG2RG2... G1BG1B.

  From the pixel array 302 having such a color filter array 281, output data 282 and 283 shown in FIGS. 7B and 7C are obtained. In FIG. 7B, g1A and g2A represent outputs from the pixel element 303A of the pixel 303 in which the green color filter is arranged. bA represents the output from the pixel element 303A of the pixel 303 in which the blue color filter is arranged. rA represents the output from the pixel element 303A of the pixel 303 in which the red color filter is arranged. In FIG. 7C, g1B and g2B represent outputs from the pixel element 303B of the pixel 303 in which the green color filter is arranged. bB represents the output from the pixel element 303B of the pixel 303 in which the blue color filter is arranged. rB represents the output from the pixel element 303B of the pixel 303 in which the red color filter is arranged.

  As described with reference to FIG. 3, the image sensor 184 obtains two outputs from the readout circuits 308A and 308B, one of which is the output data 282 shown in FIG. This is the output data 283 shown in FIG. The output data 282 becomes a video signal “picture A” after predetermined signal processing. The output data 283 becomes a video signal “picture B” after predetermined signal processing. In the following description, the video signal based on the output data 282 is referred to as “picture A”, and the video signal based on the output data 283 is referred to as “picture B”. Strictly speaking, “picture A” and “picture B” are video signals after processing such as predetermined correction, but for convenience of explanation, “picture A” is also used for video signals before or during correction. ”And“ picture B ”.

  FIG. 8 is a circuit diagram illustrating a configuration example of the pixel 303. As described above, the pixel 303 includes the pixel element 303A and the pixel element 303B. The pixel element 303A includes a photodiode 310A, a transfer transistor 311A, a floating diffusion region 313A, a reset transistor 314A, and an amplification transistor 315A. The pixel element 303B includes a photodiode 310B, a transfer transistor 311B, a floating diffusion region 313B, a reset transistor 314B, and an amplification transistor 315B. Note that the photodiode 310A corresponds to the photodiode 310A illustrated in FIG. 4, and the photodiode 310B corresponds to the photodiode 310B illustrated in FIG.

  The anode of the photodiode 310A is connected to the ground line 306, and the cathode of the photodiode 301A is connected to the source of the transfer transistor 311A. The drain of the transfer transistor 311A is connected to the source of the reset transistor 314A and the gate of the amplification transistor 315A. A connection node of the drain of the transfer transistor 311A, the source of the reset transistor 314A, and the gate of the amplification transistor 315A constitutes the first floating diffusion region 313A. The drain of the reset transistor 314A and the drain of the amplification transistor 315A are connected to the power supply line 305. The source of the amplification transistor 315A constituting the pixel signal output unit 316A is connected to the signal output line 304A.

  Similarly, the anode of the photodiode 310B is connected to the ground line 306, and the cathode of the photodiode 310B is connected to the source of the transfer transistor 311B. The drain of the transfer transistor 311B is connected to the source of the reset transistor 314B and the gate of the amplification transistor 315B. A connection node of the drain of the transfer transistor 311B, the source of the reset transistor 314B, and the gate of the amplification transistor 315B constitutes the second floating diffusion region 313B. The drain of the reset transistor 314B and the drain of the amplification transistor 315B are connected to the power supply line 305. The source of the amplification transistor 315B constituting the pixel signal output unit 316B is connected to the signal output line 304B.

  The pixels 303 in each column are connected from the vertical scanning circuit 307 to a reset control line 319 and transfer control lines 320A and 320B arranged in the row direction. The reset control line 319 is connected to the gate of the reset transistor 314A and the gate of the reset transistor 314B. The transfer control line 320A is connected to the gate of the transfer transistor 311A via the contact portion 312A. The transfer control line 320B is connected to the gate of the transfer transistor 311B via the contact portion 312B. The reset control line 319 supplies the reset pulse φRESn output from the vertical scanning circuit 307 to the gate of the reset transistor 314A and the gate of the reset transistor 314B. The transfer control line 320A supplies a transfer pulse φTXnA output from the vertical scanning circuit 307 to the gate of the transfer transistor 311A. The transfer control line 320B supplies the transfer pulse φTXnB output from the vertical scanning circuit 307 to the gate of the transfer transistor 311B. Note that n added to the signs of the reset pulse φRESn, the transfer pulse φTXnA, and the transfer pulse φTXnB is an integer corresponding to the row number. In the drawing, n is represented by a code in which n is replaced with an integer corresponding to the row number.

  The photodiode 310A is a first photoelectric conversion unit that generates charges by photoelectric conversion, and the photodiode 310B is a second photoelectric conversion unit that generates charges by photoelectric conversion. The floating diffusion regions 313A and 313B are regions for accumulating charges. The transfer transistor 311A is for transferring the charge generated by the photodiode 310A to the floating diffusion region 313A. The transfer transistor 311B is for transferring the charge generated by the photodiode 310B to the floating diffusion region 313B.

  When the high-level transfer pulse φTXnA is output from the vertical scanning circuit 307, the transfer transistor 311A is turned on, and the photodiode 310A and the floating diffusion region 313A are connected. Similarly, when the high-level transfer pulse φTXnB is output from the vertical scanning circuit 307, the transfer transistor 311B is turned on, and the photodiode 310B and the floating diffusion region 313B are connected. When the high level reset pulse φRESn is output from the vertical scanning circuit 307, the reset transistors 314A and 314B are turned on, and the photodiodes 310A and 310B and the floating diffusion regions 313A and 313B are reset.

  When the low-level transfer pulse φTXnA is output from the vertical scanning circuit 307, the transfer transistor 311A is turned off, and the photodiode 310A starts accumulating signal charges generated by photoelectric conversion. Next, when the high-level transfer pulse φTXnA is output from the vertical scanning circuit 307, the transfer transistor 311A is turned on, and the signal charge of the photodiode 310A is transferred to the floating diffusion region 313A. Then, the amplification transistor 315A amplifies the voltage of the floating diffusion region 313A according to the amount of signal charge transferred from the photodiode 310A and outputs the amplified voltage to the signal output line 304A.

  Similarly, when the low-level transfer pulse φTXnB is output from the vertical scanning circuit 307, the transfer transistor 311B is turned off, and the photodiode 310B starts accumulating signal charges generated by photoelectric conversion. Next, when a high-level transfer pulse φTXnB is output from the vertical scanning circuit 307, the transfer transistor 311B is turned on, and the signal charge of the photodiode 310B is transferred to the floating diffusion region 313B. Then, the amplification transistor 315B amplifies the voltage of the floating diffusion region 313B corresponding to the amount of signal charge transferred from the photodiode 310B and outputs the amplified voltage to the signal output line 304B.

  9 and 10 are plan layout diagrams showing the main part of the pixel 303. FIG. 9 shows photodiodes 310A and 310B, transfer transistors 311A and 311B, and floating diffusion regions 313A and 313B among the components of the pixel 303. Other circuit elements including the reset transistors 314A and 314B and the amplification transistors 315A and 315B are represented as a read circuit unit 321 in the drawing, and detailed illustration is omitted. Further, the signal output lines 304A and 304B and the power supply line 305 arranged in the vertical direction of the pixel 303 are omitted, and the contact portions of the reset control line 319, the power supply line 305, and the ground line 306 are omitted. FIG. 10 shows the light guide 255 described in FIG. 4 in addition to the components shown in FIG. In the light guide 255, a hatched portion indicates a low refractive index region, and a white portion indicates a high refractive index region, that is, a light guide portion.

  9 and 10, the contact portion 312A is a contact portion that connects the transfer control line 320A and the gate of the transfer transistor 311A. The contact portion 312B is a contact portion that connects the transfer control line 320B and the gate of the transfer transistor 311B. The photodiodes 310 </ b> A and 310 </ b> B are photoelectric conversion units that perform photoelectric conversion. The photodiodes 310 </ b> A and 310 </ b> B are a first conductivity type (for example, P type) semiconductor region and a second conductivity type (for example, a PN junction with the first conductivity type semiconductor region). N-type) semiconductor region (N-type electron storage region). The second conductivity type semiconductor region of the photodiode 310A and the second conductivity type semiconductor region of the photodiode 310B are separated by a separation unit 322.

  The transfer transistors 311A and 311B, the contact portions 312A and 312B, and the transfer control lines 320A and 320B are arranged in line symmetry or substantially line symmetry with respect to the separation portion 322 between the photodiodes 310A and 310B, respectively. On the other hand, the light guide 255 is disposed at a position biased with respect to the separation portion 322 as shown in FIG. That is, the photodiode 310A occupies a large area of the bottom portion of the light guide 255, whereas the photodiode 310B is slightly hooked on the bottom portion of the light guide 255. As a result, the light receiving efficiency of the photodiode 310A is high, and the light receiving efficiency of the photodiode 310B is low.

  In the image sensor 184 according to the present embodiment, the ratio of the light receiving efficiencies of the photodiodes 310A and 310B is set to about 8: 1, that is, the sensitivity difference is set to about three stages. Then, while shooting two images with different storage time settings, the pixel element obtains the same level of signal charge, and both have a good S / N ratio and no noise, or This makes it possible to synthesize high HDR video. Details will be described later.

  FIG. 11 is a circuit diagram illustrating a configuration example of the readout circuits 308A and 308B of the image sensor 184. In FIG. 11, “A” is added to the end of the reference numerals of some components assuming the reading circuit 308A. In the readout circuit 308B, it should be understood that “B” is appended to the end of the reference numerals of the corresponding components.

  As shown in FIG. 11, the read circuit 308A includes a clamp capacitor C0, a feedback capacitor Cf, an operational amplifier 406, a reference voltage source 407, and a switch 423. One input terminal of the operational amplifier 406 is connected to the signal output line 304A via a clamp capacitor C0. A feedback capacitor Cf and a switch 423 are connected in parallel between the one input terminal and the output terminal of the operational amplifier 406. The other input terminal of the operational amplifier is connected to the reference voltage source 407. The reference voltage source 407 is for supplying the reference voltage Vref to the operational amplifier 406. The switch 423 is a switch controlled by the signal PC0R, and is turned on when the signal PC0R is at a high level, and short-circuits both ends of the feedback capacitor Cf.

  The readout circuit 308A also includes switches 414, 415, 418, and 419, a capacitor CTSA, a capacitor CTNA, horizontal output lines 424 and 425, and an output amplifier 421. The switches 414 and 415 are switches that control writing of pixel signals to the capacitors CTSA and CTNA. The switch 414 is a switch controlled by the signal PTSA, and is turned on when the signal PTSA is at a high level, and connects the output terminal of the operational amplifier 406 and the capacitor CTSA. The switch 415 is a switch controlled by the signal PTNA and is turned on when the signal PTNA is at a high level, and connects the output terminal of the operational amplifier 406 and the capacitor CTNA.

  The switches 418 and 419 are switches for controlling the output of the pixel signals held in the capacitors CTSA and CTNA to the output amplifier 421. The switches 418 and 419 are turned on in response to a control signal from the horizontal shift register. As a result, the signal written in the capacitor CTSA is output to the output amplifier 421 via the switch 418 and the horizontal output line 424. Further, the signal written in the capacitor CTNA is output to the output amplifier 421 via the switch 419 and the horizontal output line 425. The signal PC0R, the signal PTNA, and the signal PTSA are signals that are supplied from the timing generation unit 189 under the control of the system control CPU 178.

  The readout circuit 308B has a configuration equivalent to that of the readout circuit 308A. Further, the signal PTNB and the signal PTSB in the following description are signals supplied from the timing generation unit 189 under the control of the system control CPU 178, and play the same role as the signal PTNA and the signal PTSA in the read circuit 308A. ing.

  Next, reset, accumulation, and readout operations in the image sensor 184 will be sequentially described using the readout operation from the pixels 303 in the first row as an example with reference to the timing chart of FIG.

  First, at time t1, the vertical scanning circuit 307 transitions the transfer pulses φTX1A and TX1B output to the transfer control lines 320A and 320B from the low level to the high level. As a result, the transfer transistors 311A and 311B are turned on. At this time, a high level reset pulse φRES1 is output from the vertical scanning circuit 307 to the reset control line 319, and the reset transistors 314A and 314B are also in the ON state. As a result, the photodiodes 310A and 310B are connected to the power supply line 305 via the transfer transistors 311A and 311B and the reset transistors 314A and 314B, and are in a reset state. At this time, the floating diffusion regions 313A and 313B are also reset.

Next, at time t2, the vertical scanning circuit 307 changes the transfer pulse φTX1B from the high level to the low level. As a result, the transfer transistor 311B is turned off, and accumulation of signal charges by photoelectric conversion is started in the photodiode 310B.
Next, at time t3, the vertical scanning circuit 307 changes the transfer pulse φTX1A from the high level to the low level. As a result, the transfer transistor 311A is turned off, and the photodiode 310A starts to accumulate signal charges by photoelectric conversion.
Next, at time t4, the vertical scanning circuit 307 causes the reset pulse φRES1 to transition from the high level to the low level. Thereby, the reset transistors 314A and 314B are turned off, and the reset of the floating diffusion regions 313A and 313B is released.

  As a result, the potential of the floating diffusion region 313A is read to the signal output line 304A as the reset signal level via the amplification transistor 315A and input to the read circuit 308A. In addition, the potential of the floating diffusion region 313B is read as a reset signal level to the signal output line 304B via the amplification transistor 315B and input to the reading circuit 308B.

  At time t4, the high-level signal PC0R is output from the timing generation unit 189 to the read circuit 308A and the read circuit 308B, and the switch 423 is in the on state. Therefore, the pixel signal at the reset signal level is input from the pixel element 303A to the readout circuit 308A in a state where the operational amplifier 406 buffers the output of the reference voltage Vref. Although not shown in the drawing, the reset signal level pixel signal is similarly input to the readout circuit 308B from the pixel element 303B.

Next, at time t5, the signal PC0R output from the timing generation unit 189 to the read circuit 308A and the read circuit 308B is changed from the high level to the low level, and the switch 423 is turned off.
Next, at time t6, the signal PTNA output from the timing generation unit 189 to the reading circuit 308A is changed from the low level to the high level to turn on the switch 415, and the output of the operational amplifier 406 at that time is written into the capacitor CTNA. Similarly, the signal PTNB output from the timing generation unit 189 to the reading circuit 308B is changed from the low level to the high level to turn on the switch 415, and the output of the operational amplifier 406 at that time is written to the capacitor CTNB.

  Next, at time t7, the signal PTNA output from the timing generation unit 189 to the reading circuit 308A is changed from the high level to the low level, the switch 415 is turned off, and the writing to the capacitor CTNA is completed. Similarly, the signal PTNB output from the timing generation unit 189 to the reading circuit 308B is changed from the high level to the low level, the switch 415 is turned off, and the writing to the capacitor CTNB is completed.

  Next, at time t8, the vertical scanning circuit 307 changes the transfer pulses φTX1A and φTX1B from the low level to the high level, and turns on the transfer transistors 311A and 311B. As a result, the signal charge accumulated in the photodiode 310A is transferred to the floating diffusion region 313A, and the signal charge accumulated in the photodiode 310B is transferred to the floating diffusion region 313B.

  By simultaneously setting the transfer pulses φTX1A and φTX1B to the high level at time t8, the end timings of the accumulation times of the photodiodes 310A and 310B are aligned. Therefore, crosstalk correction, such as correcting “picture B” data using “picture A” data and correcting “picture A” data using “picture B” data, is a very simple calculation. Can be realized.

  Next, at time t9, the vertical scanning circuit 307 changes the transfer pulses φTX1A and φTX1B from the high level to the low level, and turns off the transfer transistors 311A and 311B. Thereby, the reading of the signal charge accumulated in the photodiode 310A to the floating diffusion region 313A and the reading of the signal charge accumulated in the photodiode 310B to the floating diffusion region 313B are completed.

As a result, the potential of the floating diffusion region 313A changed by the signal charge is read as an optical signal level to the signal output line 304A via the amplification transistor 315A, and input to the readout circuit 308A. Further, the potential of the floating diffusion region 313B changed by the signal charge is read as an optical signal level to the signal output line 304B via the amplification transistor 315B and is input to the reading circuit 308B.
In the readout circuit 308 </ b> A, the operational amplifier 406 outputs a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio between the clamp capacitor C <b> 0 and the feedback capacitor Cf. Similarly, in the readout circuit 308B, a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio of the clamp capacitor C0 and the feedback capacitor Cf is output from the operational amplifier 406.

  Next, at time t10, the signal PTSA output from the timing generation unit 189 to the reading circuit 308A is changed from the low level to the high level to turn on the switch 414, and the output of the operational amplifier 406 at that time is written into the capacitor CTSA. Similarly, the signal PTSB output from the timing generator 189 to the reading circuit 308B is changed from the low level to the high level to turn on the switch 414, and the output of the operational amplifier 406 at that time is written to the capacitor CTSB.

  Next, at time t11, the signal PTSA output from the timing generation unit 189 to the reading circuit 308A is changed from the high level to the low level, the switch 414 is turned off, and writing to the capacitor CTSA is completed. Similarly, the signal PTSB output from the timing generation unit 189 to the reading circuit 308B is changed from the high level to the low level to turn off the switch 414, and the writing to the capacitor CTSB is completed.

  Next, at time t12, the vertical scanning circuit 307 changes the reset pulse φRES1 from the low level to the high level to turn on the reset transistors 314A and 314B. As a result, the floating diffusion regions 313A and 313B are connected to the power supply line 305 via the reset transistors 314A and 314B to be in a reset state.

  FIG. 13 is a graph showing temporal changes in signal charges generated and accumulated by photoelectric conversion in the photodiodes 310A and 310B. In the figure, the horizontal axis of the graph represents time, and the vertical axis represents the amount of signal charge. On the time axis, time t1 to time t12 shown in FIG. 12 are displayed.

  At time t2, when the transfer pulse 3TX is turned off by setting the transfer pulse φTX1B to the low level and signal charge accumulation in the photodiode 310B is started, the amount of signal charge held by the photodiode 310B increases with time. A straight line 952 in FIG. 13 indicates a change in the amount of signal charge held by the photodiode 310B. The increase in the signal charge continues until the transfer pulse 3TXB is turned on at time t8 to turn on the transfer transistor 311B and the signal charge of the photodiode 310B is transferred to the floating diffusion region 313B.

  At time t3, the transfer pulse φTX1A is set to low level to turn off the transfer transistor 311A, and signal charge accumulation in the photodiode 310A is started. As a result, the amount of signal charge held by the photodiode 31A increases with time. A straight line 951 in FIG. 13 shows a change in the amount of signal charge held by the photodiode 310A. The increase in the signal charge continues until the transfer pulse 3TXA is turned on at time t8 to turn on the transfer transistor 311A and the signal charge of the photodiode 310A is transferred to the floating diffusion region 313A.

  At time t8, the signal charge amount LB held by the photodiode 310B and the signal charge amount LA held by the photodiode 310A are approximately the same by offsetting the difference in light receiving efficiency by the difference in accumulation time. .

  In a period TM1 in which both the transfer pulse φTX1B and the transfer pulse φTX1A are at a low level, crosstalk occurs between the photodiode 310A and the photodiode 310B. The period TM1 is a shorter value of the accumulation period of the photodiode 310A and the accumulation period of the photodiode 310B. Since the amount of crosstalk is substantially proportional to the amount of signal charge, a relatively large amount of crosstalk occurs in the second half period TM2 of the period TM1 in which the signal charge amount increases.

The amount of crosstalk CTAB from the photodiode 310A to the photodiode 310B is proportional to the area of the region 953 with a downward slanting shaded line. In addition, the amount of crosstalk CTBA from the photodiode 310B to the photodiode 310A is proportional to the area of a region 954 with a slanting line descending to the left. If these proportionality constants are defined as k and g, respectively, the crosstalk amounts CTAB and CTBA are
CTAB = k × (LA × TM1) / 2 (1)
CTBA = g × (LA + LBS) × TM1 / 2 (2)
It can be expressed as. Here, LBS is the signal charge amount of the photodiode 310B at time t3. Although not shown in this figure, if the time from time t2 to time t3 is sufficiently small with respect to the period TM1, it can be approximated as LB = LBS. Therefore, equation (2) is
CTBA = g × LB × TM1 (3)
And can be transformed.

  Therefore, from the equations (1) and (3), the crosstalk amount CTAB is the signal charge amount LA, and the shorter value (period TM1) of the accumulation period of the photodiode 310A and the accumulation time of the photodiode 310B. It turns out that it is a function of. It can also be seen that the crosstalk amount CTBA is a function of the signal charge amount LB and the shorter value (period TM1) of the accumulation period of the photodiode 310A and the accumulation time of the photodiode 310B.

  FIG. 14 is a potential diagram of the pixel 303 along the line AB in FIG. 14A is a potential diagram at time ta in FIG. 12, FIG. 14B is a potential diagram at time tb in FIG. 12, and FIG. 14C is a potential diagram at time tc in FIG.

  At time ta, as shown in FIG. 14A, the transfer transistors 311A and 311B are in an off state, and signal charges of signal accumulation levels 323A and 323B are accumulated in the photodiodes 310A and 310B, respectively. As described above, although the light receiving efficiency differs between the photodiode 310A and the photodiode 310B, the signal storage levels 323A and 323B are approximately the same by canceling out the difference in the light receiving efficiency due to the difference in the storage time. Since this state continues for a relatively long time, the phenomenon that the accumulated charge of the photodiode 310A leaks to the adjacent photodiode 310B and the phenomenon that the accumulated charge of the photodiode 310B leaks to the adjacent photodiode 310A occur at levels that cannot be ignored.

  At time tb, as shown in FIG. 14B, the transfer transistors 311A and 311B are in the on state, and the potential barriers of the transfer transistors 311A and 311B are low. Thereby, the signal charge accumulated in the photodiode 310A is transferred to the floating diffusion region 313A, and the signal charge accumulated in the photodiode 310B is transferred to the floating diffusion region 313B. At this time, the potential barrier of the separation unit 322 is also lowered, but the potential barriers of the transfer transistors 311A and 311B are sufficiently small. Therefore, a phenomenon in which the accumulated charges of the photodiodes 310A and 310B leak to the adjacent photodiodes 310B and 310A via the separation portion 322 hardly occurs at this timing.

  At time tc, as shown in FIG. 14C, the transfer transistors 311A and 311B are in the OFF state, and the potential returns to the state shown in FIG.

  FIG. 15 is a cross-sectional view showing the behavior of signal charges generated by light propagation and photoelectric conversion inside the image sensor 184. The light beam 451 incident on the pixel 303 first enters the color filter 256 and a predetermined wavelength component is absorbed here, passes through an interface deactivation film (not shown) corresponding to the uppermost portion of the insulating layer 254, and the light guide. Incident to 255. In the light guide 255, as described above with reference to FIG. 5, the light beam azimuth information, that is, pupil information disappears due to the wave behavior of the light. The light beam 451 proceeds to the silicon substrate 251 side while being confined inside the light guide 255 due to the difference in refractive index between the light guide 255 and the insulating layer 254, and reaches the bottom portion of the light guide 255. The bottom portion of the light guide 255 is adjacent to the silicon substrate 251, and the light beam emitted from the light guide 255 is incident on the silicon substrate 251. The photodiode 310A and the photodiode 310B provided adjacent to each other in the silicon substrate 251 are arranged so as to be largely eccentric with respect to the light guide 255. Therefore, most of the light beam 452 emitted from the light guide 255 is incident on the photodiode 310A, and the remaining part of the light beam 453 emitted from the light guide 255 is incident on the photodiode 310B. In the photodiodes 310A and 310B, incident photons are converted into signal charges.

  At this time, signal charges generated inside the silicon substrate 251 of the image sensor 184 leak into adjacent pixel elements by diffusion. For example, the signal charge 454 generated in the photodiode 310A leaks into the photodiode 310B by diffusion. Also. The signal charge 455 generated in the photodiode 310B leaks into the photodiode 310A by diffusion. This phenomenon has an adverse effect on the video and appears as blurring of the image.

  FIG. 16 is a timing chart for explaining an imaging sequence in the imaging apparatus according to the present embodiment. The “time code” at the top of the drawing indicates the time since the power is turned on, and “00: 00: 00: 00” indicates “hour: minute: second: frame”.

Time t31 is the power-on time of the imaging apparatus 100.
At time t32, the switch MV155, which is a moving image shooting button, is operated by the user to be turned on, and in response to this, the imaging of “picture B” and the imaging of “picture A” are started. In response to the operation of the switch MV155, which is a button for moving image shooting, the video data of “picture B” is written to the recording medium 193 through predetermined signal processing.

  The reason why the “picture A” is picked up simultaneously with the “picture B” is because the crosstalk correction described later is always effective. If the transfer pulse φTX1A shown in FIG. 13 does not go to a low level, the transfer transistor 311A is in an on state, so that the signal charge generated in the photodiode 310A is not accumulated. However, if only the period during which the switch ST154 is operated is subject to crosstalk correction, a subtle change in luminance or hue in “picture B” recorded at the operation timing of the switch ST154 due to the influence of the crosstalk correction error. Will occur.

  In the period from time t33 to time t34 and in the period from time t35 to time t36, the switch ST154 used for taking a still image is operated. In response to this, the video data of “picture A” is also written to the recording medium 193 through predetermined signal processing during these periods. The “picture A” video data is written to the recording medium 193 not only during the period from time t33 to time t34 and from time t35 to time t36, but also during the same period as the video data of “picture B”. May be.

  For both “picture A” and “picture B”, each video data recorded on the recording medium 193 is a moving image with the same frame rate, for example, 60 fps, and an NTSC time code is added. . The value of the time code added to each frame of the moving image data is, for example, as shown in FIG.

  FIG. 18 is a diagram illustrating an example of a file structure of video data of “picture A” and “picture B”. Here, an example of the MP4 file is shown as the format of the video data, but the format of the video data is not limited to this. The MP4 file format is standardized by ISO / IEC 144961 / AMD6. All information is stored in a structure called a box, and is composed of multiplexed video and audio bitstreams (media data) and management information (metadata) for these media data. Each Box is an identifier of 4 characters and represents each Box type. The file type Box 501 (ftyp) is a box at the head of the file for identifying the file. The media data box 502 (mdat) stores a multiplexed video and audio bit stream. The movie box 503 (moov) stores management information for reproducing the bit stream stored in the media data box 502. The skip box 504 (skip) is a box for skipping and skipping data stored in the skip box 504 during reproduction.

  In the skip box 504, the clip name 508 of the clip including the video data file and the UMID (Unique Material Identifier) 509 (CLIP-UMID) of the clip attached to the material are stored. In the skip box 504, the time code value (time code start value) 510 of the clip start frame and the serial number 511 of the recording medium on which the material file is recorded are stored. In this figure, the skip box 504 includes free space 505, user data 506, and metadata 507. Since special data such as the UMID of the material file and the serial number of the recording medium are stored in the skip box, they are not affected when played back by a general-purpose viewer.

  The same CLIP-UMID is set in the MP4 files of “picture A” and “picture B”. As a result, the same CLIP-UMID file can be searched from one material file using CLIP-UMID, and mechanical association can be performed without manual confirmation.

  FIG. 19 is a diagram for explaining a screen for setting the imaging conditions of “picture A” and “picture B”. Suppose that the shooting mode selection lever 156 is rotated 90 degrees clockwise from the position of FIG. 1B, for example, to enter a dual video mode in which two videos can be shot simultaneously. The display unit 153 displays the Bv value 521 corresponding to the luminance of the subject at that time, the F number 522, the ISO sensitivities 523 and 524 of “picture A” and “picture B”, and the shutter speeds 525 and 526, respectively. . In addition, currently set picture modes 527 and 528 are displayed for each of “picture A” and “picture B”. In the picture mode, the up / down switches 158 and 159 and the dial 160 can be used to select one that meets the purpose of shooting from a plurality of options.

  As described above, the difference in light receiving efficiency between the photodiode 310A and the photodiode 310B is set to three stages. Therefore, there is a three-stage difference in the ISO sensitivity range between “picture A” and “picture B”. As shown in FIG. 20, “picture A” is ISO 100 to ISO 102400, and “picture B” is ISO 12 to ISO 12800.

  FIG. 21 is a program AE (Automatic Exposure) diagram in the dual video mode. The horizontal axis shows the Tv value and the corresponding shutter speed, and the vertical axis shows the Av value and the corresponding aperture value. Further, the oblique direction is an equal Bv line. The relationship between the Bv value of “picture A” and the ISO sensitivity is represented in the gain notation area 556, and the relationship between the Bv value of “picture B” and the ISO sensitivity is represented in the gain notation area 557. In FIG. 21, each Bv value is represented by a numerical value surrounded by a square to distinguish it from other parameters.

  How the shutter speed, aperture value, and ISO sensitivity change as the brightness changes from high to low will be described with reference to FIG.

  First, in the case of Bv13, the ISO sensitivity is set to ISO100 in “picture A”. The equal Bv line of “picture A” intersects the program diagram 558 of “picture A” at a point 551, and the shutter speed 1/4000 and the aperture value F11 are determined from the point 551. On the other hand, in “picture B”, the ISO sensitivity is set to ISO12. The equal Bv line of “picture B” intersects the program diagram 559 of “picture B” at a point 552, and the shutter speed 1/500 and the aperture value F11 are determined from the point 552.

  In the case of Bv10, in “picture A”, the ISO sensitivity is increased by one step and set to ISO200. An equal BV line of “picture A” intersects with the program diagram 558 of “picture A” at a point 553, and a shutter speed 1/1000 and an aperture value F11 are determined from the point 553. On the other hand, in “picture B”, the ISO sensitivity is set to ISO12. The equal Bv line of “picture B” intersects the program diagram 559 of “picture B” at a point 560, and the shutter speed 1/60 and the aperture value F11 are determined from the point 560.

  In the case of Bv6, the ISO sensitivity is set to ISO200 in “picture A”. The equal Bv line of “picture A” intersects with the program diagram 558 of “picture A” at a point 554, and the shutter speed is 1/1000 and the aperture value F2.8 is determined from the point 554. On the other hand, in “picture B”, the ISO sensitivity is set to ISO12. The equal Bv line of “picture B” intersects the program diagram 559 of “picture B” at a point 555, and the shutter speed 1/60 and the aperture value F2.8 are determined from the point 555.

  In the case of Bv5, in “picture A”, the ISO sensitivity is increased by one step and set to ISO400. The equal Bv line of “picture A” intersects with the program diagram 558 of “picture A” at a point 554, and the shutter speed is 1/1000 and the aperture value F2.8 is determined from the point 554. On the other hand, in “picture B”, the ISO sensitivity is set to ISO25. The equal Bv line of “picture B” intersects the program diagram 559 of “picture B” at a point 555, and the shutter speed 1/60 and the aperture value F2.8 are determined from the point 555.

  Thereafter, as the brightness decreases, both the “picture A” and “picture B” increase the gain without changing the shutter speed and the aperture value, and the ISO sensitivity increases.

  By performing the exposure operation shown in the program AE diagram, “picture A” maintains a shutter speed of 1/1000 or more in the entire luminance range described, and “picture B” is 1/60 in many luminance ranges. The shutter speed is maintained. As a result, while “picture A” obtains a stop motion effect, “picture B” makes it possible to obtain a high-quality moving image in which so-called jerkiness is suppressed, such as frame advance.

  In general, when the shutter speed at the time of moving image reproduction is high, so-called jerkiness such as frame advance appears at the time of reproduction, and the smoothness of the image is lost. In order to obtain such a smooth image with reduced jerkiness, it is necessary to set an accumulation time close to one frame period in a series of photographing. That is, if the frame rate is 30 fps, a relatively long accumulation time such as 1/30 seconds or 1/60 seconds is appropriate. This setting is particularly important in situations where the camera posture is unstable, such as aerial photography.

  On the other hand, in still images, it is required to shoot a video with a so-called stop motion effect that suppresses blur and stops a moment. For this reason, it is necessary to set a short accumulation time of, for example, about 1/1000 second. In addition, since one frame period is short in a high frame rate moving image, for example, if the frame rate is 120 fps, a short accumulation time such as 1/125 seconds or 1/250 seconds is inevitably set.

  Shooting two images of a moving image and a still image or a normal frame rate moving image and a high frame rate moving image simultaneously through a single shooting lens means that the aperture used for the shooting is the same. . At this time, while the two images are shot with different storage time settings, the image sensor obtains the same level of signal charge, and both of them have a good S / N ratio and no noise. Is desirable.

  FIG. 22 is a diagram illustrating a difference in shutter speed between “picture A” and “picture B” on the imaging sequence. In the figure, the horizontal axis indicates time, and the V synchronization signal 481, “picture A” accumulation periods 482 and 483, and “picture B” accumulation periods 484 and 485 are shown. n is a frame number.

  The accumulation period 482 is the accumulation period of the “picture A” screen top line, and the accumulation period 483 is the accumulation period of the “picture A” screen bottom line. In order to perform an exposure operation with the rolling electronic shutter function, the image sensor 184 sequentially starts accumulation at a predetermined time interval from the line at the upper end of the screen toward the line at the lower end of the screen, and ends the accumulation at the time interval. When the accumulation ends, signal charges are sequentially read from the image sensor 184 and input to the analog front end 185. The accumulation period 482 is from time t53 to time t54, and the accumulation period 483 is from time t55 to time t56.

  The accumulation period 484 is the accumulation period of the screen top line of “picture B”, and the accumulation period 485 is the accumulation period of the screen bottom line of “picture B”. Similarly to “picture A”, accumulation in “picture B” starts from a line at the upper end of the screen toward a line at the lower end of the screen at a predetermined time interval, and the accumulation ends sequentially at the time interval. When the accumulation is completed, signal charges are sequentially read from the image sensor 184 and input to the analog front end 186. The accumulation period 484 is from time t51 to time t54, and the accumulation period 485 is from time t52 to time t56.

  Two pictures of “picture A” and “picture B” are photographed with different accumulation time settings, but the gain of the “picture A” is not increased, but the signal charge of the same level is obtained in the image sensor 184. It has gained. Therefore, both “picture A” and “picture B” are images having a good S / N ratio and no noise.

  FIG. 23 is a diagram illustrating a state of the display unit 153 during live view display after the imaging apparatus 100 is powered on. On the display unit 153, the sports scene of the person 163 captured through the photographing optical system 152 is displayed. Also, since the shooting mode selection lever 156 is at a position rotated 90 degrees clockwise from the state of FIG. 1B, the shutter speeds 491 and 492 of “picture A” and “picture B” in the dual video mode and An F number 493 is displayed.

  FIG. 24 shows one frame of the video obtained by operating the switch ST154 and the switch MV155. FIG. 24A is an image of “picture A” taken at a shutter speed of 1/1000 and an aperture value of F4.0. FIG. 24B is an image of “picture B” taken at a shutter speed of 1/60 and an aperture value of F4.0. The image shown in FIG. 24B is blurred without stopping the movement of the subject because the shutter speed is slow. However, when this is played back as a moving image having a frame rate of about 60 fps, the blur works in a rather good direction, resulting in a smooth high-quality image in which jerkiness is suppressed. On the other hand, the image shown in FIG. 24A has a high shutter speed, and a stop motion effect should appear. However, as described above with reference to FIG. 15, the signal charge generated inside the silicon substrate leaks into adjacent pixel elements due to diffusion, so that the image shown in FIG. It has become a video. This crosstalk phenomenon also occurs in the image shown in FIG. 24B, but is hardly noticeable because the image is originally blurred.

  Therefore, in order to obtain the original stop motion effect due to the fast shutter speed, the imaging apparatus according to the present embodiment performs crosstalk correction on the video signal output from the imaging element 184. The crosstalk correction may be performed by appropriately subtracting each other in consideration of the mutual influences of “picture A” and “picture B”.

  FIG. 25 is an image of “picture A” after crosstalk correction is performed on an image of “picture A” (FIG. 24A) taken at a shutter speed of 1/1000 and an aperture value of F4.0. is there. In the image of FIG. 24A, the signal charge generated inside the silicon substrate leaks into adjacent pixel elements due to diffusion, and the image shown in FIG. 24B is blurred as if they were added together. . On the other hand, in the video of FIG. 25, the stop motion effect due to the original fast shutter speed appears. On the display unit 153 of the digital still motion camera, when the play button 161 is operated, it is desirable that both “picture A” 496 and “picture B” 497 can be displayed side by side as shown in FIG. 26, for example. In this way, the level of the stop motion effect can be confirmed by comparing the images. In this process, the video data may be supplied to a system or apparatus via a network, and a computer of the system or apparatus may read and execute the program.

FIG. 27 is a diagram for explaining an application example of “picture A” and “picture B” stored in the storage in a tablet terminal, a personal computer, a television monitor, and the like.
It is assumed that the data files of “picture A” and “picture B” are stored in a storage on the network. In the figure, a frame group 581 is a “picture A” frame group stored in an MP4 file, and a frame group 571 is a “picture B” frame group stored in another MP4 file. These MP4 files are associated with the same CLIP-UMID set at the time of shooting.

  First, when playback of a moving image is started, frames are sequentially played back at a frame rate determined from the first frame 572 of the “picture B” frame group 571. Since “picture B” is shot at a setting that prevents the shutter speed from becoming too fast (in this example, 1/60 seconds), the reproduced video is high-quality with reduced jerkiness.

  Here, it is assumed that the user gives a playback mode switching instruction during the presentation of the “picture B” video. For example, when the user performs a pause operation at the time when playback has progressed to frame 573, a frame 582 of the same time code is automatically searched from the data file of “picture A” corresponding to “picture B”, Is displayed. “Picture A” is a powerful image that is captured at a high shutter speed (in this example, 1/1000 second) at which a stop motion effect is easily obtained, and captures a moment in a sports scene. Two pictures of “picture A” and “picture B” are taken with different accumulation time settings, but the gain of “picture A” is not increased, but a similar signal charge is obtained at the image sensor 184. Yes. For this reason, both “picture A” and “picture B” are images having a good S / N ratio and no noise.

  When printing is instructed, the data of the “picture A” frame 582 is output to the printer 195 via the print interface unit 194. Therefore, the printed matter is also powerful with a stop motion effect reflecting “picture A”.

  When the user cancels the pause, the frame automatically returns to the “picture B” frame group 571 and the reproduction is resumed from the frame 574. At this time, the reproduced video is of high quality with reduced jerkiness.

  Next, an operation for creating an HDR video (high dynamic range video) by combining “picture A” and “picture B” will be described. As a technique for making movies and home television images more realistic, there is a moving picture HDR technology. This expands the luminance reproduction range of the display screen, and provides a more realistic sensation than before, mainly by momentary or partial luminance increase. In the HDR technology, two pixel groups having different sensitivities are provided in an image pickup element in an image pickup apparatus, and the dynamic range is expanded by synthesizing outputs from these pixel groups. In this technique as well, it is desirable that intermediate video data having a good S / N ratio and no noise can be created from either of the two pixel groups, and finally high quality HDR video can be synthesized.

  “Picture A” is photographed using a pixel element 303A having a high light receiving efficiency, and “picture B” is photographed using a pixel element 303B having a low light receiving efficiency. For this reason, even if the same accumulation time is set for both, “picture B” is an image that is inferior in sensitivity to “picture A” by three stages. Therefore, in the case of “picture A”, when the low luminance area is at a sufficiently reproducible level, the high luminance area often skips white, but in “picture B”, the high luminance area does not white out. Luminance information can be left. However, since the low luminance part of “picture B” has a very low signal level and blackout occurs, it cannot be used in many cases.

  In order to acquire HDR video, first, video acquired from pixel elements with low sensitivity (hereinafter referred to as “under video”) and video acquired from pixel elements with high sensitivity (hereinafter referred to as “over video”). Adjust the gain appropriately. Then, the area darker than the luminance threshold of the under video is replaced with the over video, and the area brighter than the luminance threshold is synthesized as it is. In the present embodiment, “picture B” may be used as the under video and “picture A” may be used as the over video. However, it is desirable that the brightness intermediate region near the luminance threshold is adjusted as appropriate so that the image looks more natural for the user, such as gradually changing the composition ratio of the images.

  How much lower the sensitivity of “picture B” is set with respect to “picture A” depends on the range of the luminance distribution of the subject. When the luminance range is wide and the signal level of “picture B” as an under picture is set lower than three stages, the accumulation time of “picture B” is made shorter than the accumulation time of “picture A”. On the contrary, in order to make the signal level of “picture B” as an under picture narrower than the sensitivity of 3 steps lower than the sensitivity of “picture B”, the accumulation time of “picture B” should be longer than that of “picture A”. That's fine. When the subject is a landscape or the like, it is often appropriate that “picture B” has a sensitivity that is about three steps lower than “picture A”. Therefore, the “picture A” and “picture” are offset by canceling the difference in sensitivity. The accumulation time of “B” is about the same. Since the accumulation times of “picture A” and “picture B” are approximately the same, even when the subject is moving, blurring and blurring of the subject image at the corresponding pixel positions of “picture A” and “picture B” The state will be well aligned. As a result, a higher quality HDR video can be obtained.

  FIG. 28 is a diagram for explaining the accumulation operation of “picture A” and “picture B” on the imaging sequence. In the figure, the horizontal axis indicates time, the V synchronization signal 781, the accumulation period of “picture A” and “picture B”, and the frame number n are shown. The accumulation period 782 is the accumulation period of the “picture A” screen top line, and the accumulation period 783 is the accumulation period of the “picture A” screen bottom line. Also, the accumulation period 784 is the accumulation period of the “picture B” screen upper end line, and the accumulation period 785 is the accumulation period of the “picture B” screen lower end line.

  Since the image sensor 184 performs an exposure operation with a rolling electronic shutter function, in “picture A”, accumulation starts at a predetermined time interval from the line at the upper end of the screen toward the line at the lower end of the screen, and sequentially at the time intervals. Accumulation ends. When the accumulation ends, signal charges are sequentially read from the image sensor 184 and input to the analog front end 185. The accumulation period 782 is from time t72 to time t73, and the accumulation period 783 is from time t75 to time t76.

  Similarly to “picture A”, in “picture B”, accumulation starts at a predetermined time interval from a line at the upper end of the screen toward a line at the lower end of the screen, and the accumulation ends sequentially at the time interval. When the accumulation is completed, signal charges are sequentially read from the image sensor 184 and input to the analog front end 186. The accumulation period 784 is from time t71 to time t73, and the accumulation period 785 is from time t74 to time t76.

  FIG. 29 is a graph showing the relationship between the signal level in “picture A” and “picture B”, and the combination ratio of “picture A” and “picture B” in HDR video, with the amount of received light. In the upper and lower graphs of FIG. 29, the horizontal axis indicates the amount of received light. In these two graphs, the horizontal scale is the same. The vertical axis of the upper graph indicates the signal levels in “picture A” and “picture B”. The vertical axis of the lower graph indicates the composition ratio of “picture A” and “picture B” in the HDR video. On the vertical axis of the upper graph, in addition to the case where the signal level is zero and the saturation level, a signal level (black crushing determination level) which is determined to be black crushing in the present embodiment, and saturation in this embodiment. The determination level (saturation determination level) is shown. These signal levels are recorded in the storage means in advance.

  Here, the received light amount when the signal level of “picture A” is the black crushing determination level is L1. The amount of received light when the signal level of “picture B” is the black crushing determination level is L2. The received light amount when the signal level of “picture A” is the saturation determination level is L3. The amount of received light when the signal level of “picture B” is the saturation determination level is L4. The amount of received light is determined by the brightness of the subject and the exposure time. In the present embodiment, the composition ratio of “picture A” and “picture B” in the HDR video is changed for each region divided by the received light amounts L1, L2, L3, and L4.

  A region 6121 in which the amount of received light is less than L1 is a region in which both “picture A” and “picture B” are blacked out. In the region 6121, the “picture A” signal synthesis ratio, which is relatively excellent in the S / N ratio, is set to 1, and the “picture B” signal synthesis ratio is set to 0.

  A region 6122 where the amount of received light is L1 to L2 is a region where “picture B” is black, although the signal level of “picture A” is appropriate. In the region 6122, the synthesis ratio of the “picture A” signal with the appropriate signal level is 1 and the synthesis ratio of the “picture B” signal is 0.

  A region 6123 in which the amount of received light is L2 to L3 is a region where both the signal levels of “picture A” and “picture B” are appropriate. In the region 6123, the synthesis ratio of the “picture A” signal is gradually decreased from 1 to 0, and the synthesis ratio of the “picture B” signal is gradually increased from 0 to 1. In FIG. 29, the case where the signal levels of “picture A” and “picture B” change linearly with respect to the amount of received light is shown, but the change in the composition ratio is not necessarily linear. That is, the composition ratio of “picture A” and “picture B” may be changed so that the total value thereof becomes 1, and may be changed according to another function that is not a linear function.

  A region 6124 where the amount of received light is L3 to L4 is a region where the signal level of “picture B” is appropriate, but the signal of “picture A” is saturated. In a region 6124, the synthesis ratio of the “picture B” signal having the appropriate signal level is 1 and the synthesis ratio of the “picture A” signal is 0.

  A region 6125 in which the amount of received light exceeds L4 is a region where both the “picture A” and “picture B” signals are saturated. In the region 6125, the synthesis ratio of the “picture B” signal, which may have an appropriate signal remaining just before saturation, is 1 and the synthesis ratio of the “picture A” signal is 0.

  The HDR video composition processing described above is executed by, for example, the system control CPU 178 that has received “picture A” and “picture B” from the recording medium 193. That is, the system control CPU 178 functions as a high dynamic range video composition unit.

  FIG. 30 is an example of “picture A” and “picture B” for HDR synthesis, and is an image of a landscape at dawn including the sun. FIG. 35A shows “picture A”, and FIG. 35B shows “picture B”. The imaging conditions of “picture A” and “picture B” both have a shutter speed of 1/60 and an aperture value of F2.8. However, since the actual shutter speed at the time of imaging is controlled more finely than the display, “picture A” and “picture B” are not necessarily exactly the same.

  FIG. 30A shows one frame of the “picture A” image photographed using a pixel element with high light receiving efficiency, and the details of the building 791 are hidden but reproduced. On the other hand, since the sun 792 has very high brightness, the sun 792 is also whitened including its peripheral part. On the other hand, FIG. 30B is a frame at the same timing as FIG. 30A of the “picture B” image captured using the pixel element having low light receiving efficiency, and the building 793 is hidden. The details are lost. On the other hand, the sun 794 has a very high luminance, but since it is taken with a pixel element having a low light receiving efficiency, it does not fly out, and luminance information and color information are not lost.

  FIG. 31 is a diagram illustrating a state in which the result of synthesizing the HDR video is displayed on the display unit 153 of the imaging apparatus. An HDR image in which the details of the building 795 and the gradation of the sun 796 and its surroundings are well reproduced can be obtained.

  Here, a case where a defective pixel exists in the image sensor 184 is considered. The defective pixel assumed here is a pixel 303 in which at least one of the two pixel elements 303A and 303B cannot output an appropriate signal corresponding to the amount of received light. Degradation of image quality due to the presence of defective pixels can be suppressed, for example, by performing the following correction process.

  FIG. 32 is a plan view schematically showing a method for correcting a pixel value of a defective pixel when the imaging element 184 includes a defective pixel. FIG. 32 shows 36 pixels 303 arranged in a matrix of 6 rows × 6 columns in the pixel array 302. A thin line frame 6102 represents the boundary of the pixel 303. This frame 6102 also represents the boundary of the light guide 255. A pixel block of 2 rows × 2 columns surrounded by a thick line frame 6101 represents a repeating unit of the color filter. In the frame 6101, a pixel (R) in which a filter that selectively transmits red light is disposed, a pixel (G1, G2) in which a filter that selectively transmits green light is disposed, and a blue color Pixels (B) each having a filter that selectively transmits light are disposed. A so-called Bayer array color filter is configured by repeatedly arranging the pixel blocks. Each pixel 303 includes a pixel element 6104 including a high-sensitivity photoelectric conversion unit corresponding to “picture A” and a pixel element 6103 including a low-sensitivity photoelectric conversion unit corresponding to “picture B”. In the drawing, the area of the pixel element 6104 is drawn larger than the area of the pixel element 6103 in order to visually express the difference in sensitivity between the pixel elements 6103 and 6104.

  Here, it is assumed that some of the pixel elements 6103 are defective. In FIG. 32, a defective pixel element (defective pixel element) 6103 is represented as a pixel element 6105. Here, a case where the low-sensitivity pixel element 6103 has a defect will be described as an example, but the same correction processing can be performed when the high-sensitivity pixel element 6104 has a defect.

  In FIG. 32A, an example in which the signal of the defective pixel element 6105 is corrected using the signal from the other pixel element 6106 of the same pixel 303 is schematically indicated by an arrow 6107. In other words, the other pixel element of the same pixel 303 is another pixel element that shares the same light guide 255 as the defective pixel element.

  FIG. 32B shows an example in which a signal of a defective pixel element 6105 is corrected using signals from pixel elements 6110, 6111, 6112, and 6113 having the same sensitivity and color filter of adjacent pixel blocks. , Schematically indicated by arrows. That is, the correction using signals from the pixel elements 6110, 6111, 6112, and 6113 is schematically indicated by arrows 6110a, 6111a, 6112a, and 6113a, respectively.

  When considering defect correction using a pixel element in which a color filter of the same color is arranged, as a pixel element used for correcting a signal of the pixel element 6105, as shown in FIG. Other pixel elements 6106 of the same pixel 303 are considered as candidates. Further, as shown in FIG. 32B, pixel elements 6110, 6111, 6112, and 6113 having the corresponding sensitivity and color filter of adjacent pixel blocks are considered as candidates. In the present embodiment, the correction method is switched according to the user setting in consideration of the advantages and disadvantages of using each.

First, as an example of calculating the signal level of the defective pixel element 6105 by a simple linear sum of the signal levels of the other pixel elements 6105, 6106, 6110, 6111, 6112, 6113, consider the following equation.
S6105 = g1 × S6106 + g2 × S6110 + g3 × S6111
+ G4 × S6112 + g5 × S6113 (4)
Here, S6105, S6106, S6110, S6111, S6112, and S6113 are the signal levels of the pixel elements 6105, 6106, 6110, 6111, 6112, and 6113, respectively. g1, g2, g3, g4, and g5 are gains (combining ratios) when signals from the pixel elements 6106, 6110, 6111, 6112, and 6113 are combined.

  The pixel element 6106 is a pixel element having a sensitivity different from that of the pixel elements 6110, 6111, 6112, and 6113, and the gain g1 thereof is at a level different from that of the other gains g2, g3, g4, and g5.

  For example, as described with reference to FIG. 5, when the pixel element 303 </ b> B that generates the “picture B” signal has a sensitivity that is three steps lower than the pixel element 303 </ b> A that generates the “picture A” signal, the gain g <b> 1 is set. Is set in consideration of the aforementioned sensitivity difference. That is, when the signal levels S6106, S6110, S6111, S6112, and S6113 are simply averaged, the gains g2, g3, g4, and g5 are set to 0.2, and the gain g1 is 1/8 of 0.2 (= 0. 025). That is, the signal levels S6106, S6110, S6111, S6112, and S6113 are added together at a predetermined ratio represented by gains g1, g2, g3, g4, and g5 to calculate a signal level S6105. If the defective pixel element is a high-sensitivity pixel element, the pixel element may be increased by a factor of eight. In addition, the information that the sensitivity mentioned in the above description is three steps lower is stored in the storage means in advance.

  As another method, there is a method of determining a gain using a ratio of signals from adjacent pixel elements. For example, the signal ratios between the signals from the pixel elements 6110, 6111, 6112, and 6113 and the signals from the pixel elements that share the light guide 255 (pixel elements with high sensitivity) are examined. The signal level from the pixel element 6105 can be calculated by estimating the signal ratio thus obtained as the signal ratio between the signal from the pixel element 6105 and the signal from the pixel element 6106.

  When equation (4) is used, it does not matter if the signals from the pixel elements are all at appropriate levels, but it does not matter if the signals from some of the pixel elements are at levels of saturation, blackout, or close to it. become.

  As an example, let us consider a case where a user performs so-called HDR shooting in order to obtain an image giving priority to gradation. Further, in order to facilitate the explanation, a case where a pixel element that outputs a “picture A” signal is saturated, that is, a case where the amount of generated signal charge is larger than a predetermined value (saturation determination level) is considered. That is, the amount of received light corresponds to the region 6124 in FIG. At this time, since the pixel element that outputs the “picture B” signal is not saturated, an appropriate signal can be obtained when there is no defective pixel element. However, if there is a defective pixel element 6105 and the equation (4) is used as it is for the correction process, the signal is estimated to an inappropriate value by using the saturated signal level S6106.

  Therefore, in the mode in which priority is given to gradation, the mixing ratio of signals from the pixel elements 6110, 6111, 6112, and 6113 having the same sensitivity as that of the defective pixel element 6105 is increased. Specifically, in the equation (4), the gain g1 is decreased and the gains g2, g3, g4, and g5 are increased. Typically, as shown in FIG. 32B, pixel elements 6110, 6111, 6112, 6113 having the corresponding sensitivity and color filter of the pixel block adjacent to the pixel block including the defective pixel element 6105 are included. Prioritize the use of signals. For example, if the gain g1 is 0, only signals from pixel elements 6110, 6111, 6112, and 6113 having the corresponding sensitivity and color filter of the pixel block adjacent to the pixel block including the defective pixel element 6105 are obtained. Will be used to correct.

  In this case, the signals from the pixel elements 6110, 6111, 6112, and 6113 may be used as a simple average, or the weight may be changed using the correlation according to the direction. For example, | S6110-S6112 | can be compared with | S6111-S6113 |, and the direction with a smaller numerical value can be preferentially weighted. This is because of a change in signal level in the horizontal direction (or horizontal direction) (| S6110-S6112 |) and a change in signal level in the vertical direction (or vertical direction) (| S6111−S6113 |). This is equivalent to preferentially using information in a direction with little change.

The same processing can be applied when the pixel element that outputs the “picture A” signal is blacked out, that is, when the amount of the generated signal charge is smaller than a predetermined value (blacked out determination level). . That is, the mixing ratio of signals from the pixel elements 6110, 6111, 6112, and 6113 having the same sensitivity as that of the defective pixel element 6105 may be increased.
It is the same.
As another example, consider a case where the user gives priority to resolution over gradation. In such a case, if correction is performed using the signals of adjacent pixel blocks as described above, the effect of spatially passing through a low-pass filter may appear, resulting in a decrease in resolution. Not necessarily preferred.

  Therefore, when the mode is not a mode in which priority is given to gradation, the synthesis ratio of signals from other pixel elements belonging to the same pixel as the defective pixel element is increased. That is, the ratio of the signal from the pixel element corresponding to “picture A” is increased.

  Specifically, in the equation (4), the gains g2, g3, g4, and g5 are decreased and the gain g1 is increased. Typically, as shown in FIG. 32A, signals from other pixel elements 6106 belonging to the same pixel as the defective pixel element 6105 are used preferentially. For example, if the gains g2, g3, g4, and g5 are set to 0, correction is performed using only signals from other pixel elements 6106 belonging to the same pixel as the defective pixel element 6105.

  Here, the state of the subject and defect correction will be considered. FIG. 33 is an example of an image shot with a person positioned near the center. Considering such an image, the person 6131 as the main subject includes a signal with high contrast, but the background 6132 (assuming the sky here) often does not include a signal with high contrast. In particular, when the F number of the photographing optical system is small and the blur is noticeable, the contrast of the background 6132 is low. When the contrast is low, there is almost no change in the image even if the above-described spatial low-pass filter is used. That is, the signal from the pixel element having the corresponding sensitivity and the color filter of the pixel block adjacent to the pixel block including the defective pixel element is included in the place such as the background 6132 regardless of the mode setting by the user. Should be used with priority. More specifically, the contrast is calculated by defining an area of an appropriate size including defective pixel elements. As the contrast here, for example, a value obtained by adding a difference absolute value of signals from adjacent pixels in a certain area and dividing by an average luminance value can be used. According to this definition, an image including a high frequency component has a high contrast, and a blurred image has a low contrast. When the above-described contrast is smaller than a specified value, signals from other pixel elements of the same pixel may be used with priority.

Next, a specific flow of the defect correction process of the present embodiment described above will be described with reference to the flowchart of FIG.
The defect correction process of this embodiment is started from step S6301. Step S6301 is a step indicating the start of the defect correction process.

  The subsequent step S6302 is a step of determining whether or not all defect correction processing has been completed. The position where the defective pixel exists is previously stored in the storage means. When the processes for all defective pixels stored in the storage unit have been completed (Y), the process proceeds to step S6314, and the defect correction process is terminated. If the processing has not been completed for all defective pixels (N), the process proceeds to step S6303.

  Step S6303 is a step of determining the contrast of the area where the defective pixel exists. That is, this is a step of performing processing corresponding to the processing described with reference to FIG. If the contrast is less than or equal to the specified value (Y), the process proceeds to step S6309. If not (N), the process proceeds to step S6304.

  Step S6304 is a step of determining whether or not the defective pixel is in the first pixel element. If the defective pixel is in the first pixel element (Y), the process proceeds to step S6305. If the defective pixel is in the second pixel element (N), the process proceeds to step S6306.

  Step S6305 is a step of determining whether the signal level of the other pixel element (second pixel element) sharing the light guide with the defective pixel is equal to or less than a specified value. If the signal level is equal to or lower than the specified value (Y), the process proceeds to step S6309. If not (N), the process proceeds to step S6307. By this process, it is avoided that defect correction is performed with reference to the signal of the second pixel element that has been crushed black.

  Step S6306 is a step of determining whether or not the signal level of the other pixel element (first pixel element) sharing the light guide with the defective pixel is equal to or higher than a specified value. If the signal level is equal to or higher than the specified value (Y), the process proceeds to step S6309. If not (N), the process proceeds to step S6307. By this processing, it is avoided that defect correction is performed with reference to the saturated signal of the first pixel element.

  Step S6307 is a step of determining whether or not gradation priority is given. If tone priority is given (Y), the process moves to step S6309. If not (N), the process moves to step S6308. Thereby, the defect correction process according to the user's intention is performed.

  Step S6308 is a step in which priority is given to other pixel elements. That is, in this step, as shown in FIG. 32A, a setting for defect correction is performed with priority given to the signal of the other pixel element sharing the light guide. Thereafter, the process proceeds to step S6313.

  Step S6309 is a step in which priority is given to pixel elements having defects. That is, in this step, as shown in FIG. 32B, defect correction is performed with priority given to the signal of the pixel element having the corresponding sensitivity and color filter in the pixel block adjacent to the pixel block including the defective pixel element. Settings are made. Thereafter, the process proceeds to step S6310.

  Step S6310 is a step of determining which of the vertical correlation and the horizontal correlation is higher. Here, as described in FIG. 32, the absolute value of the difference between the vertical pixel values is compared with the absolute value of the difference between the horizontal pixel values, and the direction in which the absolute value is small is assumed to have a high correlation. Define. If the correlation in the vertical direction is high, the process proceeds to step S6311. If the correlation in the horizontal direction is high, the process proceeds to step S6312.

  Step S6311 is a step of performing processing for increasing the weight in the vertical direction. Specifically, the weights of arrows 6111a and 6113a in FIG. 32 may be set larger than the weights of arrows 6110a and 6112a. Thereafter, the process proceeds to step S6313.

  Step S6312 is a step of performing processing to increase the weight in the horizontal direction. Specifically, the weights of arrows 6110a and 6112a in FIG. 32 may be set larger than the weights of arrows 6111a and 6113a. Thereafter, the process proceeds to step S6313.

  Step S6313 is a step for performing defect correction processing. In other words, this is a step of determining the value of the defective pixel using parameters such as the weights defined in the steps so far. Thereafter, the process returns to step S6302.

  In this way, by performing the steps described above for all defective pixels, the pixel value of the defective pixel can be corrected and an appropriate value can be estimated.

Next, an example in which the defect processing described in the present embodiment is performed after shooting by the information processing device will be described with reference to FIG.
FIG. 35A schematically illustrates a state in which the imaging device 6141 and the information processing device 6142 are connected via a cable 6143 used for data exchange between them. In the imaging device 6141, as shown in FIG. 35 (b), a file 6144 containing the first video data and a file 6145 containing the second video data are generated. Further, the information processing apparatus 6142 generates a file 6146 including video data generated using the files 6144 and 6145. In FIG. 35 (b), the black squares shown in the first video data and the second video data of the files 6144 and 6145 schematically indicate that there is a defective pixel element at that position.

  When the imaging device 6141 records the second video data synchronized with the first video data without loss (in the case of so-called RAW recording), the first pixel element and the second pixel element stored in the storage unit The information about the defect is recorded in the metadata. The file including the video data includes “picture A” video data (= first video data) or “picture B” video data (= second video data) and information for linking them. Metadata is stored. In addition, in the example of FIG. 35, in the file 6144 including the first video data, information regarding pixel element defects is recorded in addition to the above information.

  The information processing device 6142 includes a video data acquisition unit (not shown) that acquires files 6144 and 6145 including video data from the imaging device 6141. In the example of FIG. 35A in which the imaging device 6141 and the information processing device 6142 are connected via the cable 6143, the files 6144 and 6145 of the imaging device 6141 are sent to the information processing device 6142 via the cable 6143. The files 6144 and 6145 may be moved to the information processing apparatus 6142 using another method such as via the recording medium 193 shown in FIG.

  Thereafter, so-called development is performed in the information processing device 6142. The term “development” here refers to obtaining a single output video data by integrating a plurality of video data and defect information. Specifically, it means generating an image subjected to processing intended by the user such as HDR processing as shown in FIG. At the time of this development processing, settings are made by the user. That is, selection is made as to whether the mode prioritizes gradation or the mode prioritizes resolution. According to the setting (intention) of the user, the pixel value of the defective pixel may be corrected by switching the correction processing method as described above while referring to the information regarding the defect. As a result, as shown in FIG. 35B, a file 6146 containing the developed video data can be generated.

  As described above, according to the present embodiment, the pixel value of the defective pixel is appropriately corrected in the imaging device that simultaneously captures a plurality of images using an imaging element having pixels including a plurality of photoelectric conversion units having different light receiving efficiencies. Thus, an image according to the user's intention can be acquired.

[Modified Embodiment]
The present invention is not limited to the above embodiment, and various modifications can be made.
For example, the configuration of the imaging apparatus described in the above embodiment is an example, and the imaging apparatus to which the present invention can be applied is not limited to the configuration illustrated in FIGS. 1 and 2. Further, the circuit configuration of each part of the image sensor is not limited to the configuration shown in FIGS. For example, in the pixel circuit illustrated in FIG. 8, a selection transistor may be provided between the amplification transistor 315 and the signal output line 304. Further, the number of pixel elements constituting one pixel is not limited to two, and may be three or more.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 100 ... Imaging device 152 ... Shooting optical system 178 ... System control CPU
184: Image sensor 187, 188 ... Digital signal processing unit 303 ... Pixel 303A, 303B ... Pixel element 310A, 310B ... Photodiode (photoelectric conversion unit)
6105 ... Defective pixel element

Claims (17)

A plurality of pixels each including a first pixel element including a first photoelectric conversion unit and a second pixel element including a second photoelectric conversion unit having a sensitivity different from that of the first photoelectric conversion unit; A first video signal based on the signal charge generated in the first photoelectric conversion unit, and a video signal based on the signal charge generated in the second photoelectric conversion unit, wherein the first video signal An image sensor for acquiring a second video signal synchronized with the signal;
When the plurality of pixels includes a defective pixel in which the first pixel element is a defective pixel element, the pixel value of the defective pixel element in the first video signal is set to the second value of the defective pixel. And a signal processing unit that calculates based on a first pixel value of a pixel element and a second pixel value of the first pixel element of another pixel adjacent to the defective pixel. apparatus. 2. The imaging according to claim 1, wherein the signal processing unit adds the first pixel value and the second pixel value at a predetermined ratio to calculate the pixel value of the defective pixel element. apparatus. The signal processing unit, when acquiring an image giving priority to gradation, makes a ratio of summing the second pixel values higher than a ratio of summing the first pixel values. The imaging apparatus according to 1 or 2. When the change in the second pixel value of the other pixel differs depending on the direction adjacent to the defective pixel, the change in the pixel value of the other pixel located in a direction where the change in the pixel value is spatially small. The imaging apparatus according to claim 3, wherein the pixel value of the defective pixel is calculated using a second pixel value. The signal processing unit, when acquiring an image giving priority to resolution, sets a ratio of summing the first pixel values higher than a ratio of summing the second pixel values. Or the imaging device of 2. The signal processing unit, when the amount of signal charge generated in the second photoelectric conversion unit of the second pixel element of the defective pixel is greater than a predetermined value, a ratio of adding the second pixel value The imaging apparatus according to claim 1, wherein a higher ratio than the ratio of summing up the first pixel values is set. The signal processing unit, when the amount of signal charge generated in the second photoelectric conversion unit of the second pixel element of the defective pixel is less than a predetermined value, a ratio of adding the second pixel value The imaging apparatus according to claim 1, wherein a higher ratio than the ratio of summing up the first pixel values is set. When the contrast near the defective pixel is smaller than a predetermined value, the signal processing unit makes the ratio of summing the second pixel values higher than the ratio of summing the first pixel values. The imaging apparatus according to claim 1 or 2. The signal processing unit includes the pixel of the defective pixel element based on the first pixel value and a ratio of pixel values of the first pixel element and the second pixel element of the other pixels. The imaging device according to claim 1, wherein a value is calculated. 10. The position of the defective pixel element is recorded as metadata in the file when the first video signal and the second video signal are recorded in a file. The imaging apparatus according to item 1. The imaging device further includes a plurality of light guides that guide incident light to the pixels corresponding to the plurality of pixels,
10. The device according to claim 1, wherein the first photoelectric conversion unit and the second photoelectric conversion unit included in each of the pixels share one light guide. 11. Imaging device. The high dynamic range video composition unit for synthesizing a high dynamic range video using the first video signal and the second video signal, according to any one of claims 1 to 10. The imaging device described. A plurality of pixels each including a first pixel element including a first photoelectric conversion unit and a second pixel element including a second photoelectric conversion unit having a sensitivity different from that of the first photoelectric conversion unit; Video data acquired by the image sensor, the first video data based on the signal charge generated in the first photoelectric conversion unit, and the video based on the signal charge generated in the second photoelectric conversion unit A video data acquisition unit for acquiring second video data synchronized with the first video data;
When the plurality of pixels of the image sensor includes a defective pixel in which the first pixel element is a defective pixel element, the pixel value of the defective pixel element in the first video data is calculated as the pixel value of the defective pixel. A correction processing unit that performs correction based on a first pixel value of the second pixel element and a second pixel value of the first pixel element of another pixel adjacent to the defective pixel. A video processing device. The video processing apparatus according to claim 13, wherein a position where the defective pixel exists is recorded in a first file that records the first video data. A plurality of pixels each including a first pixel element including a first photoelectric conversion unit and a second pixel element including a second photoelectric conversion unit having a sensitivity different from that of the first photoelectric conversion unit; Video data acquired by the image sensor, the first video data based on the signal charge generated in the first photoelectric conversion unit, and the video based on the signal charge generated in the second photoelectric conversion unit Data, the second video data synchronized with the first video data,
When the plurality of pixels of the image sensor includes a defective pixel in which the first pixel element is a defective pixel element, the pixel value of the defective pixel element in the first video data is calculated as the pixel value of the defective pixel. An image processing method comprising: correcting based on a first pixel value of the second pixel element and a second pixel value of the first pixel element of another pixel adjacent to the defective pixel. . A program executed in a video processing apparatus that processes video data acquired by an imaging device having a plurality of pixels each including a first photoelectric conversion unit and a second photoelectric conversion unit, the computer comprising:
First video data based on the signal charge generated in the first photoelectric conversion unit, and video data based on the signal charge generated in the second photoelectric conversion unit, wherein the first video data Means for acquiring synchronized second video data, and the first video data when the plurality of pixels of the imaging device include a defective pixel in which the first pixel element is a defective pixel element. The pixel value of the defective pixel element in the first pixel value of the second pixel element of the defective pixel and the second pixel value of the first pixel element of another pixel adjacent to the defective pixel. A program that functions as a means of correction based on the above.   A computer-readable storage medium storing the program according to claim 16.