JP2020092346A - Imaging device and method of controlling the same - Google Patents

Abstract

To acquire a signal for image processing and a signal for phase difference detection from an imaging device without reducing a frame rate.SOLUTION: A pixel unit that configures an imaging device can selectively output a signal having phase difference information and a signal not having the phase difference information. A column amplifier unit can amplify a signal read out from the pixel unit at a plurality of different gains. A level determination unit determines a noise level of the signal (S502), and changes a drive method for the pixel unit and the column amplifier unit via a timing control unit. In a case where a level of a noise signal is equal to or less than a threshold, a plurality of signals multiplied by the first gain are acquired (S503, S504), and phase difference detection is performed. In addition, in a case where the level of the noise signal is larger than the threshold, a plurality of signals respectively multiplied by the first and second gains are acquired (S505, S506), and dynamic range expansion processing is performed.SELECTED DRAWING: Figure 5

Description

The present invention relates to a technique of acquiring an image processing signal and a phase difference detection signal from an image sensor.

The image pickup device used in the image pickup device such as a video camera outputs not only a signal photoelectrically converted in the pixel section, but also, for example, a dynamic range expansion (hereinafter also referred to as HDR) process, and output of distance information to a subject. It can be processed. The device disclosed in Patent Document 1 has a function of switching the input capacitance of an amplifier circuit provided for each column of the image sensor, and can switch the gain according to the signal level. By switching the gain, a low-gain signal and a high-gain signal are output, and by combining the two signals by image processing, it is possible to generate a video signal with a high dynamic range and low noise.

In addition, there is a method of performing focus detection of a phase difference detection method with an image sensor with respect to distance information from the image capturing apparatus to the subject. As an example of an image pickup element that outputs a signal that can be used for focus detection, there is an element that includes a two-dimensionally arranged microlens array and has a pair of photoelectric conversion units arranged for each microlens. In Patent Document 2, addition and non-addition of signals output from a pair of photoelectric conversion units on which light is incident through one microlens are arbitrarily performed in pixel units having a pair of photoelectric conversion units. An imaging device that can be used is disclosed.

JP, 2005-175517, A JP, 2001-83407, A

In the conventional apparatus, when the high-gain video signal and the low-gain video signal required for HDR processing and the signal for phase difference detection are read from the image sensor in the same frame, the data amount increases, and the frame rate Occurs. In the conventional technology, it is difficult to read out the video signal for HDR processing and the signal for phase difference detection while maintaining a desired frame rate in the image sensor.
An object of the present invention is to obtain a signal for image processing and a signal for phase difference detection from an image sensor without lowering the frame rate.

A device according to an exemplary embodiment of the present invention includes a pixel unit capable of selectively outputting a plurality of first signals having phase difference information and a plurality of second signals having no phase difference information, and the pixel unit. Read-out means for reading the first or second signal from the read-out means, amplifying means for amplifying the signal from the read-out means with a plurality of gains, the level of the signal is judged, and the pixel portion and the amplifying means are judged according to the judgment result. By controlling the first signal to obtain and process the first signal multiplied by the gain for phase difference detection, and the second signal multiplied by different gains for image processing respectively. Control means for performing second control for acquiring and processing.

According to the present invention, it is possible to acquire a signal for image processing and a signal for phase difference detection from an image sensor without lowering the frame rate.

1 is an overall block diagram of an image pickup apparatus according to an embodiment. FIG. 3 is an overall block diagram of the image sensor in the first embodiment. It is the schematic which shows the device structure of 1 pixel. It is a figure explaining the signal processing from a pixel part to a level determination part. It is a flowchart of the process in 1st Embodiment. It is a figure which shows the timing of the signal read-out for phase difference detection and HDR processing. It is a whole block diagram of the image sensor in a 2nd embodiment. It is a figure explaining the signal processing from a pixel part to a column ADC part.

Embodiments of the present invention will be described below in detail with reference to the drawings. In each embodiment, an application example to an image pickup apparatus such as a digital single-lens reflex camera, a digital still camera, and a digital video camera is shown. The imaging device included in the imaging device can selectively output a plurality of signals having phase difference information and a plurality of signals having no phase difference information.

[First Embodiment]
FIG. 1 is a block diagram showing the overall configuration of the image pickup apparatus according to this embodiment. The image pickup device includes a lens unit 100 and an image pickup element 101. The lens unit 100 has optical members such as a lens and a diaphragm that form an imaging optical system, and forms an image of light from a subject. The image sensor 101 photoelectrically converts the received light and outputs an electric signal.

The pixel distribution unit 102 acquires an image signal output from the image sensor 101 and distributes an HDR processing signal and a phase difference detection signal. The signal for dynamic range expansion processing is sent to the HDR image processing unit 103, and the signal for phase difference detection is sent to the phase difference detection unit 104.

The HDR image processing unit 103 acquires a plurality of signals with different gains from the pixel distribution unit 102 and executes HDR processing. The HDR image processing unit 103 outputs the composite image signal after the HDR processing to the signal processing unit 105. The signal processing unit 105 generates recording data on the recording medium, and also generates display data and the like to be displayed on the screen of the display unit included in the imaging device.

The phase difference detection unit 104 detects the phase difference between the plurality of signals acquired from the pixel distribution unit 102 and outputs a focus adjustment signal based on the focus detection result to the lens driving unit 106. Focus detection and focus adjustment will be described later with reference to FIG. The lens drive unit 106 performs a focus adjustment operation by driving the focus lens included in the lens unit 100 based on the focus adjustment signal.

FIG. 2 is a block diagram showing the configuration of the image sensor 101 according to this embodiment. The image sensor 101 includes a vertical scanning circuit 202, a column amplifier unit 204, a column ADC unit 205, a level determination unit 207, a horizontal transfer circuit 208, a signal processing circuit 209, and an external output circuit 210 which operate according to a signal from the timing control unit 201. , A pixel portion 203. The timing control unit 201 supplies an operation clock and timing signal to each unit of the image sensor 101 to control the operation.

The pixel units 203 of the image sensor 101 are arranged in a two-dimensional array, and image signals are acquired by a plurality of photoelectric conversion units. The photoelectric conversion unit has a photoelectric conversion element that performs photoelectric conversion according to the amount of incident light and outputs a signal voltage.

The vertical scanning circuit 202 performs timing control for sequentially reading pixel signal voltages from the pixel portion 203 during one frame. In general, a video signal is sequentially read in a row unit from the upper row to the lower row of the screen in one frame. The vertical scanning circuit 202 performs vertical scanning of the pixel portion 203, and signals output from the plurality of vertical lines are amplified by the column amplifier portion 204.

The column ADC unit 205 A/D converts the analog signal from the column amplifier unit 204 into a digital signal, and sends the digital signal to the level determination unit 207 and the horizontal transfer circuit 208 in the subsequent stage. Although the operation of the level determination unit 207 will be described later, the level determination unit 207 determines the noise level of the image signal, and performs a process of changing the driving method of the pixel unit 203, the column amplifier unit 204, and the column ADC unit 205 according to the determination result. To do.

The horizontal transfer circuit 208 transfers the signal A/D converted by the column ADC unit 205 to the signal processing circuit 209. The signal processing circuit 209 is a circuit that performs digital signal processing. The signal processing circuit 209 performs processing for adding a fixed amount of offset value as a digital value. Further, when the pixel portion 203 has a pixel area that is shielded by the light shielding area, digital black level clamp processing using a signal in the pixel area is performed. The signal processed by the signal processing circuit 209 is sent to the external output circuit 210. The external output circuit 210 acquires a signal from the signal processing circuit 209, performs conversion processing to a format for external output, and then outputs the signal to the outside of the image sensor 101.

FIG. 3 schematically shows the device structure of one pixel forming the pixel portion 203. In the example shown in FIG. 3, two photoelectric conversion units 302 and 303 corresponding to one microlens 301 are provided for detecting the phase difference. That is, in the pupil division type image pickup element, the phase difference can be detected from the output signals of the two photoelectric conversion units. A signal acquired from the first photoelectric conversion unit 302 is referred to as a pixel signal A, and a signal acquired from the second photoelectric conversion unit 303 is referred to as a pixel signal B. The pixel signal A and the pixel signal B are signals having phase difference information. The addition signal A+B of the pixel signal A and the pixel signal B can be used as a captured image signal. It should be noted that the addition signal A+B itself is a signal having no phase difference information, but in the combination with the pixel signal A or the pixel signal B, it is a signal having phase difference information. That is, the phase difference can be detected from the pixel signal A and the pixel signal B obtained by subtracting the pixel signal A from the addition signal A+B, or the pixel signal B and the pixel signal A obtained by subtracting the pixel signal B from the addition signal A+B. ..

The phase difference detection unit 104 (FIG. 1) detects the focus degree based on the phase difference between the pixel signal A and the pixel signal B. It is possible to calculate the image shift amount and the defocus amount. A color filter and a microlens are mounted on the surfaces of the photoelectric conversion units 302 and 303, respectively. Although it is common to use a Bayer array periodic structure of RGB primary color filters by using three color filters of R (red), G (green), and B (blue), this is not always the case. .. In this embodiment, the photoelectric conversion unit is divided into two, but the photoelectric conversion unit may be divided into three or more.

FIG. 4 is a diagram illustrating a flow of signal processing from the pixel unit 203 to the level determination unit 207. One of the pixel portions 203 indicated by a dotted frame includes photoelectric conversion elements 401 and 402, transfer switches 403 and 404, a charge holding portion 405, and a pixel amplifier 406. The pair of photoelectric conversion elements 401 and 402 share one microlens, and convert light into electric charges by photoelectric conversion. The photoelectric conversion elements 401 and 402 are composed of photodiodes (PD).

Transfer switches 403 and 404 are connected to the photoelectric conversion elements 401 and 402, respectively. The transfer switch 403 transfers the charge generated in the photoelectric conversion element 401 to a circuit in the subsequent stage. The transfer switch 404 transfers the charge generated in the photoelectric conversion element 402 to the circuit in the subsequent stage. The transfer switches 403 and 404 are composed of transistors.

The charge holding unit 405 is connected to the transfer switches 403 and 404, and temporarily holds the charges generated in the photoelectric conversion element 401 and the photoelectric conversion element 402. The charge holding portion 405 can hold only one of the charges of the photoelectric conversion element 401 and the photoelectric conversion element 402. Further, the charge holding portion 405 can hold a charge obtained by adding the charges of both the photoelectric conversion element 401 and the photoelectric conversion element 402.

The pixel amplifier 406 is connected to the charge holding unit 405, and outputs the signal of the charge holding unit 405 to the column amplifier unit 204 in the subsequent stage through the vertical line 413. The current controller 407 controls the current of the vertical line 413. The column amplifier unit 204 can amplify the output signal of the pixel amplifier 406 input from the vertical line 413 with two or more types of gains. The two or more types of gains include a gain for phase difference detection and a plurality of gains for image processing such as HDR processing.

The column ADC unit 205 includes an A/D conversion unit 409, memories 410 and 411, and a subtraction unit 412. The A/D converter 409 converts the analog signal from the column amplifier 204 into a digital signal. The memories 410 and 411 temporarily hold the digital signal converted by the A/D conversion unit 409. The memory 410 holds a pixel signal read from the photoelectric conversion element 401 or the photoelectric conversion element 402 and a noise signal of a reading circuit portion (a circuit from the charge holding portion 405 to the A/D conversion portion 409). The memory 411 holds the noise signal of the reading circuit unit. The subtraction unit 412 subtracts the signal held in the memory 411 from the signal held in the memory 410. The signal after the subtraction is output to the horizontal transfer circuit 208 as a pixel signal.

On the other hand, the noise signal stored in the memory 411 is output to the level determination unit 207. The level determination unit 207 outputs a determination signal according to the acquired level of the noise signal to the timing control unit 201. The timing control unit 201 switches the control method of the transfer switches 403 and 404 and the column amplifier unit 204 according to the determination signal of the level determination unit 207.

FIG. 5 is a flowchart illustrating the processing in this embodiment. In S501, the noise signal is read. The noise signal is a signal in a state where the transfer switch 403 and the transfer switch 404 are off and the charge in the charge holding portion 405 is reset. The noise signal is A/D converted by the A/D converter 409, and then stored and held in the memory 411.

In step S502, the level determination unit 207 determines the noise level by comparing the noise signal acquired from the memory 411 with a predetermined threshold value. When it is determined that the noise signal level is equal to or lower than the predetermined threshold value, the process proceeds to S503, and when it is determined that the noise signal level is higher than the threshold value, the process proceeds to S505.

In S503, the pixel signal A read from the photoelectric conversion unit 302 is amplified by the column amplifier unit 204 with the first gain. The amplified pixel signal A is stored in the memory 410. The subtraction unit 412 outputs a signal obtained by subtracting the noise signal output from the memory 411 from the pixel signal A output from the memory 410 to the horizontal transfer circuit 208.

In next step S504, the signal (denoted as pixel signal A+B) read from the photoelectric conversion unit 302 and the photoelectric conversion unit 303 is amplified by the column amplifier unit 204 with the first gain. The amplified pixel signal A+B is stored in the memory 410. The subtraction unit 412 outputs a signal obtained by subtracting the noise signal output from the memory 411 from the pixel signal A+B output from the memory 410 to the horizontal transfer circuit 208.

The pixel signal A and the pixel signal A+B are signals acquired for phase difference detection. The phase difference detection unit 104 detects the phase difference between the signal values by using the pixel signal A acquired in S503 and the pixel signal A+B acquired in S504. In the case of phase difference detection, the gain of the column amplifier section 204 is one type.

On the other hand, in step S<b>505, the pixel signal A+B read from the photoelectric conversion unit 302 and the photoelectric conversion unit 303 is amplified by the column amplifier unit 204 with the first gain, and then stored in the memory 410. The subtraction unit 412 outputs a signal obtained by subtracting the noise signal output from the memory 411 from the pixel signal A+B output from the memory 410 to the horizontal transfer circuit 208.

In next step S506, the pixel signal A+B read from the photoelectric conversion unit 302 and the photoelectric conversion unit 303 is amplified by the column amplifier unit 204 with a second gain different from the first gain, and then stored in the memory 410. It The subtraction unit 412 outputs a signal obtained by subtracting the noise signal output from the memory 411 from the pixel signal A+B output from the memory 410 to the horizontal transfer circuit 208.

The plurality of pixel signals A+B having different gains are signals acquired for the dynamic range expansion processing. The HDR image processing unit 103 generates an HDR image using the pixel signal A+B with the first gain acquired in S505 and the pixel signal A+B with the second gain acquired in S506. In the case of the dynamic range expansion processing, the gain of the column amplifier unit 204 is of two types.

Signal reading for each application will be described with reference to FIG. FIG. 6A is a diagram showing the timing of signal reading for phase difference detection shown in S501 to S504 of FIG. The signal reading for phase difference detection is performed in synchronization with the horizontal synchronizing signal 601. First, in the read period 602, the noise signal is read, and the level determination unit 207 determines the level of the noise signal. In the next read period 603, the pixel signal A is read at the first gain. In the next read period 604, the pixel signal A+B is read at the first gain.

FIG. 6B is a diagram showing the timing of signal reading for HDR processing shown in S501, S502, S505, and S506 of FIG. The signal reading for HDR processing is performed in synchronization with the horizontal synchronizing signal 701. First, in the read period 702, the noise signal is read and the level determination unit 207 determines the level of the noise signal. In the next read period 703, the pixel signal A+B is read at the first gain. In the next read period 704, the pixel signal A+B is read with the second gain.

In the present embodiment, a process of switching between signal reading for phase difference detection and signal reading for HDR processing is performed according to the level of the detected noise signal. This makes it possible to acquire and process each signal for phase difference detection and HDR processing without reducing the read frame rate of the image signal.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the present embodiment, differences from the first embodiment will be mainly described, and the same reference numerals as those used in the first embodiment will be used to omit the detailed description thereof.

FIG. 7 is an overall block diagram of the image sensor 101 in this embodiment. The difference from FIG. 2 is that the level determination unit 807 detects the level of the noise signal acquired from the pixel unit 203, and outputs a read method switching signal to the timing control unit 201 according to the noise level. is there. In this embodiment, the level determination unit 807 is arranged in the subsequent stage of the pixel unit 203. That is, the level determination unit 807 is arranged in a stage before the column amplifier unit 204.

FIG. 8 is a diagram showing a flow of signal processing from the pixel unit 203 to the column ADC unit 205. Since the configuration other than the level determination unit 807 is the same as the configuration shown in FIG. 4, description thereof will be omitted. The level determination unit 807 acquires the noise signal of the read circuit unit (the circuit from the charge holding unit 405 to the vertical line 413) from the vertical line 413. The level determination unit 807 determines the level of the noise signal and outputs a determination signal according to the level to the timing control unit 201. The timing control unit 201 switches control methods of the transfer switches 403 and 404 and the column amplifier unit 204 according to the determination signal.

In the present embodiment, in addition to the effects of the first embodiment, the level determination unit 807 determines the noise signal level before the column amplifier unit 204 and the column ADC unit 205, so that the processing time can be shortened. Become.

In the above embodiment, the level of the signal read from the pixel section is determined, and the driving method of the pixel section and the amplification section of the image sensor is changed according to the determination result. The level of the noise signal of the readout circuit unit whose level changes according to the brightness of the subject is determined, and with respect to the brightness of the subject, the signal for HDR processing is read when the subject is bright, and the position is read when the subject is dark. Signal reading for phase difference detection is performed. Further, by referring to the noise signal, it is possible to switch the signal reading method in a short time. The signal reading for phase difference detection is a method of reading the pixel signal A+B after reading the pixel signal A, and the read circuit portions (circuit portions after the charge holding portion 405) of the photoelectric conversion elements 401 and 402 can be shared and the circuit can be used. The area can be reduced.

In the above-described embodiment, one value may be used as the threshold value used by the level determination unit for determining the noise signal, or the threshold value may be changed according to the conditions. For example, the threshold value is changed corresponding to each gain by which the signal is multiplied by the column amplifier unit 204 and the A/D conversion unit 409. When the noise signal changes depending on the horizontal address or the vertical address of the pixel, the threshold value may be changed according to the horizontal address or the vertical address of the pixel. Further, when the noise signal changes according to the temperature of the surrounding environment, a threshold value depending on the temperature may be used. By doing so, the phase difference detection and the HDR process can be switched at an appropriate brightness level.

In the above embodiment, the level determination unit has been described as an example in which the level determination is performed by referring to the noise signal related to one target pixel, but the noise signals related to the target pixel and a plurality of pixels (peripheral pixels) located around the target pixel. The level may be determined by referring to. By doing so, it is possible to reduce signal variations in noise detection and to switch the signal reading method appropriately.

Although the present invention has been described above in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments. Although the example of the phase difference detection and the HDR processing has been described, the present invention changes the gain corresponding to the synthesis ratio of the parallax image composed of a plurality of images with different viewpoints acquired by using the pupil division type image sensor or the like, and the image is changed. It can be applied to a process of performing composition. Further, various forms within the scope of the present invention are also included in the technical scope of the present invention. Part of the above-described embodiments may be combined as appropriate.

101... Image sensor 103... HDR image processing unit 104... Phase difference detection unit 201... Timing control unit 203... Pixel unit 204... Column amplifier unit 205... Column ADC unit 207 , 807... Level determination unit 301... Micro lens 302, 303... Photoelectric conversion unit

Claims (10)

A pixel unit capable of selectively outputting a plurality of first signals having phase difference information and a plurality of second signals not having phase difference information;
Reading means for reading the first or second signal from the pixel portion;
Amplification means for amplifying the signal from the reading means with a plurality of gains;
A first control for determining the level of the signal and controlling the pixel unit and the amplifying means according to the determination result to acquire and process the first signal multiplied by the gain for phase difference detection; An image pickup apparatus comprising: a control unit configured to perform a second control for acquiring and processing the second signals that are respectively multiplied by different gains for image processing. The image pickup apparatus according to claim 1, wherein the control unit determines a noise level of the signal and performs a process of changing a driving method of the pixel unit and the amplification unit. When the noise level of the signal is higher than a threshold value, the amplifying means amplifies the plurality of read second signals by two types of gains, respectively.
The image pickup apparatus according to claim 2, wherein when the noise level of the signal is equal to or lower than a threshold value, the amplification unit amplifies each of the plurality of read first signals with one type of gain. The image pickup apparatus according to claim 2, wherein the control unit determines the level of the signal by using a threshold value corresponding to the gain. The image pickup according to claim 2 or 3, wherein the control unit determines the level of the signal using a horizontal address or a vertical address of the pixel unit or a threshold value corresponding to a temperature of a surrounding environment. apparatus. A conversion unit for converting a signal read from the pixel unit into a digital signal,
The control unit obtains a noise signal converted by the conversion unit or a noise signal before being converted by the conversion unit to perform level determination, and the level determination is performed. The image pickup device according to item 1. 7. The image pickup apparatus according to claim 2, wherein the control unit makes a level determination by comparing a noise signal for a pixel of interest in the pixel unit with a threshold value. 7. The control unit makes a level determination by comparing a noise signal for a pixel of interest in the pixel unit and a plurality of pixels located around the pixel of interest with a threshold value to determine a level. The image pickup device according to item 1. A first processing unit that acquires the first signal and detects a phase difference; and a second processing unit that acquires the second signal and performs a process of expanding a dynamic range. The imaging device according to any one of claims 1 to 8. A control method executed in an image pickup apparatus including an image pickup element capable of selectively outputting a plurality of first signals having phase difference information and a plurality of second signals not having phase difference information from a pixel unit And
Reading the first or second signal from the pixel portion;
A step of amplifying the read signal with a plurality of gains by an amplification means;
A first control for determining the level of the signal and controlling the pixel unit and the amplifying means according to the determination result to acquire and process the first signal multiplied by the gain for phase difference detection; And a step of performing a second control of acquiring and processing the second signal multiplied by different gains for image processing, respectively.

Images