JP2020171055A - Image pickup device - Google Patents

Abstract

To solve the problem that it is difficult to perform imaging while changing respective storage times of two photoelectric conversion parts included in one pixel.SOLUTION: An image pickup device comprises: a first photoelectric conversion unit and a second photoelectric conversion unit which convert light having passed through a microlens to electric charges and stores it; a first transfer unit which transfers the electronic charges from the first photoelectric conversion unit; a first transfer control line which sends a control signal to the first transfer unit; a second transfer unit for transferring electric charges from the second photoelectric conversion unit; and a second transfer control line which is provided separately from the first transfer control line and sends a control signal to the second transfer unit. The first transfer control line and the second transfer control line are provided on a side opposite from the incident side of the light having passed through the microlens with respect to the first photoelectric conversion unit and the second photoelectric conversion unit.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image sensor.

A CMOS image sensor having a plurality of photoelectric conversion units in one pixel is known. (See, for example, Patent Document 1). It has been difficult to perform imaging by changing the accumulation times of the two photoelectric conversion units in one pixel.

Japanese Unexamined Patent Publication No. 2001-250931

The image pickup device according to one aspect of the present invention may include a first photoelectric conversion unit and a second photoelectric conversion unit that convert light transmitted through the microlens into electric charges and store them. The image sensor may include a first transfer unit that transfers charges from the first photoelectric conversion unit. The image pickup device may include a first transfer control line that transmits a control signal to the first transfer unit. The image sensor may include a second transfer unit that transfers charges from the second photoelectric conversion unit. The image sensor may be provided separately from the first transfer control line, and may include a second transfer control line that transmits a control signal to the second transfer unit. The first transfer control line and the second transfer control line may be provided on the side opposite to the incident side of the light transmitted through the microlens with respect to the first photoelectric conversion unit and the second photoelectric conversion unit.

It is a figure which shows the image sensor which concerns on embodiment. It is a figure which shows the circuit structure of the image sensor which concerns on embodiment. It is a timing chart which shows an example of the operation of the image sensor which concerns on embodiment. It is a timing chart which shows other example of the operation of the image sensor which concerns on embodiment. It is a figure which shows other example of the plurality of photoelectric conversion part of the image pickup device which concerns on embodiment. It is a figure which shows the circuit structure of the image sensor which concerns on embodiment. It is a timing chart which shows an example of the operation of the image sensor which concerns on embodiment. It is a block diagram which shows the image pickup apparatus which concerns on embodiment. It is a flowchart which shows an example of the operation of the image pickup apparatus which concerns on embodiment.

[First Embodiment]
1 (A) is a plan view showing the image sensor 1 according to the present embodiment, FIG. 1 (B) is a plan view showing enlarged pixels, and FIG. 1 (C) is A-A of FIG. 1 (B). It is a cross-sectional view of line A. As shown in FIG. 1A, the image pickup device 1 has a plurality of pixels P arranged in the pixel region 1a. The plurality of pixels P are two-dimensionally arranged in a grid pattern. Of the arrangement directions of the pixels P, one is called the horizontal scanning direction and the other is called the vertical scanning direction. In the present embodiment, the direction perpendicular to each of the horizontal scanning direction and the vertical scanning direction is appropriately referred to as the normal direction of the pixel region 1a (imaging element 1). The image pickup device 1 has a substantially plate shape, and the normal direction of the pixel region 1a corresponds to the thickness direction of the image pickup device 1.

As shown in FIG. 1 (B), the pixel region 1a is provided with a light-shielding portion 2 that blocks light. The light-shielding portion 2 is, for example, a black matrix. The light-shielding portions 2 are formed in a grid pattern and extend in each of the horizontal scanning direction and the vertical scanning direction. The area surrounded by the light-shielding portion 2 is a pixel opening Pa through which light passes. In the present embodiment, the pixel P is a region surrounded by the center line of the light-shielding portion 2 when viewed from the normal direction of the pixel region 1a in FIG. 1, and includes a pixel opening Pa and a light-shielding portion 2 around the pixel opening Pa.

A photoelectric conversion unit 3 and a photoelectric conversion unit 4 are provided in each of the plurality of pixels P. When viewed from the normal direction of the pixel region 1a, at least a part of the photoelectric conversion unit 3 and at least a part of the photoelectric conversion unit 4 are arranged inside one pixel opening Pa. The photoelectric conversion unit 3 and the photoelectric conversion unit 4 each convert the light that has passed through the pixel aperture Pa into electric charges.

In the present embodiment, the image sensor 1 can capture a full-color image. The plurality of pixels P include a red pixel R, a green pixel Gr, a green pixel Gb, and a blue pixel B. In FIG. 1B, the arrangement of the pixels P is a Bayer arrangement, and the red pixels R and the green pixels Gr are alternately arranged, and the green pixels Gb and the blue pixels B are alternately arranged in the horizontal scanning direction. Further, in the vertical scanning direction, the red pixel R and the green pixel Gb are alternately arranged, and the green pixel Gr and the blue pixel B are alternately arranged. In the present embodiment, a photoelectric conversion unit 3 and a photoelectric conversion unit 4 are provided in each color pixel.

The image sensor 1 includes an image pickup unit 5, a signal processing unit 6, and a storage unit 7 that are laminated on each other. The signal processing unit 6 is laminated on the storage unit 7 and is electrically connected to the storage unit 7 by a conductive bump or the like (not shown). The image pickup unit 5 is laminated on the signal processing unit 6 and is electrically connected to the signal processing unit 6 by a conductive bump or the like (not shown).

The image pickup unit 5 is, for example, a back-illuminated CMOS image sensor. The imaging unit 5 includes an element layer 11, a light receiving layer 12, a color filter layer 13, and a lens layer 14. A passivation film, a flattening film, an antireflection film, or the like (not shown) may be provided between each of these layers. The imaging unit 5 is arranged with the lens layer 14 facing the imaging target when performing imaging. The light from the image pickup target enters the light receiving layer 12 via the lens layer 14 and the color filter layer 13.

The lens layer 14 is laminated on the color filter layer 13. The lens layer 14 is, for example, a microlens array and includes a plurality of lens elements 14a. The lens element 14a has, for example, one-to-one correspondence with the pixel P, and is provided for each pixel P. The optical axis 14b of the lens element 14a is set so as to pass through the center of the pixel aperture Pa (pixel P), for example. Each of the plurality of lens elements 14a collects the light incident from the outside on the light receiving layer 12 of the pixel P provided with the lens element 14a.

The color filter layer 13 is laminated on the light receiving layer 12. The color filter layer 13 includes a first filter 15, a second filter 16, and a light-shielding portion 2. The first filter 15 is provided in the pixel opening Pa of the red pixel R. The first filter 15 transmits light in the red wavelength band and absorbs light other than the red wavelength band. The red wavelength band includes, for example, 700 nm, and is a wavelength band of 620 nm or more and 750 nm or less. The second filter 16 is provided in the pixel opening Pa of the green pixel Gr. The second filter 16 transmits light in the green wavelength band and absorbs light other than the green wavelength band. The green wavelength band includes, for example, 546.1 nm, and is a wavelength band of 495 nm or more and 570 nm or less.

Although not shown in FIG. 1C, the pixel opening Pa of the green pixel Gb is provided with the second filter 16, and the pixel opening Pa of the blue pixel B is provided with the third filter. The third filter transmits light in the blue wavelength band and absorbs light other than the blue wavelength band. The blue wavelength band includes, for example, 435.8 nm, and is a wavelength band of 450 nm or more and 495 nm or less.

The light-shielding unit 2 absorbs light in a wavelength band in which the photoelectric conversion unit 3 and the photoelectric conversion unit 4 have sensitivity. For example, in the case of an image sensor used in a visible light camera or the like, the light-shielding portion 2 is formed of a material that absorbs light in a wavelength band of 380 nm or more and 780 nm or less. The light-shielding portion 2 is provided, for example, so as to suppress crosstalk between two adjacent pixels. For example, the thickness of the light-shielding unit 2 is set so as to block the light passing through the pixel opening Pa of the pixel P toward the light receiving layer 12 of another pixel P.

The light receiving layer 12 is laminated on the element layer 11. The light receiving layer 12 includes a photoelectric conversion unit 3 and a photoelectric conversion unit 4. The photoelectric conversion unit 3 and the photoelectric conversion unit 4 include, for example, a photodiode, but may include, for example, a phototransistor as long as it converts light into electric charges. In FIG. 1C, the photoelectric conversion unit 4 is provided symmetrically with respect to the optical axis of the lens element 14a and the surface parallel to the vertical scanning direction (the surface perpendicular to the horizontal scanning direction). ..

The photoelectric conversion unit 4 is provided in the vicinity of the photoelectric conversion unit 3 of the same pixel P. For example, the distance d1 between the photoelectric conversion unit 3 and the photoelectric conversion unit 4 in the same pixel P is set narrower than the distance d2 between the photoelectric conversion unit 3 and the photoelectric conversion unit 4 in the adjacent pixel P, but the distance d2. It may be set to be the same as, or it may be set longer than the interval d2.

The photoelectric conversion unit 3 and the photoelectric conversion unit 4 are insulated from each other. For example, the transfer of electric charge between the photoelectric conversion unit 3 and the photoelectric conversion unit 4 in the same pixel P is suppressed by, for example, a PN junction. Further, between the photoelectric conversion unit 3 and the photoelectric conversion unit 4 of the adjacent pixel P, for example, an insulating portion such as an oxide film or a trench structure formed by the LOCOS method is provided. The structure between the photoelectric conversion unit 3 and the photoelectric conversion unit 4 in the same pixel P may be the same as the structure between the photoelectric conversion unit 3 and the photoelectric conversion unit 4 in the adjacent pixel P. ..

The element layer 11 is provided on the side opposite to the incident side of light with respect to the light receiving layer 12. The element layer 11 is provided with a circuit (later shown in FIG. 2 and the like) for reading charges from each of the photoelectric conversion unit 3 and the photoelectric conversion unit 4. Further, the element layer 11 may be provided with a circuit that generates a signal based on the electric charge read from at least one of the photoelectric conversion unit 3 and the photoelectric conversion unit 4. These circuits include, for example, at least one of switching elements such as transistors, various wirings, capacitances, and electronic components of terminals. These circuits are formed in, for example, a multilayer structure using an interlayer insulating film and vias. At least a part of the electronic components constituting these circuits may be provided in the light receiving layer 12, or may be provided between the light receiving layer 12 and the lens layer 14.

The signal processing unit 6 has a first surface 6a facing the imaging unit 5 and a second surface 6b facing the storage unit 7. A circuit (not shown) for processing a signal output from the imaging unit 5 is provided on at least one of the first surface 6a and the second surface 6b. In addition, at least one of the first surface 6a and the second surface 6b may be provided with routing wiring used for rewiring or the like. The signal processing unit 6 is provided with a conductive unit that electrically connects the first surface 6a side and the second surface 6b side. This conductive portion includes, for example, a TSV (Through Silicon Via; through silicon via), and can transmit a signal between the first surface 6a and the second surface 6b. The storage unit 7 includes, for example, at least one of a non-volatile memory and a volatile memory, and stores a signal processed by the signal processing unit 6.

The signal processing unit 6 may be provided, for example, around the pixel region 1a shown in FIG. 1 instead of being laminated with the imaging unit 5. Further, the image sensor 1 does not have to include at least a part of the signal processing unit 6 and the storage unit 7. At least a part of the signal processing unit 6 and the storage unit 7 may be provided in an external device of the image pickup device 1. This external device may be provided in a device such as a camera or a measuring device on which the image sensor 1 is mounted.

FIG. 2 is a diagram showing a circuit configuration of the image sensor 1 according to the present embodiment. FIG. 2 shows an equivalent circuit diagram for one pixel. The image sensor 1 receives a first circuit system that reads out a first output signal based on the electric charge accumulated in the photoelectric conversion unit 3 of the pixel P, and a second output signal based on the electric charge accumulated in the photoelectric conversion unit 4 of the pixel P. Includes a second circuit system to read.

First, the first circuit system will be described. The first circuit system includes a transfer unit 21, a transfer control line 22, a charge-voltage conversion unit 23, a reset unit 24, a reset control line 25, a selection unit 26, a selection control line 27, an amplification unit 28, a power supply line 29, and a vertical signal. Includes line 30. The transfer unit 21, the reset unit 24, the selection unit 26, and the amplification unit 28 each include a switching element such as an n-type MOS transistor.

The anode of the photoelectric conversion unit 3 is grounded, and the cathode is connected to the source of the transfer unit 21. The transfer unit 21 is used to transfer electric charges from the photoelectric conversion unit 3. The drain of the transfer unit 21 is electrically connected to the node 31. The gate of the transfer unit 21 is connected to the transfer control line 22. The transfer control line 22 can transmit the control signal TX1 to the gate of the transfer unit 21.

The charge-voltage conversion unit 23 is, for example, a floating diffusion, and converts the charge from the transfer unit 21 into a voltage. The charge-voltage conversion unit 23 includes a capacity 32. The capacitance 32 has its first electrode grounded and its second electrode connected to the node 31. The capacity 32 may include at least a part of the capacity connected to the node 31. The capacity connected to the node 31 includes, for example, the capacity of the drain of the transfer unit 21, the coupling capacity of the wiring connected to the node 31 and other wiring, and the capacity of the source of the reset unit 24.

The reset unit 24 resets the voltage converted by the charge-voltage conversion unit 23. The source of the reset unit 24 is connected to the node 31. The drain of the reset unit 24 is connected to a power supply line 29 to which a predetermined potential is supplied with respect to the reference potential (ground potential). The gate of the reset unit 24 is connected to the reset control line 25. The reset control line 25 can transmit the control signal RST1 to the gate of the reset unit 24.

The selection unit 26 is used to select the pixel P for reading a signal from the plurality of pixels P arranged in the vertical scanning line direction shown in FIG. The source of the selection unit 26 is connected to the power line 29. The drain of the selection unit 26 is connected to the source of the amplification unit 28. The gate of the selection unit 26 is connected to the selection control line 27. The selection control line 27 can transmit the control signal SEL to the gate of the selection unit 26.

The amplification unit 28 amplifies the voltage converted by the charge-voltage conversion unit 23. The source of the amplification unit 28 is connected to the power supply line 29 via the selection unit 26. The power supply voltage VDD can be supplied to the source of the amplification unit 28 via the selection unit 26 and the power supply line 29. The drain of the amplification unit 28 is connected to the vertical signal line 30. The gate of the amplification unit 28 is connected to the node 31.

Next, a second circuit system for reading out a second output signal based on the electric charge accumulated in the photoelectric conversion unit 4 of the pixel P will be described. The second circuit system includes a transfer unit 41, a transfer control line 42, a charge-voltage conversion unit 43, a reset unit 44, a reset control line 45, a selection unit 46, an amplification unit 47, a selection control line 27, a power supply line 29, and a vertical signal. Includes line 48. The transfer unit 41, the reset unit 44, the selection unit 46, and the amplification unit 47 each include a switching element such as an n-type MOS transistor.

The anode of the photoelectric conversion unit 4 is grounded, and the cathode is connected to the source of the transfer unit 41. The transfer unit 41 is used to transfer electric charges from the photoelectric conversion unit 4. The drain of the transfer unit 41 is electrically connected to the node 50. The gate of the transfer unit 41 is connected to the transfer control line 42. The transfer control line 42 is provided separately from the transfer control line 22, and is insulated from the transfer control line 22. The transfer control line 42 can transmit the control signal TX2 to the gate of the transfer unit 41.

The charge-voltage conversion unit 43 is, for example, a floating diffusion, and converts the charge from the transfer unit 41 into a voltage. The charge-voltage conversion unit 43 includes a capacity 49. The capacitance 49 has its first electrode grounded and its second electrode connected to the node 50. The capacity 49 may include at least a part of the capacity connected to the node 50. The capacity connected to the node 50 includes, for example, the capacity of the drain of the transfer unit 41, the coupling capacity of the wiring connected to the node 50 and other wiring, and the capacity of the source of the reset unit 44.

The reset unit 44 resets the voltage converted by the charge-voltage conversion unit 43. The source of the reset unit 44 is connected to the node 50. The drain of the reset unit 44 is connected to the power supply line 29. The gate of the reset unit 44 is connected to the reset control line 45. The reset control line 45 can transmit the control signal RST2 to the gate of the reset unit 44.

The source of the selection unit 46 is connected to the power line 29. The drain of the selection unit 46 is connected to the source of the amplification unit 47. The gate of the selection unit 46 is connected to the selection control line 27. The selection control line 27 can transmit the control signal SEL to the gate of the selection unit 46.

The amplification unit 47 amplifies the voltage converted by the charge-voltage conversion unit 43. The source of the amplification unit 47 is connected to the power supply line 29 via the selection unit 46. The power supply voltage VDD can be supplied to the source of the amplification unit 47 via the selection unit 46 and the power supply line 29. The drain of the amplification unit 47 is connected to the vertical signal line 30. The gate of the amplification unit 47 is connected to the node 50.

Here, the circuit configuration for one pixel out of the plurality of pixels P shown in FIG. 1 has been described, but the plurality of pixels P all have the same circuit configuration. For example, the transfer control lines 22 extend substantially in the horizontal scanning direction and are periodically arranged in the vertical scanning direction. The plurality of transfer control lines 22 are connected to the gates of the respective transfer units 21 of the two or more pixels P arranged in the horizontal scanning direction. Further, the transfer control lines 42 extend substantially in the horizontal scanning direction and are periodically arranged in the vertical scanning direction. Each of the plurality of transfer control lines 42 is connected to the gate of each transfer unit 41 of two or more pixels P arranged in the horizontal scanning direction.

Further, the reset control lines 25 extend substantially in the horizontal scanning direction and are periodically arranged in the vertical scanning direction. Each of the plurality of reset control lines 25 is connected to the gate of each reset unit 24 of two or more pixels P arranged in the horizontal scanning direction. Further, the reset control lines 45 extend substantially in the horizontal scanning direction and are periodically arranged in the vertical scanning direction. Each of the plurality of reset control lines 45 is connected to the gate of each reset unit 44 of two or more pixels P arranged in the horizontal scanning direction.

Further, the selection control lines 27 extend substantially in the horizontal scanning direction and are periodically arranged in the vertical scanning direction. The plurality of selection control lines 27 are connected to the gate of the selection unit 26 and the gate of the selection unit 46 of each of the two or more pixels P arranged in the horizontal scanning direction, respectively. Further, the power supply line 29 is connected to the source of the selection unit 26, the source of the selection unit 46, the source of the reset unit 24, and the source of the reset unit 44, respectively, of the two or more pixels P. Further, the vertical signal lines 30 extend substantially in the vertical scanning direction and are periodically arranged in the horizontal scanning direction. Each of the plurality of vertical signal lines 30 is connected to the drain of each amplification unit 47 of two or more pixels P arranged in the vertical scanning direction.

In the present embodiment, the transfer control line 22 and the transfer control line 42 are provided on the element layer 11 shown in FIG. 1 (C). That is, the transfer control line 22 and the transfer control line 42 are arranged on the side opposite to the incident side of the light with respect to the photoelectric conversion unit 3 and the photoelectric conversion unit 4. Similarly, the reset control line 25, the reset control line 45, the selection control line 27, the power supply line 29, and the vertical signal line 30 are also on the side opposite to the light incident side with respect to the photoelectric conversion unit 3 and the photoelectric conversion unit 4. Have been placed.

At least a part of the transfer control line 22, the transfer control line 42, the reset control line 25, the reset control line 45, the selection control line 27, the power supply line 29, and the vertical signal line 30 are the photoelectric conversion unit 3 and the photoelectric conversion unit 3. It may be arranged on the same side as the incident side of the light with respect to 4. At least a part of these wirings may be provided in the light receiving layer 12 shown in FIG. 1C, or may be provided between the light receiving layer 12 and the lens layer 14.

The image pickup element 1 is provided with a vertical scanning circuit 51, a horizontal scanning circuit 52, a constant current source 53, and a constant current source 54. The vertical scanning circuit 51 supplies a control signal for reading a charge-based signal from each of the plurality of pixels P.

First, an example of an operation of reading out the first output signal based on the electric charge accumulated in the photoelectric conversion unit 3 will be described. The vertical scanning circuit 51 supplies the control signal SEL to the selection control line 27. When the vertical scanning circuit 51 sets the control signal SEL for the selection control line 27 to H level (high level), between the source and drain of the selection unit 26 connected to the selection control line 27 and the source of the selection unit 46. The space between the drains can be energized (hereinafter referred to as the on state). As a result, the power supply voltage VDD is supplied to the source of the amplification unit 28 via the power supply line 29 and the selection unit 26.

When reading the electric charge from the pixel P in another row, the selection unit 26 sets the control signal SEL for the selection control line 27 to the L level (low level), so that the selection unit 26 can be used as the source of the selection unit 26. The space between the drains cannot be energized (hereinafter referred to as the off state). Further, the vertical scanning circuit 51 can turn off the selection unit 26 during operation of a circuit (eg, an amplifier 55 described later) arranged downstream of the vertical signal line 30. In this case, the transmission of noise from the vertical signal line 30 to the amplification unit 28 can be suppressed.

Further, the vertical scanning circuit 51 supplies the control signal TX1 to the transfer control line 22. When the vertical scanning circuit 51 sets the control signal TX1 for the transfer control line 22 to H level, the transfer unit 21 connected to the transfer control line 22 is turned on, and the electric charge accumulated in the photoelectric conversion unit 3 is charged. It is transferred to the voltage conversion unit 23. The capacitance 32 is charged by the electric charge transferred from the photoelectric conversion unit 3 via the transfer unit 21, and the voltage between the first electrode and the second electrode becomes a voltage corresponding to the electric charge. This voltage is applied to the gate of the amplification unit 28 via the node 31, and the resistance value between the source and drain of the amplification unit 28 becomes a value corresponding to the voltage applied to the gate of the amplification unit 28. Further, between the source and the drain of the amplification unit 28, depending on the predetermined voltage applied to the source via the power supply line 29 and the selection unit 26 and the resistance value between the source and the drain of the amplification unit 28. Current (signal) flows. This signal is output to the vertical scanning circuit 51 via the vertical signal line 30.

Further, the vertical scanning circuit 51 supplies the control signal RST1 to the reset control line 25. When the vertical scanning circuit 51 sets the control signal RST1 for the reset control line 25 to H level, the reset unit 24 connected to the reset control line 25 is turned on and stored in the capacity 32 of the charge-voltage conversion unit 23. The charged charge is discharged via the reset unit 24 and the power supply line 29.

Next, an example of an operation of reading out the second output signal based on the electric charge accumulated in the photoelectric conversion unit 4 will be described. When the vertical scanning circuit 51 sets the control signal TX1 for the transfer control line 22 to H level, the power supply voltage VDD is supplied to the source of the amplification unit 47 via the power supply line 29 and the selection unit 46. The vertical scanning circuit 51 supplies the control signal TX2 to the transfer control line 42. When the vertical scanning circuit 51 sets the control signal TX2 for the transfer control line 42 to H level, the transfer unit 41 connected to the transfer control line 42 is turned on, and the electric charge accumulated in the photoelectric conversion unit 4 is charged. It is transferred to the voltage conversion unit 43.

The capacitance 49 is charged by the electric charge transferred from the photoelectric conversion unit 4 via the transfer unit 41, and the voltage between the first electrode and the second electrode becomes a voltage corresponding to the electric charge. This voltage is applied to the gate of the amplification unit 47 via the node 50, and the resistance value between the source and the drain of the amplification unit 47 becomes a value corresponding to the voltage applied to the gate of the amplification unit 47. Further, between the source and the drain of the amplification unit 47, depending on the predetermined voltage applied to the source via the power supply line 29 and the selection unit 46 and the resistance value between the source and the drain of the amplification unit 47. Current (signal) flows. This signal is output to the vertical scanning circuit 51 via the vertical signal line 48.

Further, the vertical scanning circuit 51 supplies the control signal RST2 to the reset control line 45. When the vertical scanning circuit 51 sets the control signal RST2 for the reset control line 45 to H level, the reset unit 44 connected to the reset control line 45 is turned on and stored in the capacity 49 of the charge-voltage conversion unit 43. The charged charge is discharged via the reset unit 44 and the power supply line 29.

In the present embodiment, the constant current source 53 is connected to the vertical signal line 30, and the constant current source 54 is connected to the vertical signal line 48. When the selection unit 26 is in the ON state, the source follower circuit is configured by the amplification unit 28, the selection unit 26, and the constant current source 53 connected to the vertical signal line 30. In this case, the signal of the pixel P belonging to the line selected by the selection unit 26 is output to the vertical signal line 30. Similarly, when the selection unit 46 is in the ON state, the source follower circuit is configured by the amplification unit 47, the selection unit 46, and the constant current source 54 connected to the vertical signal line 48.

The horizontal scanning circuit 52 converts the signals output from each of the plurality of pixels P into image data in a predetermined format. The horizontal scanning circuit 52 reduces the noise of the signal output from the pixel P by the correlated double sampling (Correlated Double Sampling).

The horizontal scanning circuit 52 includes an amplifier 55, an amplifier 56, and an analog-to-digital converter (not shown). The amplifier 55 is an amplification unit that amplifies a signal corresponding to the output from the photoelectric conversion unit 3. The amplifier 55 is, for example, a column amplifier, and is provided for each vertical signal line 30. The amplifier 55 amplifies the signal output via the vertical signal line 30. The amplifier 56 is an amplification unit that amplifies a signal according to the output from the photoelectric conversion unit 4. The amplifier 56 is, for example, a column amplifier, and is provided for each vertical signal line 48. The amplifier 56 amplifies the signal output via the vertical signal line 48.

In the present embodiment, the amplification factor (gain) of the amplifier 55 and the amplification factor of the amplifier 56 are variable. The amplification factor of the amplifier 55 can be set to the same value as or different from the amplification factor of the amplifier 56. The amplification factor of each of the amplifier 55 and the amplifier 56 can be set, for example, from 1 to several tens of times. When adjusting the respective amplification factors of the amplifier 55 and the amplifier 56, the amplification factor of the amplifier 55 and the amplification factor of the amplifier 56 may be adjusted in conjunction with each other, or the amplification factor of the amplifier 55 and the amplification factor of the amplifier 56 may be adjusted. May be adjusted independently. For example, the ratio of the amplification factor of the amplifier 55 to the amplification factor of the amplifier 56 may be constant or fixed.

The analog-to-digital converter of the horizontal scanning circuit 52 converts the analog signals output from each of the amplifier 55 and the amplifier 56 into digital signals. This digital signal includes, for example, a gradation value (pixel value) in which the output of each pixel P is represented by 8 bits. The horizontal scanning circuit 52 arranges pixel values representing the output of each pixel P according to the pixel array, and outputs the pixel values in a predetermined data format. The horizontal scanning circuit 52 may include at least a part of the constant current source 53 and the constant current source 54.

Next, the timing of reading the electric charge from the photoelectric conversion unit will be described. FIG. 3 is a timing chart showing an example of the operation of the image sensor 1.

The image sensor 1 performs dummy reading from the time it is activated until the first image is taken. As shown in FIG. 3, the vertical scanning circuit 51 sets the control signal RST1 for the reset control line 25 to the H level at time t1 before the period of the first frame, and the control signal RST2 for the reset control line 45. Is set to H level. As a result, the voltage between the electrodes in the capacity 32 of the charge-voltage conversion unit 23 and the voltage between the electrodes in the capacity 49 of the charge-voltage conversion unit 43 are reset. The vertical scanning circuit 51 sets the control signal RST1 and the control signal RST2 to the L level (low level) at time t2. At time t3, the vertical scanning circuit 51 sets the control signal TX1 for the transfer control line 22 to the H level, and sets the control signal TX2 for the transfer control line 42 to the H level. As a result, when the electric charge is accumulated in the photoelectric conversion unit 3, this charge is transferred to the charge-voltage conversion unit 23. Further, when the electric charge is accumulated in the photoelectric conversion unit 4, this charge is transferred to the charge-voltage conversion unit 43. The vertical scanning circuit 51 sets the control signal TX1 and the control signal TX2 to the L level at time t4.

Here, the dummy read from the photoelectric conversion unit 3 is performed in parallel with the dummy read from the photoelectric conversion unit 4, but the period during which the dummy read from the photoelectric conversion unit 3 and the dummy read from the photoelectric conversion unit 4 do not overlap. You may go to. For example, the vertical scanning circuit 51 may set the period for setting the control signal RST1 to H level and the period for setting the control signal RST2 to H level so that at least a part thereof overlaps or does not overlap. You may. Further, the vertical scanning circuit 51 may set at least a part of the period for setting the control signal TX1 to the H level and the period for setting the control signal TX2 to the H level so as not to overlap. You may.

The image sensor 1 starts the image pickup process of the first frame at time t5. The vertical scanning circuit 51 maintains the control signal SEL for the selection unit 26 at the H level. Further, the vertical scanning circuit 51 performs a reset operation on the photoelectric conversion unit 3 during the period from time t6 to time t7. During this period, the vertical scanning circuit 51 sets the control signal RST1 for the reset control line 25 to the H level and then to the L level. Next, the vertical scanning circuit 51 sets the control signal TX1 for the transfer unit 21 to the H level and then to the L level. Here, the vertical scanning circuit 51 sets each of the control signal RST1 and the control signal TX1 in the form of three pulses, but the number of pulses may be once, twice, or four or more. After the control signal TX1 is set to the L level at time t7, charges corresponding to the exposure amount are accumulated in the photoelectric conversion unit 3.

The vertical scanning circuit 51 reads out the electric charge from the photoelectric conversion unit 3 during the period from time t8 to time t10. The vertical scanning circuit 51 sets the control signal RST1 to the H level and then to the L level at time t8. Further, the vertical scanning circuit 51 sets the control signal TX1 to the H level at the time t9 after the time t8, and sets the control signal TX1 to the L level at the time t10. The exposure to the photoelectric conversion unit 3 during the period of the first frame ends at time t9. The photoelectric conversion unit 3 outputs the charge accumulated during the exposure period from the time t7 to the time t9 to the charge-voltage conversion unit 23 via the transfer unit 21. Further, the amplification unit 28 outputs a signal corresponding to the voltage converted by the charge-voltage conversion unit 23 to the vertical signal line 30.

In the present embodiment, the image pickup element 1 exposes the photoelectric conversion unit 4 during a period that overlaps with at least a part of the exposure period for the photoelectric conversion unit 3. The vertical scanning circuit 51 performs a reset operation on the photoelectric conversion unit 4 during the period from time t11 to time t12 after the time t7 in the period of the first frame. During this period, the vertical scanning circuit 51 sets the control signal RST2 for the reset control line 45 to the H level and then to the L level. Next, the vertical scanning circuit 51 sets the control signal TX2 for the transfer unit 41 to the H level and then to the L level. Here, the vertical scanning circuit 51 sets each of the control signal RST2 and the control signal TX2 in the form of three pulses, but the number of pulses may be one, two, or four or more. After the control signal TX2 is set to the L level at time t12, charges corresponding to the exposure amount are accumulated in the photoelectric conversion unit 4.

The vertical scanning circuit 51 reads out the electric charge from the photoelectric conversion unit 4 when the exposure period of the photoelectric conversion unit 4 ends. Here, the vertical scanning circuit 51 reads out the electric charge from the photoelectric conversion unit 4 in parallel with reading out the electric charge from the photoelectric conversion unit 3. The vertical scanning circuit 51 sets the control signal RST2 to the H level and then to the L level at time t8. Further, the vertical scanning circuit 51 sets the control signal TX2 at the H level at the time t9 and sets the control signal TX2 at the L level at the time t10. The photoelectric conversion unit 4 outputs the charge accumulated during the exposure period from the time t12 to the time t9 to the charge-voltage conversion unit 43 via the transfer unit 41. Further, the amplification unit 47 outputs a signal corresponding to the voltage converted by the charge-voltage conversion unit 43 to the vertical signal line 48.

As described above, the period of the first frame ends at time t10, and the period of the next second frame starts. In FIG. 3, the same operation as the period of the first frame is repeated in the period of the second frame.

In the example of FIG. 3, during the period in which the electric charge is accumulated in the photoelectric conversion unit 3 (exposure period from time t7 to time t9) in the period of the first frame, the electric charge is accumulated in the photoelectric conversion unit 4. It is set longer than the period (exposure period from time t12 to time t9).

As described above, in the imaging element 1, the photoelectric conversion unit 3 and the photoelectric conversion unit 4 have different exposure periods, and the first output signal based on the electric charge accumulated in the photoelectric conversion unit 3 and the photoelectric conversion unit 3 are used. A second output signal based on the charge stored in 4 can be generated. By using such a first output signal and a second output signal, it is possible to generate two images having different exposure periods. Thereby, for example, an image having a wide dynamic range can be generated. For example, when the exposure period of the first output signal is longer than the exposure period of the second output signal, the first output signal is used to generate an image of a relatively dark dark part of the imaged object, and the second output signal is used. To generate an image of a relatively bright bright part of the imaged object. By synthesizing these images, it is possible to brighten the dark part while suppressing the saturation of the bright part.

By the way, in order to widen the dynamic range, there is a method of alternately repeating a frame having a long exposure time and a frame having a short exposure time by using an image pickup element in which the number of photoelectric conversion units arranged in one pixel is one. .. In this method, two frames are used to obtain an image of one frame, which slows down the frame rate. Further, as another method, there is a method of using a signal from an image having a short exposure time and a signal having a long exposure time. In this method, a signal of two pixels is used to form one pixel of the image, and the resolution is lowered.

In the present embodiment, since a plurality of photoelectric conversion units (photoelectric conversion unit 3 and photoelectric conversion unit 4) are arranged in one pixel, the dynamic range can be widened without lowering the resolution of the captured image. Further, the transfer control line 22 that transmits the control signal TX1 from the photoelectric conversion unit 3 to the transfer unit 21 for transferring the electric charge transfers the control signal TX2 to the transfer unit 41 for transferring the electric charge from the photoelectric conversion unit 4. Since it is provided separately from the wire 42, the exposure to the photoelectric conversion unit 3 and the exposure to the photoelectric conversion unit 4 can be performed in parallel, and the decrease in the frame rate can be suppressed. In such a case, if the transfer control line 22 and the transfer control line 42 are arranged on the opposite side of the light incident side with respect to the photoelectric conversion unit 3 and the photoelectric conversion unit 4, the light receiving of the photoelectric conversion unit 3 and the photoelectric conversion unit 4 is received. It is possible to avoid narrowing the area. The dynamic range can be widened by adjusting the ratio between the amplification factor of the amplifier 55 and the amplification factor of the amplifier 56.

By the way, in FIG. 3, the time t9 at which the exposure to the photoelectric conversion unit 4 ends is set to be substantially the same as the time t9 at which the exposure to the photoelectric conversion unit 3 ends. Therefore, it is possible to avoid complicated control for reading a signal from the pixel P.

FIG. 4 is a timing chart showing another example of the operation of the image sensor 1. 4 (A) to 4 (D) show timing charts for a period of one frame, respectively. In each of FIGS. 4A to 4D, the operation of reading the electric charge from the photoelectric conversion unit 3 is the same as the example shown in FIG. 3, so the description thereof is omitted or simplified. To become.

In FIG. 4A, the time at which the exposure period for the photoelectric conversion unit 4 ends is set to a time different from the time at which the exposure period for the photoelectric conversion unit 3 ends. The vertical scanning circuit 51 resets the photoelectric conversion unit 3 and resets the photoelectric conversion unit 4 in parallel during the period from time t6 to time t7. At time t7, the exposure to the photoelectric conversion unit 3 starts, and the exposure to the photoelectric conversion unit 4 starts.

At time t15, the vertical scanning circuit 51 sets the control signal RST2 for the reset unit 44 to the H level and then to the L level. Further, the vertical scanning circuit 51 sets the control signal TX2 for the transfer unit 41 to the H level and then to the L level at time t16. The exposure period (charge accumulation period) for the photoelectric conversion unit 4 in the period of the first frame is a period from time t7 to time t16.

In FIG. 4B, the time at which the exposure period for the photoelectric conversion unit 4 starts is set to a time different from the time at which the exposure period for the photoelectric conversion unit 3 starts. Further, the time at which the exposure period for the photoelectric conversion unit 4 ends is set to a time different from the time at which the exposure period for the photoelectric conversion unit 3 ends. The vertical scanning circuit 51 resets the photoelectric conversion unit 4 during the period from time t17 to time t18 after the time t6 when the reset to the photoelectric conversion unit 3 is started. The exposure period for the photoelectric conversion unit 4 starts at time t18.

At time t19, the vertical scanning circuit 51 sets the control signal RST2 for the reset unit 44 to the H level and then to the L level. Further, the vertical scanning circuit 51 sets the control signal TX2 for the transfer unit 41 to the H level and then to the L level at time t20. The exposure period (charge accumulation period) for the photoelectric conversion unit 4 in the period of the first frame is a period from time t18 to time t20. Here, the exposure period for the photoelectric conversion unit 4 is set so that the time at the center thereof coincides with the time at the center of the exposure period for the photoelectric conversion unit 3.

In FIG. 4C, a plurality of exposure periods for the photoelectric conversion unit 4 are set. The vertical scanning circuit 51 resets the photoelectric conversion unit 4 during the period from time t6 to time t7. The first exposure period for the photoelectric conversion unit 4 starts at time t7. Further, at time t21, the vertical scanning circuit 51 sets the control signal RST2 for the reset unit 44 to the H level and then to the L level. Further, the vertical scanning circuit 51 sets the control signal TX2 for the transfer unit 41 to the H level and then to the L level at time t22. The first exposure period to the photoelectric conversion unit 4 in the period of the first frame is the period from time t7 to time t22. Next, the vertical scanning circuit 51 resets the photoelectric conversion unit 4 during the period from time t23 to time t24. The second exposure period for the photoelectric conversion unit 4 starts at time t24. At time t8, the vertical scanning circuit 51 sets the control signal RST2 for the reset unit 44 to the H level and then to the L level. Further, the vertical scanning circuit 51 sets the control signal TX2 for the transfer unit 41 to the H level and then to the L level at time t9. The second exposure period for the photoelectric conversion unit 4 in the period of the first frame is the period from time t24 to time t9.

The length of the first exposure period for the photoelectric conversion unit 4 in one frame may be set to the same length as the second exposure period, or may be set to a length different from the second exposure period. You may be. When a plurality of exposures are performed in the period of one frame in this way, the signal based on the charge accumulated in the photoelectric conversion unit 4 in the first exposure period and the signal accumulated in the photoelectric conversion unit 4 in the second exposure period are accumulated. Calculations such as averaging of these signals may be performed using the signals based on the electric charge. When the length of the first exposure period and the length of the second exposure period for the photoelectric conversion unit 4 are different, the signal based on the charge accumulated in the photoelectric conversion unit 4 in the first exposure period and the second The dynamic range can be further expanded by using the signal based on the electric charge accumulated in the photoelectric conversion unit 4 during the exposure period and the signal based on the electric charge accumulated in the photoelectric conversion unit 3 during the period of this frame.

In FIG. 4D, the exposure period for the photoelectric conversion unit 4 is set to the same length as the exposure period for the photoelectric conversion unit 3. Here, the time t7 at which the exposure period for the photoelectric conversion unit 4 starts is set to be the same as the time t7 at which the exposure period for the photoelectric conversion unit 3 starts. Further, the time t9 at which the exposure period for the photoelectric conversion unit 4 ends is set to be the same as the time t9 at which the exposure period for the photoelectric conversion unit 3 ends.

When the exposure period is set to the same length in the photoelectric conversion unit 3 and the photoelectric conversion unit 4, the first output signal based on the electric charge accumulated in the photoelectric conversion unit 3 during the period of the first frame and the first output signal Information on focusing in the image pickup element 1 may be detected by using the second output signal based on the electric charge accumulated in the photoelectric conversion unit 4 during the period of one frame. For example, when the levels of the first output signal and the second output signal are the same, it may be determined that the light receiving layer 12 of the image sensor 1 is in focus. Further, when one level of the first output signal and the second output signal is higher than the other level, it may be determined that the focus is on the object surface side of the light receiving layer 12 (front pin). .. Further, when one level of the first output signal and the second output signal is lower than the other level, it may be determined that the light receiving layer 12 is on the object surface side of the focal point (rear pin). ..

When the exposure periods of the photoelectric conversion unit 3 and the photoelectric conversion unit 4 are set to different lengths, the first output signal and the second output signal are used to detect information on focusing in the image sensor 1. You may. In this case, using the estimated value of the signal level when the lengths of the exposure periods are the same, the exposure periods are set to the same length in the photoelectric conversion unit 3 and the photoelectric conversion unit 4. And can detect information about focusing. The estimated value of the signal level when the lengths of the exposure periods are the same is, for example, the level of the first output signal when the length of the exposure period for the photoelectric conversion unit 4 is half the exposure period for the photoelectric conversion unit 3. Can be obtained by halving or doubling the level of the second output signal. To adjust the level of the first output signal, the amplification factor of the amplifier 55 shown in FIG. 2 may be adjusted, and to adjust the level of the second output signal, the amplification factor of the amplifier 56 may be adjusted. You may. Further, in order to adjust at least one of the level of the first output signal and the level of the second output signal, arithmetic processing may be performed by the horizontal scanning circuit 52 shown in FIG. 2 or an external device.

By the way, in the present embodiment, when a plurality of photoelectric conversion units (photoelectric conversion unit 3 and photoelectric conversion unit 4) are arranged in one pixel, the light receiving area is equal to the total light receiving area of the plurality of photoelectric conversion units1. Compared with the case of using one photoelectric conversion unit, the output per photoelectric conversion unit becomes smaller. Therefore, when the level of the first output signal or the second output is insufficient, the level of the first output signal or the second output signal may be adjusted by adjusting the amplification factor of the amplifier 55 or the amplifier 56. .. For example, when the amplification factor of the amplifier 55 is set to double, when one photoelectric conversion unit having a light receiving area equal to the total light receiving area of the photoelectric conversion unit 3 and the photoelectric conversion unit 4 is used for the level of the first output signal. Can be at the same level as.

The example shown in FIG. 3 and the examples shown in FIGS. 4 (A) to 4 (D) can be combined. For example, the image pickup element 1 has a first mode (see FIG. 4D) in which the exposure time of the photoelectric conversion unit 3 and the exposure time of the photoelectric conversion unit 4 are the same length, and the exposure time and photoelectric light of the photoelectric conversion unit 3. It is possible to switch the exposure time of the conversion unit 4 to a different length from the second mode. In the first mode, the vertical scanning circuit 51 sets, for example, the waveform of the control signal TX1 for the transfer unit 21 and the waveform of the control signal TX2 for the transfer unit 41 to be the same. Further, in the second mode, the vertical scanning circuit 51 sets, for example, the waveform of the control signal TX1 for the transfer unit 21 and the waveform of the control signal TX2 for the transfer unit 41 as different waveforms. The second mode may include one of the examples shown in FIG. 3 and the examples shown in FIGS. 4 (A) to 4 (C), or may include a combination of two or more types. ..

In the above example, the length of the exposure period for the photoelectric conversion unit 3 is set to be equal to or longer than the length of the exposure period for the photoelectric conversion unit 4, but is less than the length of the exposure period for the photoelectric conversion unit 4. It may be set. Of the exposure time of the photoelectric conversion unit 3 and the exposure time of the photoelectric conversion unit 4, one of the exposure times may be set to 1/10 times or more and 1 times or less of the exposure time of the other, or 1/10 times. It may be set to less than.

Next, another example of a plurality of photoelectric conversion units arranged in one pixel will be described. FIG. 5 is a diagram showing another example of the plurality of photoelectric conversion units.

In FIG. 5A, the photoelectric conversion unit 3 is provided asymmetrically with the photoelectric conversion unit 4 when viewed from the normal direction of the pixel region 1a. The light receiving area of the photoelectric conversion unit 3 is set to be larger than the light receiving area of the photoelectric conversion unit 4. When the light receiving area is different between the photoelectric conversion unit 3 and the photoelectric conversion unit 4, the dynamic range can be further expanded. As described above, the photoelectric conversion unit 3 may be different from the photoelectric conversion unit 4 in at least one of the shape and the dimensions.

In FIG. 5B, a photoelectric conversion unit 3, a photoelectric conversion unit 4, and a photoelectric conversion unit 60 are provided in one pixel. The light receiving area of the photoelectric conversion unit 3 is set to be larger than the light receiving area of the photoelectric conversion unit 4. The light receiving area of the photoelectric conversion unit 60 is set to be substantially the same as the light receiving area of the photoelectric conversion unit 4. The photoelectric conversion unit 60 is provided symmetrically with respect to the center line of the pixel P with respect to the photoelectric conversion unit 4. As described above, the number of photoelectric conversion units arranged in one pixel may be three or four or more. Further, of the plurality of photoelectric conversion units, at least one of the shapes and dimensions of any of the photoelectric conversion units may be the same as the other photoelectric conversion units or may be different from the other photoelectric conversion units. ..

When such an image pickup element 1 is used, for example, an image pickup is performed using a second output signal based on the electric charge accumulated in the photoelectric conversion unit 4 and a third output signal based on the electric charge accumulated in the photoelectric conversion unit 60. Information on focusing in element 1 can also be detected. Further, the dynamic range can be widened by using at least two of the first output signal, the second output signal, and the third output signal based on the electric charge accumulated in the photoelectric conversion unit 3.

[Second Embodiment]
Next, the second embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment are designated by the same reference numerals to simplify or omit the description thereof.

FIG. 6 is a diagram showing a circuit configuration of the image sensor 1 according to the present embodiment. FIG. 6 shows an equivalent circuit diagram for two pixels. The circuit configuration of the pixel P1 is the same as that of the above-described embodiment (see FIG. 2). Pixel P2 is a pixel arranged next to pixel P1. For example, the pixel P is the red pixel R in FIG. 1 (B), and the pixel P2 is the green pixel Gb.

Within the pixel P2, the pixel P2 is provided with a photoelectric conversion unit 61 and a photoelectric conversion unit 62. The photoelectric conversion unit 61 and the photoelectric conversion unit 62 include, for example, a photodiode, but may include, for example, a phototransistor as long as it converts light into electric charges. The image sensor 1 has a third circuit system that reads out an output signal based on the electric charge stored in the photoelectric conversion unit 61 of the pixel P2, and a fourth circuit that reads out an output signal based on the electric charge stored in the photoelectric conversion unit 62 of the pixel P2. Including the system.

First, the third circuit system will be described. The third circuit system includes a transfer unit 63, a transfer control line 64, a charge / voltage conversion unit 23, a reset unit 24, a reset control line 25, a selection unit 26, a selection control line 27, an amplification unit 28, a power supply line 29, and a vertical signal. Includes line 30. That is, the third circuit system is shared with the first circuit system except for the transfer unit 63 and the transfer control line 64. The transfer unit 63 includes, for example, a switching element such as an n-type MOS transistor.

The anode of the photoelectric conversion unit 61 is grounded, and the cathode is connected to the source of the transfer unit 63. The transfer unit 63 is used to transfer electric charges from the photoelectric conversion unit 61. The drain of the transfer unit 63 is electrically connected to the node 31. The gate of the transfer unit 63 is connected to the transfer control line 64. The transfer control line 64 can transmit the control signal TX3 to the gate of the transfer unit 63. The charge-voltage conversion unit 23 converts the charge from the transfer unit 21 into a voltage and the charge from the transfer unit 63 into a voltage.

Next, the fourth circuit system will be described. The fourth circuit system includes a transfer unit 65, a transfer control line 66, a charge / voltage conversion unit 43, a reset unit 44, a reset control line 45, a selection unit 46, a selection control line 27, an amplification unit 47, a power supply line 29, and a vertical signal. Includes line 30. That is, the fourth circuit system is shared with the second circuit system except for the transfer unit 65 and the transfer control line 66. The transfer unit 65 includes, for example, a switching element such as an n-type MOS transistor.

The anode of the photoelectric conversion unit 62 is grounded, and its cathode is connected to the source of the transfer unit 65. The transfer unit 65 is used to transfer electric charges from the photoelectric conversion unit 62. The drain of the transfer unit 65 is electrically connected to the node 50. The gate of the transfer unit 65 is connected to the transfer control line 66. The transfer control line 66 can transmit the control signal TX4 to the gate of the transfer unit 65. The charge-voltage conversion unit 43 converts the charge from the transfer unit 41 into a voltage, and converts the charge from the transfer unit 65 into a voltage.

In the present embodiment, the transfer control line 64 and the transfer control line 66 are provided on the element layer 11 shown in FIG. 1 (C). That is, the transfer control line 64 and the transfer control line 66 are arranged on the side opposite to the incident side of the light with respect to the photoelectric conversion unit 61 and the photoelectric conversion unit 62. At least a part of the transfer control line 64 and the transfer control line 66 may be arranged on the same side as the incident side of the light with respect to the photoelectric conversion unit 61 and the photoelectric conversion unit 62. At least a part of these wirings may be provided in the light receiving layer 12 shown in FIG. 1C, or may be provided between the light receiving layer 12 and the lens layer 14.

Next, the timing of reading the electric charge from the photoelectric conversion unit will be described. FIG. 7 is a timing chart showing an example of the operation of the image sensor 1.

The image sensor 1 performs dummy reading as in the first embodiment from the time it is activated until the first image is taken. Further, the image sensor 1 reads out a signal from the pixel P in the period of the first frame as in the first embodiment. The image sensor 1 reads a signal from the pixel P2 in the period of the second frame following the period of the first frame.

The image sensor 1 starts the image pickup process of the second frame at time t10. The vertical scanning circuit 51 maintains the control signal SEL for the selection unit 26 at the H level. Further, the vertical scanning circuit 51 performs a reset operation on the photoelectric conversion unit 61 during the period from the time t30 to the time t31. During this period, the vertical scanning circuit 51 sets the control signal RST1 for the reset control line 25 to the H level and then to the L level. Next, the vertical scanning circuit 51 sets the control signal TX3 for the transfer unit 21 to the H level and then to the L level. After the control signal TX3 is set to the L level at time t31, charges corresponding to the exposure amount are accumulated in the photoelectric conversion unit 61.

The vertical scanning circuit 51 reads out the electric charge from the photoelectric conversion unit 61 during the period from the time t32 to the time t33. The vertical scanning circuit 51 sets the control signal RST1 to the H level and then to the L level at time t32. Further, the vertical scanning circuit 51 sets the control signal TX3 at the H level at the time t33 after the time t32, and sets the control signal TX3 at the L level at the time t34. The exposure to the photoelectric conversion unit 3 during the period of the second frame ends at time t33. The photoelectric conversion unit 61 outputs the charge accumulated during the exposure period from the time t31 to the time t33 to the charge-voltage conversion unit 23 via the transfer unit 63. Further, the amplification unit 28 outputs a signal corresponding to the voltage converted by the charge-voltage conversion unit 23 to the vertical signal line 30.

In the present embodiment, the image pickup element 1 can expose the photoelectric conversion unit 62 during a period that overlaps with at least a part of the exposure period for the photoelectric conversion unit 61. The vertical scanning circuit 51 resets the photoelectric conversion unit 62 during the period from time t35 to time t36 after the time t31 in the period of the second frame. During this period, the vertical scanning circuit 51 sets the control signal RST2 for the reset control line 45 to the H level and then to the L level. Next, the vertical scanning circuit 51 sets the control signal TX4 for the transfer unit 65 to the H level and then to the L level. After the control signal TX4 is set to the L level at time t36, charges corresponding to the exposure amount are accumulated in the photoelectric conversion unit 62.

The vertical scanning circuit 51 reads out the electric charge from the photoelectric conversion unit 62 when the exposure period of the photoelectric conversion unit 62 ends. Here, the vertical scanning circuit 51 reads out the electric charge from the photoelectric conversion unit 62 in parallel with reading out the electric charge from the photoelectric conversion unit 61. The vertical scanning circuit 51 sets the control signal RST2 to the H level and then to the L level at time t32. Further, the vertical scanning circuit 51 sets the control signal TX4 at the H level at the time t33 and sets the control signal TX4 at the L level at the time t34. The photoelectric conversion unit 62 outputs the charge accumulated during the exposure period from the time t36 to the time t33 to the charge-voltage conversion unit 43 via the transfer unit 65. Further, the amplification unit 47 outputs a signal corresponding to the voltage converted by the charge-voltage conversion unit 43 to the vertical signal line 48. In this way, the period of the second frame ends at time t34, and the period of the next frame starts. In the period of the next frame after the period of the second frame, the exposure to the pixel P1 and the reading of the signal from the pixel P1 are performed as in the period of the first frame.

In the above-described embodiment, the image pickup device 1 can detect light in three wavelength bands, but may detect only light in one wavelength band (monochromatic light) or two. It may detect only light in a wavelength band or light in four or more wavelength bands. Further, the pixel arrangement does not have to be the Bayer arrangement and can be arbitrarily selected. Further, the image sensor 1 may have sensitivity in an invisible light region, and may, for example, detect at least one of infrared light and ultraviolet light.

The selection unit 26 and the selection control line 27 shown in FIGS. 2 and 6 can be omitted. When the selection unit 26 and the selection control line 727 are not provided, the timing of reading the electric charge from the pixel P can be controlled by the timing of turning on the transfer unit 21, the transfer unit 41, and the like.

[Third Embodiment]
Next, an image pickup device using the image pickup device 1 described in the above-described embodiment will be described. FIG. 8 is a block diagram showing the configuration of the image pickup apparatus 70 according to the present embodiment. The imaging device 70 shown in FIG. 8 is at least a part of an electronic device such as a digital camera, a smartphone, a mobile phone, a personal computer, or a measuring device. The image pickup device 70 includes a lens unit 71, a lens drive unit 72, an image sensor 1, a control unit 73, a work memory 74, a display unit 75, an operation unit 76, and a recording unit 77.

The lens unit 71 includes, for example, an imaging optical system including a plurality of lenses and a lens barrel. The image pickup optical system of the lens unit 71 collects the light from the subject and guides it to the image pickup element 1. The image pickup optical system of the lens unit 71 forms an image of the subject on the light receiving layer 12 (see FIG. 1C) of the image pickup element 1. The lens unit 71 may be integrated with at least a body of the image pickup device 70 that houses the control unit 73, or may be an interchangeable lens that can be attached to and detached from the body. The lens unit 71 may include a focus lens or a zoom lens. The lens driving unit 72 drives the lens of the lens unit 71 and adjusts the focus of the lens unit 71.

The control unit 73 controls each unit of the image pickup device 70. The control unit 73 includes an image pickup control unit 80, an image processing unit 81, and a focus detection unit 82.

The image pickup control unit 80 causes the image pickup element 1 to execute the image pickup process by sending a command to the image pickup element 1. For example, the image pickup control unit 80 controls the timing of starting the exposure and the timing of ending the exposure of each photoelectric conversion unit by sending a command to the vertical scanning circuit 51 of the image pickup element 1 shown in FIG. Further, the image pickup control unit 80 controls the timing of reading a signal from each pixel to the horizontal scanning circuit 52 by sending a command to the vertical scanning circuit 51. Further, the image pickup control unit 80 sends a command to the image pickup element 1 to output (transmit) the image pickup result to the control unit 73. For example, the imaging control unit 80 sends a command to the horizontal scanning circuit 52 to output an imaging result from the horizontal scanning circuit 52 to the control unit 73.

The image processing unit 81 acquires the image pickup result from the image pickup device 1 and executes various image processing. The image processing unit 81 uses the work memory 74 as a workspace to perform image processing on RAW data composed of pixel signals of each pixel to generate image data. The work memory 74 temporarily stores image data and the like when image processing is performed by the image processing unit 81.

In the present embodiment, the image processing unit 81 uses a first output signal based on the electric charge accumulated in the photoelectric conversion unit 3 and a second output signal based on the electric charge accumulated in the photoelectric conversion unit 4. Then, HDR processing is performed to widen the dynamic range. In this case, the imaging control unit 80 determines the period during which the electric charge is accumulated in the photoelectric conversion unit 3 (exposure period) and the period during which the electric charge is accumulated in the photoelectric conversion unit 4 (exposure period) in the period of one frame. , Set different lengths from each other.

For example, as shown in FIG. 3, the image pickup control unit 80 sets the exposure period for the photoelectric conversion unit 3 to be longer than the exposure period for the photoelectric conversion unit 4. Then, the image processing unit 81 uses the first output signal having a long exposure period (a large amount of exposure) to generate an image of a relatively dark dark portion of the imaging target. Further, the image processing unit 81 uses the second output signal having a short exposure period (the amount of exposure is small) to generate an image of a relatively bright bright part of the imaged object. The image processing unit 81 generates an image in which the dark portion is brightly expressed while suppressing the saturation of the bright portion by synthesizing the image of the dark portion and the image of the bright portion.

The image pickup apparatus 70 can switch between a mode in which HDR processing is performed to generate image data and a mode in which image data is generated without performing HDR processing. When the HDR processing is not performed, the image pickup control unit 80 sets the period in which the electric charge is accumulated in the photoelectric conversion unit 3 and the period in which the electric charge is accumulated in the photoelectric conversion unit 4 to different lengths. To do. The image processing unit 81 generates image data by adding, for example, a first output signal based on the electric charge stored in the photoelectric conversion unit 3 and a second output signal based on the electric charge stored in the photoelectric conversion unit 4. To do.

The image processing unit 81 can perform various image processing in addition to the HDR processing described above. For example, the image processing unit 81 generates an RGB image signal by performing color signal processing (color tone correction) on the signal obtained in the Bayer array. Further, the image processing unit 81 performs image processing such as white balance adjustment, sharpness adjustment, gamma correction, and gradation adjustment on the RGB image signal. Further, the image processing unit 81 performs a process of compressing in a predetermined compression format (JPEG format, MPEG format, etc.) as necessary. The control unit 73 can output the image data generated by the image processing unit 81 to the recording unit 77. Further, the control unit 73 can output the image data generated by the image processing unit 81 to the display unit 75.

The image processing unit 81 performs processing for detecting a main subject from image data, for example, as described in Japanese Patent Application Laid-Open No. 2010-16621 (US2010 / 0002940). Here, the "main subject" refers to a subject that is an object to be imaged and is a subject that is noticed by the user (photographer) or a subject that is estimated to be noticed by the user. The main subject is not limited to the case where only one is present in the image data, and may be present in a plurality of main subjects. Further, the image processing unit 81 detects the human body included in the image data as the main subject, as described in, for example, Japanese Patent Application Laid-Open No. 2010-16621 (US2010 / 0002940).

The parameters referred to when the image processing unit 81 performs image processing are included in the control parameters (imaging conditions). For example, parameters such as color signal processing (color tone correction), white balance adjustment, gradation adjustment, and compression rate are included in the control parameters. The signal read from the image sensor 1 changes according to the charge accumulation time and the like, and the parameters referred to when performing image processing also change according to the change in the signal.

The image processing unit 81 extracts frames at predetermined timings from a plurality of frames obtained from the image sensor 1 in time series. Alternatively, the image processing unit 81 discards the frames at predetermined timings among the plurality of frames obtained from the image sensor 1 in time series. As a result, the amount of data can be reduced, so that the load of post-stage processing can be reduced. Further, the image processing unit 81 calculates one or a plurality of frames to be interpolated between the frames based on the plurality of frames obtained from the image sensor 1 in time series. Then, the image processing unit 81 adds the calculated one or a plurality of frames between the frames. As a result, it is possible to play a moving image with smoother movement during moving image playback.

The focus detection unit 82 detects the focus information (information about focusing) of the image pickup optical system of the lens unit 71 by using the signal output from the image sensor 1. The focus detection unit 82 performs, for example, a phase difference type focus detection. When the focus detection unit 82 detects the focus information, the image pickup control unit 80 has, for example, a period in which the electric charge is accumulated in the photoelectric conversion unit 3 and the electric charge in the photoelectric conversion unit 4 as shown in FIG. 4 (D). Set the accumulation period to the same length. The focus detection unit 82 compares the first output signal based on the electric charge stored in the photoelectric conversion unit 3 with the second output signal based on the electric charge stored in the photoelectric conversion unit 4, and detects the focus information.

For example, when the difference between the level of the first output signal and the level of the second output signal is equal to or less than the threshold value, the focus detection unit 82 determines that the light receiving layer 12 of the image sensor 1 is in focus of the lens unit 71. .. Further, in the focus detection unit 82, when the difference between the level of the first output signal and the level of the second output signal is larger than the threshold value, and one level of the first output signal and the second output signal is higher than the other level. , It is determined that the focus is on the object surface side of the light receiving layer 12 (front pin). Further, in the focus detection unit 82, when the difference between the level of the first output signal and the level of the second output signal is larger than the threshold value and the other level of the first output signal and the second output signal is higher than one level. , It is determined that the focus is on the object surface side of the light receiving layer 12 (front pin). The focus detection unit 82 detects the amount of deviation between the focus of the lens unit 71 and the photoelectric conversion unit (light receiving layer 12) as focus information based on the difference between the level of the first output signal and the level of the second output signal. You may.

The focus detection unit 82 supplies the detected focus information to the image pickup control unit 80. The image pickup control unit 80 uses, for example, the focus information detected by the focus detection unit 82 to calculate the amount of drive by the lens drive unit 72 required to focus on the photoelectric conversion unit. The image pickup control unit 80 controls the focus of the lens unit 71 by sending a command to the lens drive unit 72 based on the calculated drive amount.

When the focus detection unit 82 detects the focus information, the image pickup control unit 80 sets the period in which the electric charge is accumulated in the photoelectric conversion unit 3 and the period in which the electric charge is accumulated in the photoelectric conversion unit 4 to be different lengths. It may be set. In this case, the focus detection unit 82 may detect the focus information based on the estimated value of the output signal level when the length of the exposure period is aligned between the photoelectric conversion unit 3 and the photoelectric conversion unit 4.

For example, the image pickup control unit 80 controls at least one of the amplification factor of the amplifier 55 and the amplification factor of the amplifier 56 by sending a command to the horizontal scanning circuit 52 of the image pickup element 1, and the photoelectric conversion unit 3 and the photoelectric conversion unit 4 A signal corresponding to a state in which the lengths of the exposure periods are the same may be output. For example, when the length of the exposure period for the photoelectric conversion unit 3 is twice the length of the exposure period for the photoelectric conversion unit 4, the imaging control unit 80 reduces the amplification factor of the amplifier 55 to half the amplification factor of the amplifier 56. You may adjust. Further, the control unit 73 (eg, the image processing unit 81, the focus detection unit 82) calculates the signal level corresponding to the state in which the photoelectric conversion unit 3 and the photoelectric conversion unit 4 have the same length of exposure period. You may.

The display unit 75 includes, for example, a liquid crystal display panel, and displays an image according to image data supplied from the control unit 73. The image data supplied from the control unit 73 is an image of an image showing settings such as image data based on an image captured by the image pickup element 1 (eg, still image, moving image, live view image), shooting conditions of the image pickup device 70, and the like. Data etc. The live view image is an image in which the image data generated by the image processing unit 81 is sequentially output to the display unit 75 and displayed on the display unit 75. The live view image is used for the user to confirm the image of the subject captured by the image sensor 1. The live view image is also called a through image or a preview image. The display unit 75 may include an input unit such as a touch panel.

The operation unit 76 is, for example, a release switch, a moving image switch, various operation switches, etc. operated by the user when the image pickup device 70 is a camera. The operation unit 76 outputs a signal corresponding to the operation by the user to the control unit 73. For example, the control unit 73 causes the image sensor 1 to execute an image pickup process in response to the operation of the operation unit 76. Further, the control unit 73 causes the display unit 75 to display an image showing shooting conditions and the like, and the operation unit 76 can receive an input from the user.

The recording unit 77 has a card slot into which a storage medium 83 such as a memory card can be mounted. The recording unit 77 stores the image data and various data generated by the image processing unit 81 in the storage medium 83 mounted in the card slot. For example, the control unit 73 stores the data based on the image captured by the image sensor 1 in the storage medium 83 in response to the operation of the operation unit 76. The control unit 73 may be stored in, for example, the image data generated by the HDR processing and the storage medium 83, or may be stored in the image data and the storage medium 83 generated without performing the HDR processing. Further, the control unit 73 transmits the first image data generated based on the first output signal based on the electric charge accumulated in the photoelectric conversion unit 3 and the second output signal based on the electric charge accumulated in the photoelectric conversion unit 4. The second image data originally generated may be stored in the storage medium 83 separately. The first image data may be image data generated without using the second output signal, and the second image data may be image data generated without using the first output signal. ..

Next, the operation when the image pickup apparatus 70 is a camera will be described. FIG. 9 is a flowchart showing an example of the operation of the image pickup apparatus 70. In step S1, the image pickup control unit 80 determines whether or not the still image mode is set.

First, the case where the still image mode is set will be described. When the image pickup control unit 80 determines in step S1 that the still image mode is set (step S1; Yes), the image pickup control unit 80 determines in step S2 whether or not the HDR mode is set. When the image pickup control unit 80 determines in step S2 that the HDR mode is set (step S2; Yes), the image pickup control unit 80 sets the exposure period in step S3. Here, it is assumed that the exposure period is set to be the same for the photoelectric conversion unit 3 and the photoelectric conversion unit 4 in the initial state. In step S3, the image pickup control unit 80 sets the length of the exposure period for the photoelectric conversion unit 3 to be different from the exposure period for the photoelectric conversion unit 4.

When the image pickup control unit 80 determines in step S2 that the HDR mode is not set (step S2; No), or when the process of step S3 is completed, the release switch (hereinafter, abbreviated as release SW) is abbreviated in step S4. ) Was operated or not. In step S4, when it is determined that the release SW is half-pressed (step S4; Yes), the image pickup control unit 80 performs an auto-exposure process (AE process) in step S5. Further, in step S4, when it is determined that the release SW is not operated (step S4; No), the imaging control unit 80 continues to monitor the operation of the release SW. In step S4, when the image pickup control unit 80 determines that the release SW is fully pressed, the image sensor 1 causes the image pickup element 1 to perform the image pickup process of step S8 described later.

After the process of step S5 is completed, the image pickup control unit 80 performs the autofocus process (AF process) in step S6. In step S5, the image pickup control unit 80 causes the image pickup device 1 to execute the image pickup process. Further, the focus detection unit 82 detects the focus information by using the first output signal based on the electric charge stored in the photoelectric conversion unit 3 and the second output signal based on the electric charge stored in the photoelectric conversion unit 4. Also. The image pickup control unit 80 controls the lens driving unit 72 and adjusts the focus of the lens unit 71 based on the focus information detected by the focus detection unit 82.

When the HDR mode is set, the imaging control unit 80 may display the live view image on the display unit 75 after the processing of step S5 is completed or after the processing of step S6 is completed. .. Further, the image pickup control unit 80 may accept an input requesting a change of at least one of the length of the exposure period for the photoelectric conversion unit 3 and the length of the exposure period for the photoelectric conversion unit 4. For example, the user can change the brightness setting in the HDR processing while viewing the live view image. Further, the imaging control unit 80 automatically changes at least one of the length of the exposure period for the photoelectric conversion unit 3 and the length of the exposure period for the photoelectric conversion unit 4 based on, for example, the distribution of pixel values in the live view image. You may. Further, the image pickup control unit 80 may execute such an exposure period change process while monitoring the operation of the release SW in, for example, step S4.

Further, the imaging control unit 80 may adjust at least one of the amplification factor of the amplifier 55 and the amplification factor of the amplifier 56 by using the result of the AE processing. For example, the image pickup control unit 80 sets the aperture so that a dark dark part of the image pickup target can be detected according to the longer exposure time of the photoelectric conversion unit 3 and the photoelectric conversion unit 4, and the exposure time is set under this condition. The amplification factor may be adjusted so that the output is not saturated with the shorter one.

After the process of step S6 is completed, the image pickup control unit 80 determines whether or not the release SW has been operated in step S7. When the image pickup control unit 80 determines in step S7 that the release SW is fully pressed (step S7; Yes), the image sensor 1 causes the image pickup device 1 to execute the image pickup process in step S8. Further, when the image pickup control unit 80 determines in step S7 that the release SW is not operated (step S7; No), the imaging control unit 80 continues to monitor the operation of the release SW. If it is determined in step S7 that the release SW has been pressed halfway, the image pickup control unit 80 continues to monitor the operation of the release SW after executing the AE process in step S5 and the AF process in step S6 again.

When the HDR mode is set, the imaging control unit 80 sets various control signals output from the vertical scanning circuit 51 to, for example, the waveform shown in FIG. 3 in step S8. Further, when the HDR mode is not set, the imaging control unit 80 sets various control signals output from the vertical scanning circuit 51 to, for example, the waveform shown in FIG. 4D in step S8. The image pickup control unit 80 causes the horizontal scanning circuit 52 to output the image pickup result after the period of one frame is completed.

After the processing of step S8 is completed, the control unit 73 causes the image processing unit 81 to generate image data according to the imaging result in step S9. When the HDR mode is set, the image processing unit 81 performs HDR processing in step S9 to generate image data. Further, when the HDR mode is not set, the image processing unit 81 generates image data without performing HDR processing in step S9. After the processing of step S9 is completed, the control unit 73 outputs the image data generated in step S8 to at least one of the storage unit 7 and the display unit 75 in step S10. In this way, a series of processes is completed.

Next, the case where the moving image mode is set (when the still image mode is not set) will be described. When set to the moving image mode, the imaging control unit 80 focuses on, for example, the center of the angle of view or a position designated by the user. When the image pickup control unit 80 determines in step S1 that the still image mode is not set (step S1; No), the image pickup control unit 80 determines in step S11 whether or not the HDR mode is set. When the image pickup control unit 80 determines in step S11 that the HDR mode is set (step S11; Yes), the image pickup control unit 80 sets the exposure period in step S12. Here, it is assumed that the exposure period is set to be the same for the photoelectric conversion unit 3 and the photoelectric conversion unit 4 in the initial state. In step S12, the image pickup control unit 80 sets the length of the exposure period for the photoelectric conversion unit 3 to be different from the exposure period for the photoelectric conversion unit 4.

When the image pickup control unit 80 determines in step S11 that the HDR mode is not set (step S11; No), or when the process of step S12 is completed, the moving image switch (step S13) requests the start of recording in step S13. Hereinafter, it is determined whether or not the operation of (abbreviated as moving image SW) has been performed. In step S13, when it is determined that the moving image SW has been operated (step S13; Yes), the image pickup control unit 80 causes the image pickup element 1 to execute the image pickup process in step S14. Further, in step S13, when the imaging control unit 80 determines that the moving image SW is not operated (step S13; No), the imaging control unit 80 continues to monitor the operation of the moving image SW.

When the HDR mode is set, the imaging control unit 80 sets various control signals output from the vertical scanning circuit 51 to, for example, the waveform shown in FIG. 3 in step S14. Further, when the HDR mode is not set, the imaging control unit 80 sets various control signals output from the vertical scanning circuit 51 to, for example, the waveform shown in FIG. 4D in step S14. The image pickup control unit 80 causes the horizontal scanning circuit 52 to output the image pickup result after the period of one frame is completed.

After the processing of step S14 is completed, the control unit 73 causes the image processing unit 81 to generate image data according to the imaging result. When the HDR mode is set, the image processing unit 81 performs HDR processing to generate image data. Further, when the HDR mode is not set, the image processing unit 81 generates image data without performing HDR processing. The control unit 73 outputs the image data generated by the image processing unit 81 to at least one of the storage unit 7 and the display unit 75.

When the process of step S14 is completed, the image pickup control unit 80 determines in step S15 whether or not the operation of the moving image SW requesting the end of recording has been performed. When it is determined in step S15 that the moving image SW has been operated (step S15; Yes), the image pickup control unit 80 ends a series of processes. Further, when it is determined in step S15 that the moving image SW is not operated (step S15; No), the imaging control unit 80 determines whether or not the imaging of a predetermined number of frames is completed in step S16. When the image pickup control unit 80 determines in step S16 that the imaging of a predetermined number of frames has not been completed (step S16; No), the image pickup control unit 80 returns to step S14 and causes the image sensor 1 to execute the image pickup process for the next frame.

When the image pickup control unit 80 determines in step S16 that the imaging of a predetermined number of frames is completed (step S15; Yes), the image pickup control unit 80 executes the AF process in step S17. In step S16, the image pickup control unit 80 causes the image pickup device 1 to execute the image pickup process. Further, the focus detection unit 82 detects the focus information by using the first output signal based on the electric charge stored in the photoelectric conversion unit 3 and the second output signal based on the electric charge stored in the photoelectric conversion unit 4. Also. The image pickup control unit 80 controls the lens driving unit 72 and adjusts the focus of the lens unit 71 based on the focus information detected by the focus detection unit 82.

In step S17, the imaging control unit 80 may execute the imaging process during the period between the frames of the output moving image, and execute the AF process using the result. Further, in step S17, the image pickup control unit 80 may execute the AF process using the image data of the output moving image frame. Further, the image pickup control unit 80 may execute the AF process using the outputs of the plurality of pixels P, or may use the signals output from some of the pixels P of the plurality of pixels P to perform AF. The process may be executed. When AF processing is executed using signals output from some pixels P, some pixels P may be fixed among a plurality of pixels P, or may be changed to different pixels for each AF processing. May be done. Further, when the HDR mode is set, for some pixels P used for AF processing, the photoelectric conversion unit 3 and the photoelectric conversion unit 4 may set the same exposure period length.

The predetermined number of frames is set according to the response speed of the lens driving unit 72 and the like. For example, when the time required for the lens driving unit 72 to achieve the driving amount specified by the imaging control unit 80 is 1 second and the frame rate is 30 FPS, the predetermined number of frames has a lower limit of 30 frames. It is set to any value. A default value is set for the predetermined number of frames, and the value can be changed to a value equal to or higher than the above lower limit value by the user's specification.

After the process of step S17 is completed, the image pickup control unit 80 returns to step S13 and causes the image pickup element 1 to perform the image pickup process of the next frame. In this way, the image pickup control unit 80 repeatedly executes the image pickup process in step S13 until it is determined in step S14 that the operation of the moving image SW requesting the stop of recording has been performed. Further, the image pickup control unit 80 executes the AF process in step S17 every time the image pickup process for a predetermined number of frames is performed.

The control unit 73 as described above may include, for example, a CPU (Central Processing Unit), and the CPU may execute at least one process according to an imaging control program. In this imaging control program, among the photoelectric conversion unit 3 and the photoelectric conversion unit 4 which are arranged in the first pixel in the control unit 73 and convert light into electric charges, the photoelectric conversion unit 3 gives the first period of one frame. The electric charge may be transferred and the electric charge may be transferred from the photoelectric conversion unit 4 to a second period different from the first period in the period of one frame. Further, the control unit 73 may include an arithmetic circuit such as an ASIC. For example, the image processing unit 81 may include an arithmetic circuit, and the arithmetic circuit may execute at least one process.

In the image pickup apparatus 70 according to the present embodiment, since a plurality of photoelectric conversion units are arranged in one pixel of the image pickup element 1, the dynamic range can be widened without lowering the resolution of the captured image. Further, since the transfer unit 21 for transferring the electric charge from the photoelectric conversion unit 3 and the transfer unit 41 for transferring the electric charge from the photoelectric conversion unit 4 can be independently controlled, the decrease in the frame rate can be suppressed. Further, the image pickup apparatus 70 may perform AF processing by using the first output signal based on the electric charge accumulated in the photoelectric conversion unit 3 and the second output signal based on the electric charge accumulated in the photoelectric conversion unit 4. it can.

The technical scope of the present invention is not limited to the above-described embodiment or modification. For example, one or more of the requirements described in the embodiments or modifications described above may be omitted. In addition, the requirements described in the above embodiments or modifications can be combined as appropriate.

1 Imaging element, 3 photoelectric conversion unit, 4 photoelectric conversion unit, P pixel, 21 transfer unit, 22 transfer control line, 23 charge voltage conversion unit, 24 reset unit, 25 reset control line, 41 transfer unit, 42 transfer control line, 43 Charge-voltage conversion unit, 44 reset unit, 45 reset control line, 55 amplifier, 56 amplifier, 60 photoelectric conversion unit, 61 photoelectric conversion unit, 62 photoelectric conversion unit, 63 transfer unit, 64 transfer control line, 65 transfer unit, 66 Transfer control line, 70 image pickup device, 80 image pickup control unit, 81 image processing unit, 82 focus detection unit, P pixel, P1 pixel, P2 pixel, TX1 control signal, TX2 control signal, TX3 control signal, TX4 control signal, RST1 control Signal, RST2 control signal

Claims (1)

The first photoelectric conversion unit and the second photoelectric conversion unit that convert the light transmitted through the microlens into electric charges and store them,
A first transfer unit that transfers electric charges from the first photoelectric conversion unit, and
A first transfer control line that transmits a control signal to the first transfer unit,
A second transfer unit that transfers electric charges from the second photoelectric conversion unit, and
A second transfer control line, which is provided separately from the first transfer control line and transmits a control signal to the second transfer unit, is provided.
The first transfer control line and the second transfer control line are provided on the side opposite to the incident side of the light transmitted through the microlens with respect to the first photoelectric conversion unit and the second photoelectric conversion unit. ..

Images