WO2024154623A1

WO2024154623A1 - Photodetection device and electronic device

Info

Publication number: WO2024154623A1
Application number: PCT/JP2024/000277
Authority: WO
Inventors: 雄馬小野
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2023-01-18
Filing date: 2024-01-10
Publication date: 2024-07-25

Abstract

[Problem] To enable both a dynamic range expansion and a frame rate increase. [Solution] This photodetection device comprises: a plurality of pixels each having a photoelectric conversion element that stores charge corresponding to the light intensity of incident light. The plurality of pixels each have: a charge-to-voltage conversion part that converts the charge stored in the photoelectric conversion element into a voltage; a first retention part that retains the charge stored in the photoelectric conversion element; a second retention part that retains, alternately with the first retention part, the charge stored in the photoelectric conversion element; and a third retention part that, using the same timing as another pixel, retains the charge stored in the photoelectric conversion element.

Description

Photodetection device and electronic device

This disclosure relates to a light detection device and an electronic device.

An imaging device has been proposed that has a global shutter function that exposes all pixels simultaneously when capturing an image of a moving subject (Patent Document 1). In Patent Document 1, in order to expand the dynamic range, a holding section is provided that holds charge that overflows from the photoelectric conversion section, separate from the holding section that holds charge for the global shutter.

JP 2020-178163 A

However, the method of Patent Document 1 cannot hold the charge that has overflowed from the photoelectric conversion unit while the charge is being read out from the holding unit. This causes the problem that exposure cannot be performed during the charge readout period, resulting in a drop in the frame rate.

This disclosure provides a photodetector and electronic device that can achieve both an expanded dynamic range and an improved frame rate.

In order to solve the above problems, according to the present disclosure, there is provided a liquid crystal display device comprising a plurality of pixels each having a photoelectric conversion element that accumulates a charge according to the amount of incident light,
Each of the plurality of pixels is
a charge-voltage converter for converting the charge stored in the photoelectric conversion element into a voltage;
a first holding unit that holds the charge accumulated in the photoelectric conversion element;
a second holding unit that holds the charges accumulated in the photoelectric conversion element alternately with the first holding unit;
a third holding unit that holds the charge accumulated in the photoelectric conversion element at the same timing as other pixels;
A light detection device is provided.

While either the first holding unit or the second holding unit transfers charge to the charge-voltage conversion unit, the other of the first holding unit or the second holding unit may hold the charge accumulated in the photoelectric conversion element.

Each of the plurality of pixels may alternately hold charge in the first holding section or the second holding section on a frame-by-frame basis, and may also hold charge in the third holding section for each frame.

each of the plurality of pixels outputs, for each frame, a pixel signal of a reset level and a pixel signal corresponding to the charge held in the first holding unit or the second holding unit and the charge held in the third holding unit;
The pixel signal of the reset level and the pixel signal according to the charge may be output at different timings.

The saturation charge amount of the first holding unit and the second holding unit may be greater than the saturation charge amount of the third holding unit.

The charge retention period of the first and second retention units per frame may be longer than the charge retention period of the third retention unit.

Each of the plurality of pixels is
a first transfer control unit that controls the transfer of charges from the photoelectric conversion element to the first holding unit;
a second transfer control unit that controls the transfer of charges from the first holding unit to the charge-voltage conversion unit;
a third transfer control unit that controls the transfer of charges from the photoelectric conversion element to the second holding unit;
The pixel may further include a fourth transfer control unit that controls the transfer of charges from the second holding unit to the charge-voltage conversion unit.

The first transfer control unit or the third transfer control unit may alternately transfer the charge accumulated in the photoelectric conversion element to the first holding unit or the second holding unit on a frame-by-frame basis.

The second transfer control unit or the fourth transfer control unit may alternately transfer the charge held in the first holding unit or the second holding unit to the charge-voltage conversion unit on a frame-by-frame basis.

When the first transfer control unit transfers charges from the photoelectric conversion element to the first holding unit, a negative bias voltage is supplied to a gate of the third transfer control unit,
When the third transfer control unit transfers the charges from the photoelectric conversion elements to the second holding units, a negative bias voltage may be supplied to the gates of the first transfer control units.

Each of the plurality of pixels is
a fifth transfer control unit that controls the transfer of charges from the photoelectric conversion element to the third holding unit;
The pixel may further include a sixth transfer control unit that controls the transfer of charges from the third holding unit to the charge-voltage conversion unit.

the fifth transfer control unit transfers charges from the photoelectric conversion element to the third holding unit at the same timing as other pixels for each frame;
The sixth transfer control unit may transfer the charge from the third holding unit to the charge-voltage conversion unit for each frame.

The fifth transfer control unit may transfer charges from the photoelectric conversion element to the third holding unit after charges are transferred from the photoelectric conversion element to the first holding unit or the second holding unit for each frame.

Each of the plurality of pixels is
a fourth holding unit that holds the charge accumulated in the photoelectric conversion element at the same timing as other pixels;
Each of the plurality of pixels may alternately hold charge in the first holding unit or the second holding unit on a frame-by-frame basis, and may alternately hold charge in the third holding unit or the fourth holding unit on a frame-by-frame basis.

each of the plurality of pixels outputs, for each frame, a pixel signal of a reset level and a pixel signal corresponding to the charges held in the first holding unit and the third holding unit or a pixel signal corresponding to the charges held in the second holding unit or the fourth holding unit;
The pixel signal of the reset level and the pixel signal according to the charge may be output at different timings.

The saturated charge amount of the first holding unit and the second holding unit may be greater than the saturated charge amount of the third holding unit and the fourth holding unit.

The charge retention period of the first and second retention units per frame may be longer than the charge retention period of the third and fourth retention units.

Each of the plurality of pixels is
a seventh transfer control unit that controls the transfer of charges from the photoelectric conversion element to the third holding unit;
an eighth transfer control unit that controls the transfer of charges from the third holding unit to the first holding unit;
a ninth transfer control unit that controls the transfer of charges from the first holding unit to the charge-voltage conversion unit;
a tenth transfer control unit that controls the transfer of charges from the photoelectric conversion element to the fourth holding unit;
an eleventh transfer control unit that controls the transfer of charges from the fourth holding unit to the second holding unit;
The pixel may further include a twelfth transfer control unit that controls the transfer of charges from the second holding unit to the charge-voltage conversion unit.

the seventh transfer control unit or the tenth transfer control unit alternately transfers the charges accumulated in the photoelectric conversion element to the third holding unit or the fourth holding unit on a frame basis;
the eighth transfer control unit or the eleventh transfer control unit alternately transfers charges from the third holding unit or the fourth holding unit to the first holding unit or the second holding unit on a frame-by-frame basis;
The ninth transfer control unit or the twelfth transfer control unit may alternately transfer the charges held in the first holding unit or the second holding unit to the charge-voltage conversion unit on a frame-by-frame basis.

The present disclosure also provides a method for detecting a light-generating particle beam from a light-generating device, comprising:
A processing unit that processes pixel data output from the light detection device,
The light detection device includes:
The image sensor includes a plurality of pixels each having a photoelectric conversion element that accumulates an electric charge according to the amount of incident light,
Each of the plurality of pixels is
a charge-voltage converter for converting the charge stored in the photoelectric conversion element into a voltage;
a first holding unit that holds the charge accumulated in the photoelectric conversion element;
a second holding unit that holds the charges accumulated in the photoelectric conversion element alternately with the first holding unit;
a third holding unit that holds the charge accumulated in the photoelectric conversion element at the same timing as other pixels;
An electronic device is provided.

FIG. 1 is a block diagram of an electronic device according to a first embodiment of the present disclosure. 1 is a block diagram illustrating a configuration example of a light detection device according to a first embodiment of the present disclosure. FIG. 2 is a diagram showing a configuration of a pixel and a peripheral portion according to the first embodiment of the present disclosure. 1 is a diagram showing a first example of a layered structure of a light detection device. FIG. 13 is a diagram showing a second example of a laminated structure of the photodetector. 4 is a timing chart showing an imaging operation by the electronic device. FIG. 2 is a diagram showing an exposure period of a first frame by a pixel according to the first embodiment of the present disclosure. FIG. 2 is a diagram showing a readout period of a first frame by a pixel according to the first embodiment of the present disclosure. FIG. 2 is a diagram showing an exposure period of a second frame after a first frame is read out by a pixel according to the first embodiment of the present disclosure. FIG. 11 is a diagram showing a readout period of a second frame by the pixel of the first embodiment of the present disclosure. 4 is a timing chart showing an imaging operation of a pixel according to the first embodiment of the present disclosure. FIG. 1 is a diagram illustrating the potential barriers of two OF memories. FIG. 1 is a diagram showing a configuration of a pixel and a peripheral portion of a comparative example. FIG. 13 is a diagram showing an exposure period for a pixel of a comparative example. FIG. 13 is a diagram showing a readout period by a pixel of a comparative example. FIG. 11 is a diagram showing a configuration of a pixel and a peripheral portion according to a second embodiment of the present disclosure. FIG. 11 is a diagram showing an exposure period of a first frame by a pixel according to a second embodiment of the present disclosure. 1 is a diagram showing a readout period of a first frame and an exposure period of a second frame by a pixel according to a first embodiment of the present disclosure; 10 is a timing chart showing pixel control in the second embodiment of the present disclosure. 1 is a diagram illustrating an example of a schematic configuration of an endoscopic surgery system. 2 is a block diagram showing an example of the functional configuration of a camera head and a CCU. FIG.

Below, embodiments of the light detection device and electronic device are described with reference to the drawings. The following description focuses on the main components of the light detection device and electronic device, but the light detection device and electronic device may have components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

First Embodiment
1 is a block diagram of an electronic device 1 according to a first embodiment of the present disclosure. The electronic device 1 captures image data and includes an imaging lens 11, a light detection device 2, a processing unit 3, and a control unit 4. The electronic device 1 is assumed to be, for example, a camera mounted on an industrial robot or an in-vehicle camera, but the specific use and configuration of the electronic device 1 are arbitrary.

The imaging lens 11 collects the incident light and guides it to the photodetector 2. The photodetector 2 is, for example, a CMOS (Complementary Metal) image sensor, which performs photoelectric conversion on the incident light to capture image data. The photodetector 2 can also generate video data by continuously capturing image data. The image data output from the photodetector 2 is input to the processor 3 via the transmission line 12. The processor 3 performs predetermined image processing on the image data output from the photodetector 2.

The electronic device 1 may include a recording unit 5. The recording unit 5 records image data from the light detection device 2. The recording unit 5 may be disposed in a server or the like connected via a network.

The control unit 4 controls the imaging timing of the light detection device 2 via the control line 13. For example, the control unit 4 controls the start and end of imaging by the light detection device 2 in response to the operation of a shutter operating member (not shown).

FIG. 2 is a block diagram showing an example configuration of the photodetector 2 in the first embodiment of the present disclosure. The photodetector 2 includes a pixel array section 20, a vertical drive circuit 21, a system control circuit 22, and a signal processing section 23.

The pixel array section 20 has pixels 30 arranged in multiples in a first direction X and a second direction Y. In this specification, the left-right (horizontal) direction in FIG. 2 is called the first direction X, and the up-down (vertical) direction in FIG. 2 is called the second direction Y. A group of pixels 30 arranged in multiples along the first direction X is called a pixel row, and a group of pixels 30 arranged in multiples along the second direction Y is called a pixel column.

Each pixel 30 has a photoelectric conversion element that generates an electric charge according to the amount of incident light. A pixel circuit (not shown in FIG. 1) connected to pixel 30 generates a pixel signal Vimg based on the electric charge of the photoelectric conversion element.

The vertical drive circuit 21 is composed of a shift register, an address decoder, etc. The vertical drive circuit 21 drives each pixel 30 of the pixel array section 20, either all pixels at once or on a pixel row basis. The vertical drive circuit 21 discharges electric charge and reads out signals from each pixel 30 and pixel circuit.

When discharging electric charge, unnecessary electric charge is swept out (reset) from the photoelectric conversion element of pixel 30. This allows the photoelectric conversion element in pixel 30 to start a new exposure. The operation of discarding the charge of the photoelectric conversion element and starting a new exposure (starting the accumulation of electric charge) is also called an electronic shutter operation.

The photodetector 2 performs a global shutter operation, which is an electronic shutter operation that is performed simultaneously on all pixels. A feature of the global shutter operation is that there is no deviation in the exposure timing for each pixel.

When reading out a signal, a pixel signal Vimg based on the charge accumulated in the photoelectric conversion element in the pixel 30 is read out from the pixel circuit. The period from the electronic shutter operation to the signal readout is also called the exposure period. When reading out a signal, a pixel signal Vimg corresponding to the amount of light incident on the pixel 30 during the exposure period is read out from the pixel circuit.

The system control circuit 22 is composed of a timing generator that generates various timing signals. Based on the various timing signals, the system control circuit 22 controls the read and discharge timing of the vertical drive circuit 21, and controls the drive of the signal processing unit 23.

The signal processing unit 23 receives the pixel signal Vimg from each pixel 30 and pixel circuit in the pixel array unit 20. The signal processing unit 23 performs predetermined signal processing on the pixel signal Vimg. The predetermined signal processing includes, for example, analog-to-digital conversion of the pixel signal Vimg and noise removal processing superimposed on the pixel signal Vimg.

Correlated double sampling (CDS), for example, is used as a noise removal process for the pixel signal Vimg. The signal processing unit 23 compares the signal level of the pixel signal Vimg (hereinafter also referred to as the D-phase signal) that corresponds to the amount of incident light with the signal level of a pixel signal (hereinafter also referred to as the P-phase signal) that is at a reset level and does not depend on the amount of incident light. This removes noise (also referred to as kTC noise) that is superimposed on the pixel signal Vimg.

FIG. 3 is a diagram showing the configuration of a pixel 30 and its peripheral portion in the first embodiment of the present disclosure. The pixel 30 shown in FIG. 3 outputs a pixel signal Vimg based on the illuminance of incident light. The pixel 30 includes a photoelectric conversion element (PD) 31, a transfer unit 32, and a charge-voltage conversion unit 33. The photoelectric conversion element 31 is, for example, a photodiode (PD) and has an anode and a cathode. Either the anode or the cathode (for example, the cathode) is connected to the transfer unit 32, and the other (for example, the anode) is connected to a predetermined reference voltage (VRLD) node such as a ground voltage. The charge-voltage conversion unit 33 is also called a floating diffusion (FD). The charge resulting from photoelectric conversion and stored in the photoelectric conversion element 31 is transferred to the charge-voltage conversion unit 33 via the transfer unit 32.

The amplification transistor Q1, the selection transistor Q2, and the reset transistor Q3 are arranged on the periphery of the pixel 30. Note that the pixel 30 may be interpreted as including at least a part of the amplification transistor Q1, the selection transistor Q2, and the reset transistor Q3 in FIG. 3. In this specification, the amplification transistor Q1, the selection transistor Q2, and the reset transistor Q3 are referred to as a pixel circuit, and each transistor in the pixel circuit is collectively referred to as a pixel transistor.

In this specification, an example will be described in which the amplification transistor Q1, the selection transistor Q2, and the reset transistor Q3 are configured, for example, with NMOS (N-channel Metal-Oxide-Semiconductor) transistors. However, the conductivity type of each transistor exemplified here is arbitrary. Any of the above transistors may be configured, for example, with PMOS (P-channel Metal-Oxide-Semiconductor) transistors.

The transfer unit 32 has a GS transistor Q4 (fifth transfer control unit), a GS transistor Q5 (sixth transfer control unit), an OF transistor Q6 (first transfer control unit), an OF transistor Q7 (second transfer control unit), an OF transistor Q8 (third transfer control unit), and an OF transistor Q9 (fourth transfer control unit). The GS transistors Q4 and Q5 are cascode-connected between the photoelectric conversion element 31 and the charge-voltage conversion unit 33. Similarly, the OF transistors Q6 and Q7 are cascode-connected between the photoelectric conversion element 31 and the charge-voltage conversion unit 33. Similarly, the OF transistors Q8 and Q9 are cascode-connected between the photoelectric conversion element 31 and the charge-voltage conversion unit 33.

A GS memory (third holding unit) Mg is disposed at the connection node between GS transistor Q4 and GS transistor Q5. The GS memory Mg holds the charge transferred from the photoelectric conversion element 31 at the same timing as the other pixels 30 in conjunction with the global shutter operation. The charge in the GS memory Mg is also transferred to the charge-voltage conversion unit 33 during the signal readout period of the pixel 30.

GS transistor Q4 transfers charge from photoelectric conversion element 31 to GS memory Mg. When signal GS1 input to the gate is at high level, GS transistor Q4 is turned on and transfers charge from photoelectric conversion element 31 to GS memory Mg.

GS transistor Q5 transfers charge from GS memory Mg to charge-voltage conversion unit 33. When signal GS2 input to the gate is at high level, GS transistor Q5 is turned on and transfers charge from GS memory Mg to charge-voltage conversion unit 33.

The charge that exceeds the saturation charge amount of the photoelectric conversion element 31 and overflows from the photoelectric conversion element 31 is also called overflow charge.

An OF memory (first holding unit) M1 is disposed at the connection node between the OF transistor Q6 and the OF transistor Q7. The OF memory M1 holds the charge accumulated in the photoelectric conversion element 31. More specifically, the OF memory M1 holds the overflow charge during the exposure period. The charge held by the OF memory M1 is transferred to the charge-voltage conversion unit 33 during the readout period of the signal from the pixel 30. As will be described later, the OF transistor Q6 may be controlled so that no charge is held in the OF memory M1 when the amount of light incident on the pixel 30 is low.

The OF transistor Q6 controls the transfer of charge from the photoelectric conversion element 31 to the OF memory M1. When the signal OF1 input to the gate of the OF transistor Q6 is at a high level, the OF transistor Q6 transfers the charge stored in the photoelectric conversion element 31 to the OF memory M1. The voltage level of the signal OF1 may be set to an intermediate voltage level between a high level voltage and a low level voltage. When the signal OF1 is at an intermediate voltage level, the OF transistor Q6 can transfer only the charge that exceeds the saturation charge amount of the photoelectric conversion element 31 to the OF memory M1.

The OF transistor Q7 transfers charge from the OF memory M1 to the charge-voltage conversion unit 33. When the signal OF2 input to the gate of the OF transistor Q7 is at a high level, the OF transistor Q7 is turned on and transfers the charge of the OF memory M1 to the charge-voltage conversion unit 33.

An OF memory (second holding unit) M2 is disposed at the connection node between OF transistor Q8 and OF transistor Q9. The OF memory M2 holds the charge transferred from the photoelectric conversion element 31, alternating with the OF memory M1.

In this embodiment, the saturation charge amount of the OF memories M1 and M2 is set to be larger than the saturation charge amount of the GS memory Mg. This allows the OF memories M1 and M2 to hold charges that cannot be stored in the GS memory Mg, expanding the dynamic range. Note that the saturation charge amount of the OF memories M1 and M2 may be equal to or smaller than the saturation charge amount of the GS memory Mg.

OF memories M1 and M2 alternately hold and read charges. That is, while OF memory M1 is transferring charges to charge-voltage conversion unit 33, charges are transferred from photoelectric conversion element 31 to OF memory M2. Also, while OF memory M2 is transferring charges to charge-voltage conversion unit 33, charges are transferred from photoelectric conversion element 31 to OF memory M1.

The OF transistor Q8 transfers electric charge from the photoelectric conversion element 31 to the OF memory M2. When the signal OF3 input to the gate of the OF transistor Q8 is at a high level, the OF transistor Q8 transfers the electric charge stored in the photoelectric conversion element 31 to the OF memory M2. The voltage level of the signal OF3 may be an intermediate voltage level between a high level voltage and a low level voltage, similar to the signal OF1.

The OF transistor Q9 transfers charge from the OF memory M2 to the charge-voltage conversion unit 33. When the signal OF4 input to the gate of the OF transistor Q9 is at a high level, the OF transistor Q9 is turned on and transfers the charge of the OF memory M2 to the charge-voltage conversion unit 33.

During the signal readout period, charge is transferred from the GS memory Mg to the charge-voltage conversion unit 33, and charge is transferred from the OF memory M1 or M2. As a result, the charge-voltage conversion unit 33 becomes a voltage according to the charge accumulated in the photoelectric conversion element 31.

The gate of the amplifier transistor Q1 is at the same voltage as the charge-voltage conversion unit 33, and is used as the input of the source follower circuit. The drain of the amplifier transistor Q1 is connected to the node of the high-voltage power supply VDD, and the source is connected to the selection transistor Q2. The source voltage of the amplifier transistor Q1 changes according to the voltage of the charge-voltage conversion unit 33.

The selection transistor Q2 controls the reading of signals from the pixels 30. A selection signal SEL is input to the gate of the selection transistor Q2. The selection transistor Q2 is turned on when the selection signal SEL is at a high level. As a result, a pixel signal Vimg with a voltage level according to the voltage of the charge-voltage conversion unit 33 is output from the source of the selection transistor Q2 to the downstream signal processing unit 23, etc.

The reset transistor Q3 controls the discharge of charge from pixel 30. The source of the reset transistor Q3 is connected to the charge-voltage conversion unit 33, and the drain is connected to the node of the high-voltage power supply VDD. A reset signal RST is input to the gate of the reset transistor Q3. The reset transistor Q3 is turned on when the reset signal RST is at a high level. This causes the charge in the charge-voltage conversion unit 33 to be discharged to the node of the high-voltage power supply VDD, resetting the charge-voltage conversion unit 33. Resetting the charge-voltage conversion unit 33 makes the pixel 30 available for the next exposure.

The photodetector 2 is composed of, for example, two stacked chips. FIG. 4A is a diagram showing a first example of the stacked structure of the photodetector 2. This photodetector 2 comprises a pixel chip 41 and a logic chip 42 stacked on the pixel chip 41. These chips are bonded by vias or the like. Note that these chips may be bonded by Cu-Cu bonding or bumps in addition to vias.

In the pixel chip 41, for example, a plurality of pixels 30 in the pixel array section 20 and a pixel transistor for each pixel 30 are arranged. In the logic chip 42, for example, a vertical drive circuit 21, a system control circuit 22, and a signal processing section 23 are arranged. In the first example of FIG. 4A, it is interpreted that the pixel transistor is included in the pixel 30.

The photodetector 2 may be composed of three or more stacked chips. FIG. 4B is a diagram showing a second example of the stacked structure of the photodetector 2. In the photodetector 2a of FIG. 4B, a first pixel chip 43 and a second pixel chip 44 are stacked instead of the pixel chip 41 of FIG. 4A. The pixels 30 are arranged on the first pixel chip 43. The pixel transistors for each pixel 30 are arranged on the second pixel chip 44.

In the photodetector 2a in FIG. 4B, pixel transistors are not arranged on the first pixel chip 43, but are arranged on the second pixel chip 44. This allows the photodetector 2a to increase the proportion of the area of the photoelectric conversion element 31 in the chip area, improving sensitivity and enabling the chip to be miniaturized.

FIG. 5 is a timing chart showing the imaging operation by the photodetector 2. The left-right direction in FIG. 5 is the time axis direction t. FIG. 5 also shows an example in which the pixel signal Vimg is read out for each pixel row in the pixel array unit 20, which is arranged in a plurality of rows in the second direction Y.

The imaging operation by the light detection device 2 is performed in units of frames, synchronized with a predetermined synchronization signal. One frame includes a reset period, an exposure period, and a readout period.

In each frame, a global shutter operation is performed. First, at time Trst, electric charges are simultaneously swept out (reset) from all pixels 30 in the pixel array section 20. The reset enables the pixels 30 to be exposed.

During the exposure period, exposure of all pixels 30 starts simultaneously at time Texp, and exposure of all pixels 30 ends simultaneously at time Trd. During the readout period, the pixel signal Vimg is read out sequentially for each pixel row, starting from time Trd.

As described above, the global shutter operation of the photodetection device 2 is performed in synchronization with all pixels 30. Furthermore, the global shutter operation is performed in units of frames. The photodetection device 2 is also capable of capturing video by continuously capturing images of multiple frames.

FIGS. 6A to 6D are diagrams showing the imaging operation of pixel 30 according to the first embodiment of the present disclosure. FIG. 6A is a diagram showing the imaging operation during the exposure period of the first frame. During the exposure period of the first frame, photoelectric conversion element 31 accumulates charge according to incident light. During the exposure period of the first frame, the charge accumulated in photoelectric conversion element 31 is transferred to GS memory Mg and OF memory M1, and is not transferred to OF memory M2.

FIG. 6B shows the readout period of the first frame. During the readout period of the first frame, the charge e in the GS memory Mg and the OF memory M1 is transferred to the charge-voltage converter 33.

Even during the readout period of the first frame, light is incident on the pixel 30, and the photoelectric conversion element 31 accumulates electric charge due to photoelectric conversion. The electric charge accumulated in the photoelectric conversion element 31 during the readout period is transferred to the OF memory M2, and is not transferred to the OF memory M1. This is because the OF memory M1 is transferring electric charge to the charge-voltage conversion unit 33.

After the readout period of the first frame shown in FIG. 6B, the charges in the GS memory Mg and the OF memory M1 are reset. This makes the GS memory Mg and the OF memory M1 ready to transfer new charges.

FIG. 6C shows the exposure period of the second frame after the first frame is read out. The charge accumulated in the photoelectric conversion element 31 during the exposure period of the second frame is transferred to the GS memory Mg and the OF memory M2, but is not transferred to the OF memory M1. In this way, the OF memories M1 and M2 are used alternately for each frame.

FIG. 6D shows the readout period of the second frame. During the readout period of the second frame, the charge e of the GS memory Mg and the OF memory M2 is transferred to the charge-voltage conversion unit 33. Furthermore, the charge accumulated in the photoelectric conversion element 31 during the readout period of the second frame is transferred to the OF memory M1, and not to the OF memory M2. This is because the OF memory M2 is transferring charge to the charge-voltage conversion unit 33.

From the exposure period of the third frame onwards, the operations shown in Figures 6A to 6D are repeated.

As shown in Figures 6A to 6D, while either OF memory M1 or M2 transfers charge e to the charge-voltage conversion unit 33, the other OF memory M1 or M2 holds the charge e accumulated in the photoelectric conversion element 31. The pixel 30 of the first embodiment of the present disclosure can read charge and perform exposure in parallel by alternately using the OF memory M1 or M2 to read or transfer charge e. This allows the photodetection device 2 to increase the frame rate.

FIG. 7 is a timing chart showing the imaging operation of pixel 30 in the first embodiment of the present disclosure. Times t1 to t24 shown in FIG. 7 include initialization, the exposure period and readout period of the first frame, the exposure period and readout period of the second frame, and the exposure period of the third frame.

Between time t1 and t2, the pixels 30 are initialized before imaging. Specifically, the reset signal RST, signals GS1 and GS2, signals OF1 and OF2, and signals OF3 and OF4 are all set to a high level for a certain period of time at the same timing. As a result, the charges in the photoelectric conversion element 31, GS memory Mg, OF memory M1, OF memory M2, and charge-voltage conversion unit 33 are discharged to the node of the high-voltage power supply VDD, and the charge-voltage conversion unit 33 becomes the reset level voltage.

Times t3 to t5 are the exposure period of the first frame. As shown in FIG. 6A, charge is accumulated in the photoelectric conversion element 31 during the exposure period of the first frame. Between times t3 and t4, signal OF1 goes high and the OF transistor Q6 is turned on, causing the charge accumulated in the photoelectric conversion element 31 to be transferred to the OF memory M1.

From time t4 to t5, the signal OF1 goes low and the signal GS1 goes high. This turns off the OF transistor Q6 and turns on the GS transistor Q4, transferring the charge in the photoelectric conversion element 31 to the GS memory Mg.

As mentioned above, the OF memory M1 has a larger saturated charge amount than the GS memory Mg. Therefore, in FIG. 7, the period (time t3 to t4) during which the charge accumulated in the photoelectric conversion element 31 is held in the OF memory M1 is longer than the period (time t4 to t5) during which the charge is transferred from the photoelectric conversion element 31 to the GS memory Mg. The same is true for the OF memory M2 described below. In other words, the charge holding period of the OF memory M1 or OF memory M2 in each frame is longer than the charge holding period of the GS memory Mg. This allows the OF memories M1 and M2 to store more charge than the GS memory Mg.

Times t5 to t13 are the readout period of the first frame. As shown in FIG. 6B, during the readout period of the first frame, charges are read out from the GS memory Mg and the OF memory M1. Between times t6 and t7, the signals GS2 and OF2 go to low level, and the selection signal SEL goes to high level. This causes a pixel signal (P-phase signal) corresponding to the reset level of the charge-voltage converter 33 to be read out.

Between time t8 and t9, signal GS2 and selection signal SEL go to high level. When GS transistor Q5 is turned on, the charge in GS memory Mg is transferred to charge-voltage converter 33. As a result, a pixel signal (D-phase signal) corresponding to the charge obtained by adding the charge in GS memory Mg to the charge in charge-voltage converter 33 is read out.

Between time t10 and t11, the signal OF2 and the selection signal SEL go to high level. The OF transistor Q7 is turned on, and the charge in the OF memory M1 is transferred to the charge-voltage converter 33. As a result, a pixel signal (D-phase signal) corresponding to the charge obtained by adding the charge in the OF memory M1 to the charge in the GS memory Mg is read out from the charge-voltage converter 33.

From time t11 to t12, the signal OF2, the signal GS2, and the reset signal RST go to high level, and the charges in the GS memory Mg, the OF memory M1, and the charge-voltage conversion unit 33 are discharged to the node of the high-voltage power supply VDD. This causes the charge-voltage conversion unit 33 to go to the reset level voltage.

Between time t12 and t13, the signal OF2, the signal GS2, and the selection signal SEL go to high level, and a pixel signal (P-phase signal) corresponding to the reset level of the charge-voltage conversion unit 33 is read out.

The period from time t5 to t13 is the readout period for the first frame and also the exposure period for the second frame. During the exposure period for the second frame, a high-level signal OF3 is input, and the charge accumulated in the photoelectric conversion element 31 is held in the OF memory M2.

Times t13 to t15 are the exposure period for the second frame after the first frame is read out. As shown in FIG. 6C, from time t13 to t14, the charge of the photoelectric conversion element 31 is held in the OF memory M2. From time t14 to t15, the signal OF3 goes low and the signal GS1 goes high. This causes the charge of the photoelectric conversion element 31 to be transferred to the GS memory Mg.

Times t15 to t23 are the readout period of the second frame. As shown in FIG. 6D, during the readout period of the second frame, charges are read out from the GS memory Mg and the OF memory M2 by the same control as during the readout period of the first frame.

That is, from time t16 to t17, the selection signal SEL goes high, and a pixel signal (P-phase signal) corresponding to the reset level of the charge-voltage conversion unit 33 is read out. From time t18 to t19, the signal GS2 and the selection signal SEL go high, and a pixel signal (D-phase signal) corresponding to the charge obtained by adding the charge of the GS memory Mg to the charge of the charge-voltage conversion unit 33 is read out.

From time t20 to t21, signal OF4 and selection signal SEL go high, and a pixel signal (D-phase signal) corresponding to the charge obtained by adding the charge in OF memory M2 to the charge in charge-voltage conversion unit 33 is read out. From time t21 to t22, signal OF4, signal GS2, and reset signal RST go high, and the charges in GS memory Mg, OF memory M2, and charge-voltage conversion unit 33 are swept out. From time t22 to t23, signal OF4, signal GS2, and selection signal SEL go high, and a pixel signal (P-phase signal) corresponding to the reset level of charge-voltage conversion unit 33 is read out.

Times t15 to t24 are the readout period for the second frame and the exposure period for the third frame. During the exposure period for the third frame, signal OF1 goes high, causing the charge stored in photoelectric conversion element 31 to be transferred to OF memory M1.

As shown in FIG. 7, the transfer unit 32 transfers charges from the photoelectric conversion element 31 to the charge-voltage conversion unit 33 by controlling each transistor. That is, the OF transistor Q6 or Q8 transfers the charges stored in the photoelectric conversion element 31 to the OF memory M1 or M2 alternately on a frame-by-frame basis. The OF transistor Q7 or Q9 transfers the charges held in the OF memory M1 or M2 alternately on a frame-by-frame basis to the charge-voltage conversion unit 33. In addition, the GS transistor Q4 transfers charges from the photoelectric conversion element 31 to the GS memory Mg after the charges have been transferred from the photoelectric conversion element 31 to the OF memory M1 or M2 for each frame. The GS transistor Q5 transfers charges from the GS memory Mg to the charge-voltage conversion unit 33 for each frame.

The control in FIG. 7 is performed for each of the multiple pixels 30 in the pixel array section 20. That is, each of the multiple pixels 30 alternately stores charge in the OF memory M1 or M2 on a frame-by-frame basis, and also stores charge in the GS memory Mg for each frame. Note that the transfer of charge from the GS transistor Q4 is performed at the same timing as the other pixels 30 in the pixel array section 20.

Also, as shown in FIG. 7, each of the multiple pixels 30 in the pixel array unit 20 outputs a pixel signal Vimg of the reset level for each frame. Also, each of the multiple pixels 30 outputs a pixel signal Vimg for each frame that corresponds to the charge held in the OF memory M1 or M2 and the charge held in the GS memory Mg at a timing different from the reset level pixel signal Vimg.

As shown in Figure 7, driving in which two or more frames are executed in parallel with the exposure periods (and readout periods) shifted from each other is also called pipeline driving.

In FIG. 7, an example is shown in which OF transistors Q6 and Q8 are switched on and off when transferring charge from photoelectric conversion element 31 to OF memories M1 and M2, but the height of the potential barrier of OF memories M1 and M2 may also be controlled.

FIG. 8 is a diagram explaining the potential barrier of OF memories M1 and M2. The potential barrier can be made higher by setting the signal OF1 or OF3 input to the gate of either OF transistor Q6 or Q8 to a negative bias voltage. FIG. 8 shows an example in which the signal OF3 input to the gate of OF transistor Q8 is set to a negative bias voltage. In the case of FIG. 8, the charge accumulated in the photoelectric conversion element 31 is not transferred to the OF memory M2 because the potential barrier is high, but is transferred to the OF memory M1, which has a low potential barrier.

In this way, by controlling the height of the potential barrier, it is possible to switch the transfer destination of the charge stored in the photoelectric conversion element 31. Therefore, instead of the OF transistors Q6 and Q8, an element structure that controls the height of the potential barrier may be provided.

FIG. 9 is a diagram showing the configuration of a pixel 100 and its surroundings as a comparative example. The pixel 100 shown in FIG. 9 differs from the pixel 30 in FIG. 3 in that it does not include OF transistors Q8 and Q9 and OF memory M2.

10A and 10B are diagrams showing an imaging operation by a pixel 100 of a comparative example. FIG. 10A shows an exposure period by the pixel 100. As in FIG. 6A, during the exposure period, charge e is transferred from the photoelectric conversion element 31 to the GS memory Mg and the OF memory M1.

FIG. 10B shows the readout period by pixel 100. As in FIG. 6B, during the readout period, charge e is transferred from GS memory Mg and OF memory M1 to charge-voltage converter 33. Also, in pixel 100, charge e is accumulated in photoelectric conversion element 31.

However, because pixel 100 does not have OF memory M2, it is not possible to start exposure for the next frame while charge is being transferred from OF memory M1 to charge-voltage conversion unit 33. This requires a long interval between exposure periods, which reduces the frame rate.

In this way, the photodetection device 2 in the first embodiment of the present disclosure has a GS memory Mg and two OF memories M1 and M2 for each pixel 30. For each frame, the photodetection device 2 alternates between using the OF memories M1 and M2 in addition to the GS memory Mg to transfer charges from the photoelectric conversion element 31. This allows the photodetection device 2 to perform exposure while reading out charges for each frame, thereby increasing the frame rate. Also, as described above, the OF memories M1 and M2 expand the dynamic range. In other words, the photodetection device 2 can achieve both an expanded dynamic range and an improved frame rate.

Second Embodiment
The configuration of the transfer unit 32 in Fig. 3 is an example, and various modified examples are possible. Fig. 11 is a diagram showing the configuration of a pixel 30a and a peripheral portion in the second embodiment of the present disclosure. As described above, the transfer unit 32 in Fig. 3 includes two OF memories M1 and M2 and one GM memory Mg, whereas the transfer unit 32a in Fig. 11 includes two GS memories Mg1 (third holding unit) and Mg2 (fourth holding unit) and two OF memories M1a (first holding unit) and M2a (second holding unit).

In addition, the transfer unit 32a in FIG. 11 has two GS transistors Q4a and Q5a, and four OF transistors Q6a, Q7a, Q8a, and Q9a. The GS transistor Q4a (seventh transfer control unit), OF transistor Q6a (eighth transfer control unit), and OF transistor Q7a (ninth transfer control unit) in FIG. 11 are cascode-connected between the photoelectric conversion element 31 and the charge-voltage conversion unit 33. A GS memory Mg1 is disposed at the connection node between the OF transistor Q4a and the OF transistor Q6a. An OF memory M1a is disposed at the connection node between the OF transistor Q6a and the OF transistor Q7a.

The OF transistor Q4a transfers charge from the photoelectric conversion element 31 to the GS memory Mg1 when the signal GS1 input to its gate is at a high level. The OF transistor Q6a transfers charge from the GS memory Mg1 to the OF memory M1a when the signal OF1 input to its gate is at a high level. The OF transistor Q7a transfers charge from the OF memory M1a to the charge-voltage conversion unit 33 when the signal OF2 input to its gate is at a high level.

The GS transistor Q5a (tenth transfer control unit), OF transistor Q8a (eleventh transfer control unit), and OF transistor Q9a (twelfth transfer control unit) in FIG. 11 are cascode-connected between the photoelectric conversion element 31 and the charge-voltage conversion unit 33. A GS memory Mg2 is disposed at the connection node between the OF transistor Q5a and the OF transistor Q8a. An OF memory M2a is disposed at the connection node between the OF transistor Q8a and the OF transistor Q9a.

The OF transistor Q5a transfers charge from the photoelectric conversion element 31 to the GS memory Mg2 when the signal GS2 input to its gate is at a high level. The OF transistor Q8a transfers charge from the GS memory Mg2 to the OF memory M2a when the signal OF3 input to its gate is at a high level. The OF transistor Q9a transfers charge from the OF memory M2a to the charge-voltage conversion unit 33 when the signal OF4 input to its gate is at a high level.

As described above, the transfer unit 32a in FIG. 11 can be provided with two GS memories Mg1 and Mg2 using the same number of transistors as the transfer unit 32 in FIG. 3.

As with pixel 30 in FIG. 3, it is desirable that the saturation charge amount of OF memories M1a and M2a is greater than the saturation charge amount of GS memories Mg1 and Mg2. Also, as with pixel 30, GS memories Mg1 and Mg2 hold the charge accumulated in the photoelectric conversion element 31 at the same time as other pixels 30a.

FIGS. 12A and 12B are diagrams showing the imaging operation by pixel 30a of the second embodiment of the present disclosure. FIG. 12A shows the exposure period of the first frame by pixel 30a. As shown in FIG. 12A, in the first frame, the charge e of the photoelectric conversion element 31 is transferred to the OF memory M1a via the GS memory Mg1. Also, immediately before the readout period of the first frame, the charge e of the photoelectric conversion element 31 is transferred to the GS memory Mg1.

FIG. 12B shows the readout period of the first frame and the exposure period of the second frame by pixel 30a. During the readout period of the first frame, the charge e in the OF memory M1a and the GS memory Mg1 is transferred to the charge-voltage converter 33. In parallel with this, during the exposure period of the second frame, the charge e in the photoelectric conversion element 31 is transferred to the OF memory M2a via the GS memory Mg2, as in FIG. 12A.

Even in the second frame and thereafter, as in the first embodiment, the OF memory M1a and the GS memory Mg1, and the OF memory M2a and the GS memory Mg2 are alternately used to read and transfer the charge e. That is, as in the first embodiment, the photodetector 2 of the second embodiment can expose the pixel 30a while reading the charge e from the pixel 30a.

As described above, pixel 30a of the second embodiment has two GS memories Mg1 and Mg2. That is, pixel 30a has the feature that even when either GS memory Mg1 or Mg2 transfers charge e to charge-voltage conversion unit 33, the other GS memory Mg1 or Mg2 can hold charge e of photoelectric conversion element 31. This allows pixel 30a to improve the frame rate compared to pixel 30 of the first embodiment.

FIG. 13 is a timing chart showing the control of pixel 30a in the second embodiment of the present disclosure. Times t31 to t55 shown in FIG. 13 include initialization, the exposure period and readout period of the first frame, the exposure period and readout period of the second frame, and the exposure period of the third frame.

Between time t31 and t32, pixel 30a is initialized before imaging. Specifically, similar to FIG. 7, reset signal RST, signals GS1 and GS2, and signals OF1 to OF4 are set to a high level for a certain period at the same timing. As a result, the charges in photoelectric conversion element 31, GS memories Mg1 and Mg2, OF memories M1a and M2a, and charge-voltage conversion unit 33 are discharged to the node of the high-voltage power supply VDD, and charge-voltage conversion unit 33 becomes the reset level voltage.

Times t33 to t35 are the exposure period of the first frame. In FIG. 13, during the exposure period of the first frame, the charge of the photoelectric conversion element 31 is transferred to the OF memory M1a and the GS memory Mg1.

From time t33 to t34, signal GS1 goes high, signal OF1 goes high, and OF transistors Q4a and Q6a are turned on, causing the charge stored in photoelectric conversion element 31 to be transferred to OF memory M1a via GS memory Mg1.

From time t34 to t35, signal GS1 is at a high level and signal OF1 is at a low level. This turns on OF transistor Q4a and turns off OF transistor Q6a, and the charge in photoelectric conversion element 31 is transferred to GS memory Mg1.

As in FIG. 7, the charge retention period of the OF memories M1a and M2a per frame is longer than the charge retention period of the GS memories Mg1 and Mg2.

Times t35 to t45 are the readout period of the first frame. From time t36 to t37, signals OF1 and OF2 are at low level, and selection signal SEL goes to high level. As a result, a pixel signal (P-phase signal) corresponding to the reset level of charge-voltage conversion unit 33 is read out, similar to FIG. 7.

Between time t38 and t39, the signal OF2 and the selection signal SEL go to high level. The OF transistor Q7a is turned on, and the charge in the OF memory M1a is transferred to the charge-voltage converter 33. As a result, a pixel signal (D-phase signal) corresponding to the charge obtained by adding the charge in the OF memory M1a to the charge in the charge-voltage converter 33 is read out.

From time t39 to t40, the reset signal RST goes high. This causes the charge in the charge-voltage converter 33 to be discharged to the node of the high-voltage power supply VDD.

Between time t40 and t41, signals OF1, OF2 and selection signal SEL go to high level. By turning on OF transistors Q6a and Q7a, the charges in GS memory Mg1 and OF memory M1a are transferred to the charge-voltage converter 33. As a result, a pixel signal (D-phase signal) corresponding to the charge obtained by adding the charge in OF memory M1a to the charge in GS memory Mg1 is read out from the charge-voltage converter 33.

From time t41 to t43, the signals OF1, OF2 and the reset signal RST go to high level, and the charges in the GS memory Mg1, the OF memory M1a and the charge-voltage conversion unit 33 are discharged to the node of the high-voltage power supply VDD. This causes the charge-voltage conversion unit 33 to go to the reset level voltage.

From time t43 to t44, signals OF1, OF2 and selection signal SEL go to high level, and a pixel signal (P-phase signal) corresponding to the reset level of the charge-voltage conversion unit 33 is read out.

Times t35 to t45 are the readout period for the first frame and also the exposure period for the second frame. From time t35 to t42, signal GS2 goes high and signal OF3 goes high. This causes the charge stored in photoelectric conversion element 31 to be transferred to OF memory M2a. From time t42 to t45, signal GS2 goes high and signal OF3 goes low. This causes the charge in photoelectric conversion element 31 to be transferred to GS memory Mg2.

In FIG. 7, the time t42 when the signal OF3 goes low is set between times t41 and t43 during the readout period of the first frame, but this is not limited to this. The time when the signal OF3 goes low may be set anywhere during the readout period of the first frame.

Times t45 to t55 are the readout period for the second frame. During the readout period for the second frame, charges are read out from the GS memory Mg2 and the OF memory M2a by the same control as during the readout period for the first frame.

That is, from time t46 to t47, the selection signal SEL goes high, and a pixel signal (P-phase signal) corresponding to the reset level of the charge-voltage conversion unit 33 is read out. From time t48 to t49, the signal OF4 and the selection signal SEL go high, and a pixel signal (D-phase signal) corresponding to the charge obtained by adding the charge of the OF memory M2a to the charge of the charge-voltage conversion unit 33 is read out.

From time t49 to t50, the reset signal RST goes high, and the charge-voltage conversion unit 33 goes to the reset level voltage. From time t50 to t51, the signals OF3, OF4 and the selection signal SEL go high, and a pixel signal (D-phase signal) corresponding to the charge obtained by adding the charge of the OF memory M2a to the charge of the GS memory Mg2 is read out from the charge-voltage conversion unit 33. From time t51 to t53, the signals OF3, OF4 and the reset signal RST go high, and the charge-voltage conversion unit 33 goes to the reset level voltage. From time t53 to t54, the signals OF3, OF4 and the selection signal SEL go high, and a pixel signal (P-phase signal) corresponding to the reset level of the charge-voltage conversion unit 33 is read out.

Times t45 to t55 are the readout period for the second frame and also the exposure period for the third frame. From time t45 to t52, signal GS1 goes high and signal OF1 goes high. This causes the charge stored in photoelectric conversion element 31 to be transferred to OF memory M1a. From time t52 to t55, signal GS1 goes high and signal OF1 goes low, and the charge in photoelectric conversion element 31 is transferred to GS memory Mg1.

As described above, the OF transistor Q4a or Q5a alternately transfers the charge stored in the photoelectric conversion element 31 to the GS memory Mg1 or Mg2 on a frame-by-frame basis. The OF transistor Q6a or Q8a alternately transfers the charge from the GS memory Mg1 or Mg2 to the OF memory M1a or M2a on a frame-by-frame basis. The OF transistor Q7a or Q9a alternately transfers the charge held in the OF memory M1a or M2a to the charge-voltage conversion unit 33 on a frame-by-frame basis. In addition, each of the multiple pixels 30a in the pixel array unit 20 alternately holds a charge in the OF memory M1a or M2a on a frame-by-frame basis, and also alternately holds a charge in the GS memory Mg1 or Mg2 on a frame-by-frame basis.

The timing chart in FIG. 13 differs from the timing chart in FIG. 7 in that an exposure period for the next frame is provided even when charge is read out from the GS memory Mg1 or Mg2. In other words, the pixel 30a of the second embodiment can extend the exposure period during imaging operation compared to the pixel 30 of the first embodiment.

In this way, the pixel 30a in the second embodiment has two GS memories Mg1 and Mg2 and two OF memories M1a and M2a. The GS memory Mg1 and OF memory M1a, and the GS memory Mg2 and OF memory M2a alternately hold charges from the photoelectric conversion element 31 for each frame. In addition, the GS memory Mg1 and OF memory M1a, and the GS memory Mg2 and OF memory M2a alternately transfer the held charges to the charge-voltage conversion unit 33 for each frame. This allows exposure to be performed while the pixel signal is being read out in each frame, and the frame rate can be increased. The pixel 30a in the second embodiment can control the transfer of the two GS memories Mg1 and Mg2 and the two OF memories M1a and M2a with the same number of transistors Q4a to Q9a as the pixel 30 in the first embodiment.

(Application example)
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be applied to an endoscopic surgery system.

FIG. 14 is a diagram showing an example of the general configuration of an endoscopic surgery system to which the technology disclosed herein (the present technology) can be applied.

In FIG. 14, an operator (doctor) 11131 is shown using an endoscopic surgery system 11000 to perform surgery on a patient 11132 on a patient bed 11133. As shown in the figure, the endoscopic surgery system 11000 is composed of an endoscope 11100, other surgical tools 11110 such as an insufflation tube 11111 and an energy treatment tool 11112, a support arm device 11120 that supports the endoscope 11100, and a cart 11200 on which various devices for endoscopic surgery are mounted.

The endoscope 11100 is composed of a lens barrel 11101, the tip of which is inserted into the body cavity of the patient 11132 at a predetermined length, and a camera head 11102 connected to the base end of the lens barrel 11101. In the illustrated example, the endoscope 11100 is configured as a so-called rigid scope having a rigid lens barrel 11101, but the endoscope 11100 may also be configured as a so-called flexible scope having a flexible lens barrel.

The tip of the tube 11101 has an opening into which an objective lens is fitted. A light source device 11203 is connected to the endoscope 11100, and light generated by the light source device 11203 is guided to the tip of the tube by a light guide extending inside the tube 11101, and is irradiated via the objective lens towards an object to be observed inside the body cavity of the patient 11132. The endoscope 11100 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

An optical system and an image sensor are provided inside the camera head 11102, and reflected light (observation light) from the object being observed is focused onto the image sensor by the optical system. The image sensor converts the observation light into an electric signal corresponding to the observation light, i.e., an image signal corresponding to the observed image. The image signal is sent to the camera control unit (CCU: Camera Control Unit) 11201 as RAW data.

The CCU 11201 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the overall operation of the endoscope 11100 and the display device 11202. Furthermore, the CCU 11201 receives an image signal from the camera head 11102, and performs various types of image processing on the image signal, such as development processing (demosaic processing), in order to display an image based on the image signal.

The display device 11202, under the control of the CCU 11201, displays an image based on the image signal that has been subjected to image processing by the CCU 11201.

The light source device 11203 is composed of a light source such as an LED (light emitting diode), and supplies illumination light to the endoscope 11100 when photographing the surgical site, etc.

The input device 11204 is an input interface for the endoscopic surgery system 11000. A user can input various information and instructions to the endoscopic surgery system 11000 via the input device 11204. For example, the user inputs an instruction to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) of the endoscope 11100.

The treatment tool control device 11205 controls the operation of the energy treatment tool 11112 for cauterizing tissue, incising, sealing blood vessels, etc. The insufflation device 11206 sends gas into the body cavity of the patient 11132 via the insufflation tube 11111 to inflate the body cavity in order to ensure a clear field of view for the endoscope 11100 and to ensure a working space for the surgeon. The recorder 11207 is a device capable of recording various types of information related to the surgery. The printer 11208 is a device capable of printing various types of information related to the surgery in various formats such as text, images, or graphs.

The light source device 11203 that supplies illumination light to the endoscope 11100 when photographing the surgical site can be composed of a white light source composed of, for example, an LED, a laser light source, or a combination of these. When the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the white balance of the captured image can be adjusted in the light source device 11203. In this case, it is also possible to capture images corresponding to each of the RGB colors in a time-division manner by irradiating the observation object with laser light from each of the RGB laser light sources in a time-division manner and controlling the drive of the image sensor of the camera head 11102 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

The light source device 11203 may be controlled to change the intensity of the light it outputs at predetermined time intervals. The image sensor of the camera head 11102 may be controlled to acquire images in a time-division manner in synchronization with the timing of the change in the light intensity, and the images may be synthesized to generate an image with a high dynamic range that is free of so-called blackout and whiteout.

The light source device 11203 may also be configured to supply light of a predetermined wavelength band corresponding to special light observation. In special light observation, for example, by utilizing the wavelength dependency of light absorption in body tissue, a narrow band of light is irradiated compared to the light irradiated during normal observation (i.e., white light), and a specific tissue such as blood vessels on the surface of the mucosa is photographed with high contrast, so-called narrow band imaging is performed. Alternatively, in special light observation, fluorescence observation may be performed in which an image is obtained by fluorescence generated by irradiating excitation light. In fluorescence observation, excitation light is irradiated to body tissue and fluorescence from the body tissue is observed (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue and excitation light corresponding to the fluorescence wavelength of the reagent is irradiated to the body tissue to obtain a fluorescent image. The light source device 11203 may be configured to supply narrow band light and/or excitation light corresponding to such special light observation.

FIG. 15 is a block diagram showing an example of the functional configuration of the camera head 11102 and CCU 11201 shown in FIG. 14.

The camera head 11102 has a lens unit 11401, an imaging unit 11402, a drive unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 has a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are connected to each other via a transmission cable 11400 so that they can communicate with each other.

The lens unit 11401 is an optical system provided at the connection with the lens barrel 11101. Observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and enters the lens unit 11401. The lens unit 11401 is composed of a combination of multiple lenses including a zoom lens and a focus lens.

The imaging unit 11402 may have one imaging element (a so-called single-plate type) or multiple imaging elements (a so-called multi-plate type). When the imaging unit 11402 is configured as a multi-plate type, for example, each imaging element may generate an image signal corresponding to each of RGB, and a color image may be obtained by combining these. Alternatively, the imaging unit 11402 may be configured to have a pair of imaging elements for acquiring image signals for the right eye and the left eye corresponding to a 3D (dimensional) display. By performing a 3D display, the surgeon 11131 can more accurately grasp the depth of the biological tissue in the surgical site. Note that when the imaging unit 11402 is configured as a multi-plate type, multiple lens units 11401 may be provided corresponding to each imaging element.

Furthermore, the imaging unit 11402 does not necessarily have to be provided in the camera head 11102. For example, the imaging unit 11402 may be provided inside the lens barrel 11101, immediately after the objective lens.

The driving unit 11403 is composed of an actuator, and moves the zoom lens and focus lens of the lens unit 11401 a predetermined distance along the optical axis under the control of the camera head control unit 11405. This allows the magnification and focus of the image captured by the imaging unit 11402 to be adjusted appropriately.

The communication unit 11404 is configured with a communication device for transmitting and receiving various information to and from the CCU 11201. The communication unit 11404 transmits the image signal obtained from the imaging unit 11402 as RAW data to the CCU 11201 via the transmission cable 11400.

The communication unit 11404 also receives control signals for controlling the operation of the camera head 11102 from the CCU 11201, and supplies them to the camera head control unit 11405. The control signals include information on the imaging conditions, such as information specifying the frame rate of the captured image, information specifying the exposure value during imaging, and/or information specifying the magnification and focus of the captured image.

The imaging conditions such as the frame rate, exposure value, magnification, and focus may be appropriately specified by the user, or may be automatically set by the control unit 11413 of the CCU 11201 based on the acquired image signal. In the latter case, the endoscope 11100 is equipped with the so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function.

The camera head control unit 11405 controls the operation of the camera head 11102 based on a control signal from the CCU 11201 received via the communication unit 11404.

The communication unit 11411 is configured with a communication device for transmitting and receiving various information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted from the camera head 11102 via the transmission cable 11400.

The communication unit 11411 also transmits to the camera head 11102 a control signal for controlling the operation of the camera head 11102. The image signal and the control signal can be transmitted by electrical communication, optical communication, etc.

The image processing unit 11412 performs various image processing operations on the image signal, which is the RAW data transmitted from the camera head 11102.

The control unit 11413 performs various controls related to the imaging of the surgical site, etc. by the endoscope 11100, and the display of the captured images obtained by imaging the surgical site, etc. For example, the control unit 11413 generates a control signal for controlling the driving of the camera head 11102.

The control unit 11413 also causes the display device 11202 to display the captured image showing the surgical site, etc., based on the image signal that has been image-processed by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image using various image recognition techniques. For example, the control unit 11413 can recognize surgical tools such as forceps, specific body parts, bleeding, mist generated when the energy treatment tool 11112 is used, etc., by detecting the shape and color of the edges of objects included in the captured image. When the control unit 11413 causes the display device 11202 to display the captured image, it may use the recognition result to superimpose various types of surgical support information on the image of the surgical site. By superimposing the surgical support information and presenting it to the surgeon 11131, the burden on the surgeon 11131 can be reduced and the surgeon 11131 can proceed with the surgery reliably.

The transmission cable 11400 that connects the camera head 11102 and the CCU 11201 is an electrical signal cable that supports electrical signal communication, an optical fiber that supports optical communication, or a composite cable of these.

In the illustrated example, communication is performed wired using a transmission cable 11400, but communication between the camera head 11102 and the CCU 11201 may also be performed wirelessly.

Above, an example of an endoscopic surgery system to which the technology disclosed herein can be applied has been described. The technology disclosed herein can be applied to the camera head 11102 and other components of the configuration described above. Specifically, the light detection device 2 or electronic device 1 in FIG. 1 can be applied to the imaging unit 11402 of the camera head 11102. By applying the technology disclosed herein to the imaging unit 11402, clearer and faster images of the surgical site can be obtained, allowing the surgeon to reliably confirm the surgical site.

Note that although an endoscopic surgery system has been described here as an example, the technology disclosed herein may also be applied to other systems, such as a microsurgery system.

The present technology can be configured as follows.
(1) A pixel pixel includes a plurality of pixels each having a photoelectric conversion element that accumulates an electric charge according to the amount of incident light,
Each of the plurality of pixels is
a charge-voltage converter for converting the charge stored in the photoelectric conversion element into a voltage;
a first holding unit that holds the charge accumulated in the photoelectric conversion element;
a second holding unit that holds the charges accumulated in the photoelectric conversion element alternately with the first holding unit;
a third holding unit that holds the charge accumulated in the photoelectric conversion element at the same timing as other pixels;
Light detection device.
(2) While one of the first holding unit and the second holding unit transfers the charge to the charge-voltage converter, the other of the first holding unit and the second holding unit holds the charge accumulated in the photoelectric conversion element.
An optical detection device as described in (1).
(3) Each of the plurality of pixels alternately holds an electric charge in the first holding unit or the second holding unit on a frame-by-frame basis, and holds an electric charge in the third holding unit for each frame.
An optical detection device according to (1) or (2).
(4) each of the plurality of pixels outputs, for each frame, a pixel signal of a reset level and a pixel signal corresponding to the charge held in the first holding unit or the second holding unit and the charge held in the third holding unit;
the pixel signal of the reset level and the pixel signal according to the charge are output at different timings.
The optical detection device according to any one of (1) to (3).
(5) A saturated charge amount of the first storage unit and the second storage unit is greater than a saturated charge amount of the third storage unit.
The optical detection device according to any one of (1) to (4).
(6) A charge holding period of the first holding unit and the second holding unit per frame is longer than a charge holding period of the third holding unit.
The optical detection device according to any one of (1) to (5).
(7) Each of the plurality of pixels comprises:
a first transfer control unit that controls the transfer of charges from the photoelectric conversion element to the first holding unit;
a second transfer control unit that controls the transfer of charges from the first holding unit to the charge-voltage conversion unit;
a third transfer control unit that controls the transfer of charges from the photoelectric conversion element to the second holding unit;
a fourth transfer control unit that controls the transfer of charges from the second holding unit to the charge-voltage conversion unit,
The optical detection device according to any one of (1) to (6).
(8) The first transfer control unit or the third transfer control unit alternately transfers the charges accumulated in the photoelectric conversion element to the first holding unit or the second holding unit on a frame basis.
(7) An optical detection device according to (7).
(9) The second transfer control unit or the fourth transfer control unit alternately transfers the charges held in the first holding unit or the second holding unit to the charge-voltage conversion unit on a frame basis.
An optical detection device according to (7) or (8).
(10) When the first transfer control unit transfers electric charges from the photoelectric conversion element to the first holding unit, a negative bias voltage is supplied to a gate of the third transfer control unit;
When the third transfer control unit transfers the electric charge from the photoelectric conversion element to the second holding unit, a negative bias voltage is supplied to a gate of the first transfer control unit.
The optical detection device according to any one of (7) to (9).
(11) Each of the plurality of pixels
a fifth transfer control unit that controls the transfer of charges from the photoelectric conversion element to the third holding unit;
a sixth transfer control unit that controls the transfer of charges from the third holding unit to the charge-voltage conversion unit;
The optical detection device according to any one of (7) to (10).
(12) The fifth transfer control unit transfers electric charges from the photoelectric conversion element to the third holding unit at the same timing as other pixels for each frame;
the sixth transfer control unit transfers the charge from the third holding unit to the charge-voltage conversion unit for each frame.
An optical detection device according to (11).
(13) The fifth transfer control unit transfers the charge from the photoelectric conversion element to the third holding unit after the charge is transferred from the photoelectric conversion element to the first holding unit or the second holding unit for each frame.
The optical detection device according to (11) or (12).
(14) Each of the plurality of pixels includes a fourth holding unit that holds the charge accumulated in the photoelectric conversion element at the same timing as the other pixels,
each of the plurality of pixels alternately holds an electric charge in the first holding unit or the second holding unit on a frame-by-frame basis, and alternately holds an electric charge in the third holding unit or the fourth holding unit on a frame-by-frame basis;
An optical detection device according to (1) or (2).
(15) Each of the plurality of pixels outputs, for each frame, a pixel signal of a reset level and a pixel signal corresponding to the charges held in the first holding unit and the third holding unit, or a pixel signal corresponding to the charges held in the second holding unit or the fourth holding unit;
the pixel signal of the reset level and the pixel signal according to the charge are output at different timings.
An optical detection device according to (14).
(16) A saturated charge amount of the first holding unit and the second holding unit is greater than a saturated charge amount of the third holding unit and the fourth holding unit.
The optical detection device according to (14) or (15).
(17) A charge holding period of the first holding unit and the second holding unit per frame is longer than a charge holding period of the third holding unit and the fourth holding unit.
The optical detection device according to any one of (14) to (16).
(18) Each of the plurality of pixels comprises:
a seventh transfer control unit that controls the transfer of charges from the photoelectric conversion element to the third holding unit;
an eighth transfer control unit that controls the transfer of charges from the third holding unit to the first holding unit;
a ninth transfer control unit that controls the transfer of charges from the first holding unit to the charge-voltage conversion unit;
a tenth transfer control unit that controls the transfer of charges from the photoelectric conversion element to the fourth holding unit;
an eleventh transfer control unit that controls the transfer of charges from the fourth holding unit to the second holding unit;
a twelfth transfer control unit that controls the transfer of charges from the second holding unit to the charge-voltage conversion unit;
The optical detection device according to any one of (14) to (17).
(19) The seventh transfer control unit or the tenth transfer control unit alternately transfers the charges accumulated in the photoelectric conversion element to the third holding unit or the fourth holding unit on a frame basis,
the eighth transfer control unit or the eleventh transfer control unit alternately transfers charges from the third holding unit or the fourth holding unit to the first holding unit or the second holding unit on a frame-by-frame basis;
the ninth transfer control unit or the twelfth transfer control unit alternately transfers the charges held in the first holding unit or the second holding unit to the charge-voltage conversion unit on a frame basis;
(18) An optical detection device according to (18).
(20) a light detection device;
A processing unit that processes pixel data output from the light detection device,
The light detection device includes:
The image sensor includes a plurality of pixels each having a photoelectric conversion element that accumulates an electric charge according to the amount of incident light,
Each of the plurality of pixels is
a charge-voltage converter for converting the charge stored in the photoelectric conversion element into a voltage;
a first holding unit that holds the charge accumulated in the photoelectric conversion element;
a second holding unit that holds the charges accumulated in the photoelectric conversion element alternately with the first holding unit;
a third holding unit that holds the charge accumulated in the photoelectric conversion element at the same timing as other pixels;
Electronics.

The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by a person skilled in the art, and the effects of the present disclosure are not limited to the above. In other words, various additions, modifications, and partial deletions are possible within the scope that does not deviate from the conceptual idea and intent of the present disclosure derived from the contents defined in the claims and their equivalents.

1 Electronic device, 2, 2a Light detection device, 3 Processing unit, 4 Control unit, 5 Recording unit, 11 Imaging lens, 12 Transmission line, 13 Control line, 20 Pixel array unit, 21 Vertical drive circuit, 22 System control circuit, 23 Signal processing unit, 30, 30a, 100 Pixel, 31 Photoelectric conversion element, 32, 32a Transfer unit, 33 Charge-to-voltage conversion unit, 41 Pixel chip, 42 Logic chip, 43 First pixel chip, 44 Second pixel chip

Claims

The image sensor includes a plurality of pixels each having a photoelectric conversion element that accumulates an electric charge according to the amount of incident light,
Each of the plurality of pixels is
a charge-voltage converter for converting the charge stored in the photoelectric conversion element into a voltage;
a first holding unit that holds the charge accumulated in the photoelectric conversion element;
a second holding unit that holds the charges accumulated in the photoelectric conversion element alternately with the first holding unit;
a third holding unit that holds the charge accumulated in the photoelectric conversion element at the same timing as other pixels;
Light detection device.
While one of the first holding unit and the second holding unit transfers the charge to the charge-voltage converter, the other of the first holding unit and the second holding unit holds the charge accumulated in the photoelectric conversion element.
2. The optical detection device according to claim 1.
each of the plurality of pixels alternately holds charge in the first holding unit or the second holding unit on a frame-by-frame basis, and holds charge in the third holding unit on a frame-by-frame basis;
2. The optical detection device according to claim 1.
each of the plurality of pixels outputs, for each frame, a pixel signal of a reset level and a pixel signal corresponding to the charge held in the first holding unit or the second holding unit and the charge held in the third holding unit;
the pixel signal of the reset level and the pixel signal according to the charge are output at different timings.
2. The optical detection device according to claim 1.
a saturated charge amount of the first storage unit and the second storage unit is greater than a saturated charge amount of the third storage unit;
2. The optical detection device according to claim 1.
a charge holding period of the first holding unit and the second holding unit per frame is longer than a charge holding period of the third holding unit;
2. The optical detection device according to claim 1.
Each of the plurality of pixels is
a first transfer control unit that controls the transfer of charges from the photoelectric conversion element to the first holding unit;
a second transfer control unit that controls the transfer of charges from the first holding unit to the charge-voltage conversion unit;
a third transfer control unit that controls the transfer of charges from the photoelectric conversion element to the second holding unit;
a fourth transfer control unit that controls the transfer of charges from the second holding unit to the charge-voltage conversion unit,
2. The optical detection device according to claim 1.
the first transfer control unit or the third transfer control unit alternately transfers the charges accumulated in the photoelectric conversion element to the first holding unit or the second holding unit on a frame basis;
8. The optical detection device according to claim 7.
the second transfer control unit or the fourth transfer control unit alternately transfers the charges held in the first holding unit or the second holding unit to the charge-voltage conversion unit on a frame-by-frame basis;
8. The optical detection device according to claim 7.
When the first transfer control unit transfers charges from the photoelectric conversion element to the first holding unit, a negative bias voltage is supplied to a gate of the third transfer control unit,
When the third transfer control unit transfers the electric charge from the photoelectric conversion element to the second holding unit, a negative bias voltage is supplied to a gate of the first transfer control unit.
8. The optical detection device according to claim 7.
Each of the plurality of pixels is
a fifth transfer control unit that controls the transfer of charges from the photoelectric conversion element to the third holding unit;
a sixth transfer control unit that controls the transfer of charges from the third holding unit to the charge-voltage conversion unit;
8. The optical detection device according to claim 7.
the fifth transfer control unit transfers charges from the photoelectric conversion element to the third holding unit at the same timing as other pixels for each frame;
the sixth transfer control unit transfers the charge from the third holding unit to the charge-voltage conversion unit for each frame.
The optical detection device according to claim 11.
the fifth transfer control unit transfers the charge from the photoelectric conversion element to the third holding unit after the charge is transferred from the photoelectric conversion element to the first holding unit or the second holding unit for each frame;
The optical detection device according to claim 11.
Each of the plurality of pixels includes a fourth holding unit that holds the charge accumulated in the photoelectric conversion element at the same time as the other pixels,
each of the plurality of pixels alternately holds an electric charge in the first holding unit or the second holding unit on a frame-by-frame basis, and alternately holds an electric charge in the third holding unit or the fourth holding unit on a frame-by-frame basis;
2. The optical detection device according to claim 1.
each of the plurality of pixels outputs, for each frame, a pixel signal of a reset level and a pixel signal corresponding to the charges held in the first holding unit and the third holding unit or a pixel signal corresponding to the charges held in the second holding unit or the fourth holding unit;
the pixel signal of the reset level and the pixel signal according to the charge are output at different timings.
15. The optical detection device of claim 14.
a saturated charge amount of the first holding unit and the second holding unit is greater than a saturated charge amount of the third holding unit and the fourth holding unit;
15. The optical detection device of claim 14.
a charge holding period of the first holding unit and the second holding unit per frame is longer than a charge holding period of the third holding unit and the fourth holding unit;
15. The optical detection device of claim 14.
Each of the plurality of pixels includes a seventh transfer control unit that controls the transfer of electric charges from the photoelectric conversion element to the third holding unit, and an eighth transfer control unit that controls the transfer of electric charges from the third holding unit to the first holding unit.
a ninth transfer control unit that controls the transfer of charges from the first holding unit to the charge-voltage conversion unit;
a tenth transfer control unit that controls the transfer of charges from the photoelectric conversion element to the fourth holding unit;
an eleventh transfer control unit that controls the transfer of charges from the fourth holding unit to the second holding unit;
a twelfth transfer control unit that controls the transfer of charges from the second holding unit to the charge-voltage conversion unit;
15. The optical detection device of claim 14.
the seventh transfer control unit or the tenth transfer control unit alternately transfers the charges accumulated in the photoelectric conversion element to the third holding unit or the fourth holding unit on a frame basis;
the eighth transfer control unit or the eleventh transfer control unit alternately transfers charges from the third holding unit or the fourth holding unit to the first holding unit or the second holding unit on a frame-by-frame basis;
the ninth transfer control unit or the twelfth transfer control unit alternately transfers the charges held in the first holding unit or the second holding unit to the charge-voltage conversion unit on a frame basis;
20. The optical detection device of claim 18.
A photodetector;
A processing unit that processes pixel data output from the light detection device,
The light detection device includes:
The image sensor includes a plurality of pixels each having a photoelectric conversion element that accumulates an electric charge according to the amount of incident light,
Each of the plurality of pixels is
a charge-voltage converter for converting the charge stored in the photoelectric conversion element into a voltage;
a first holding unit that holds the charge accumulated in the photoelectric conversion element;
a second holding unit that holds the charges accumulated in the photoelectric conversion element alternately with the first holding unit;
a third holding unit that holds the charge accumulated in the photoelectric conversion element at the same timing as other pixels;
Electronics.