Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/70—SSIS architectures; Circuits associated therewith
  • H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
  • H04N25/77—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
  • H04N25/772—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising A/D, V/T, V/F, I/T or I/F converters
  • H04N25/773—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising A/D, V/T, V/F, I/T or I/F converters comprising photon counting circuits, e.g. single photon detection [SPD] or single photon avalanche diodes [SPAD]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/70—SSIS architectures; Circuits associated therewith
  • H04N25/703—SSIS architectures incorporating pixels for producing signals other than image signals
  • H04N25/705—Pixels for depth measurement, e.g. RGBZ
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/50—Control of the SSIS exposure

Definitions

• the present invention relates to a photoelectric conversion device.
• Patent Document 1 discloses a distance measuring device that measures the distance to an object by emitting light from a light source and receiving light including reflected light from the object using a light receiving element.
• a SPAD Single Photon Avalanche Diode
• Patent Document 1 discloses a method of repeatedly performing measurements while changing the gating period during which photon detection is performed in a SPAD element.
• An object of the present invention is to provide a photoelectric conversion device with an improved frame rate while ensuring appropriate distance resolution.
• a plurality of photoelectric conversion elements include a first photoelectric conversion element and a second photoelectric conversion element.
• a first exposure period in which the 1-bit signal is generated based on the light incident on the first photoelectric conversion element in the acquisition period of one of the microframes, and the light incident on the second photoelectric conversion element There is provided a photoelectric conversion device characterized in that the second exposure period during which the 1-bit signal is generated based on the 1-bit signal is different from each other.
• a photoelectric conversion device with an improved frame rate while ensuring appropriate distance resolution is provided.
• FIG. 1 is a hardware block diagram showing a schematic configuration example of a distance measuring device according to a first embodiment.
• FIG. 1 is a schematic diagram showing the overall configuration of a photoelectric conversion device according to a first embodiment.
• FIG. 2 is a schematic block diagram showing a configuration example of a sensor board according to the first embodiment.
• 1 is a schematic block diagram showing a configuration example of a circuit board according to a first embodiment.
• FIG. FIG. 2 is a schematic block diagram showing a configuration example of a photoelectric conversion unit and a pixel signal processing unit for one pixel according to the first embodiment.
• FIG. 3 is a diagram illustrating the operation of the avalanche photodiode according to the first embodiment.
• FIG. 1 is a hardware block diagram showing a schematic configuration example of a distance measuring device according to a first embodiment.
• FIG. 1 is a schematic diagram showing the overall configuration of a photoelectric conversion device according to a first embodiment.
• FIG. 2 is a schematic block diagram showing a
• FIG. 3 is a diagram illustrating the operation of the avalanche photodiode according to the first embodiment.
• FIG. 3 is a diagram illustrating the operation of the avalanche photodiode according to the first embodiment.
• FIG. 1 is a functional block diagram showing a schematic configuration example of a distance measuring device according to a first embodiment.
• FIG. 3 is a schematic diagram for explaining a distance measurement frame, a subframe, and a microframe according to the first embodiment.
• 2 is a flowchart showing the operation of the distance image generation device according to the first embodiment during one distance measurement frame period.
• FIG. 2 is a schematic diagram of a pixel array according to the first embodiment.
• FIG. 3 is a drive timing diagram according to the first embodiment.
• FIG. 1 is a functional block diagram showing a schematic configuration example of a distance measuring device according to a first embodiment.
• FIG. 3 is a schematic diagram for explaining a distance measurement frame, a subframe, and a microframe according to the first embodiment
• FIG. 3 is a schematic diagram of a pixel array according to a second embodiment.
• FIG. 7 is a drive timing diagram according to a second embodiment.
• FIG. 7 is a schematic diagram of a pixel array according to a third embodiment.
• FIG. 7 is a drive timing diagram according to a third embodiment.
• FIG. 7 is a schematic block diagram of a pixel according to a fourth embodiment.
• FIG. 7 is a drive timing diagram according to a fourth embodiment.
• It is a functional block diagram showing an example of a schematic structure of a distance measuring device concerning a 5th embodiment.
• It is a schematic diagram which shows the example of the external light map based on 5th Embodiment.
• It is a schematic diagram of the equipment concerning a 6th embodiment.
• It is a schematic diagram of the equipment concerning a 6th embodiment.
• FIG. 1 is a hardware block diagram showing a schematic configuration example of a distance image generation device 30 according to the present embodiment.
• the distance image generation device 30 includes a light emitting device 31, a light receiving device 32, and a signal processing circuit 33. Note that the configuration of the distance image generation device 30 shown in this embodiment is an example, and is not limited to the illustrated configuration.
• the distance image generation device 30 is a device that measures the distance to the target object X using a technology such as LiDAR (Light Detection and Ranging).
• the distance image generation device 30 measures the distance from the distance image generation device 30 to the object X based on the time difference between when the light emitted from the light emitting device 31 is reflected by the object X and when the light is received by the light receiving device 32. do.
• the distance image generation device 30 can measure distances at multiple points two-dimensionally by emitting a laser beam to a predetermined distance measurement range including the target object X and receiving the reflected light by a pixel array. . Thereby, the distance image generation device 30 can generate and output a distance image.
• the light received by the light receiving device 32 includes environmental light such as sunlight in addition to the reflected light from the object X. Therefore, the distance image generation device 30 measures the incident light in each of a plurality of periods (bin periods), and determines that the reflected light is incident during the period when the amount of light is at its peak. Performs distance measurement that reduces the influence of
• the light emitting device 31 is a device that emits light such as a laser beam to the outside of the distance image generating device 30.
• the signal processing circuit 33 may include a processor that performs arithmetic processing on digital signals, a memory that stores digital signals, and the like.
• the memory may be, for example, a semiconductor memory.
• the light receiving device 32 generates a pulse signal including a pulse based on the incident light.
• the light receiving device 32 is, for example, a photoelectric conversion device including an avalanche photodiode as a photoelectric conversion element. In this case, when one photon is incident on the avalanche photodiode and a charge is generated, one pulse is generated by avalanche multiplication.
• the light receiving device 32 may be one using a photoelectric conversion element using another photodiode, for example.
• the light receiving device 32 includes a pixel array in which a plurality of photoelectric conversion elements (pixels) are arranged in a plurality of rows and a plurality of columns.
• a photoelectric conversion device that is a specific configuration example of the light receiving device 32 will be described with reference to FIGS. 2 to 6C.
• the configuration example of the photoelectric conversion device described below is one example.
• the photoelectric conversion device that can be applied to the light receiving device 32 is not limited to this, and any device may be used as long as it can realize the functions shown in FIG. 7, which will be described later.
• FIG. 2 is a schematic diagram showing the overall configuration of the photoelectric conversion device 100 according to this embodiment.
• the photoelectric conversion device 100 includes a sensor substrate 11 (first substrate) and a circuit board 21 (second substrate) that are stacked on each other.
• the sensor board 11 and the circuit board 21 are electrically connected to each other.
• the sensor substrate 11 has a pixel area 12 in which a plurality of pixels 101 are arranged in a plurality of rows and a plurality of columns.
• the circuit board 21 includes a first circuit area 22 in which a plurality of pixel signal processing units 103 arranged in a plurality of rows and a plurality of columns are arranged, and a second circuit area 22 arranged around the outer periphery of the first circuit area 22. It has a circuit area 23.
• the second circuit area 23 may include a circuit that controls the plurality of pixel signal processing units 103 and the like.
• the sensor substrate 11 has a light entrance surface that receives incident light and a connection surface that faces the light entrance surface.
• the sensor board 11 is connected to the circuit board 21 on the connection surface side. That is, the photoelectric conversion device 100 is a so-called back-illuminated type.
• planar view refers to viewing from a direction perpendicular to the surface opposite to the light incidence surface.
• cross section refers to a surface in a direction perpendicular to the surface of the sensor substrate 11 on the opposite side to the light incident surface. Note that there may be cases where the light entrance surface is a rough surface when viewed microscopically, but in that case, a planar view is defined based on the light entrance surface when viewed macroscopically.
• the sensor board 11 and the circuit board 21 are described below as being diced chips, the sensor board 11 and the circuit board 21 are not limited to chips.
• the sensor substrate 11 and the circuit board 21 may be wafers.
• the photoelectric conversion device 100 may be manufactured by stacking them in a wafer state and then dicing them, or by stacking them after being diced. It may be manufactured by
• FIG. 3 is a schematic block diagram showing an example of the arrangement of the sensor board 11.
• a plurality of pixels 101 are arranged in a plurality of rows and a plurality of columns.
• Each of the plurality of pixels 101 has within its substrate a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter referred to as APD) as a photoelectric conversion element.
• APD avalanche photodiode
• the conductivity type of the charge used as the signal charge is called the first conductivity type.
• the first conductivity type refers to a conductivity type in which majority carriers are charges having the same polarity as the signal charges.
• a conductivity type opposite to the first conductivity type that is, a conductivity type in which the majority carriers are charges having a polarity different from that of the signal charges, is referred to as a second conductivity type.
• the anode of the APD is at a fixed potential, and a signal is extracted from the cathode of the APD.
• the semiconductor region of the first conductivity type is an N-type semiconductor region
• the semiconductor region of the second conductivity type is a P-type semiconductor region.
• the configuration may be such that the cathode of the APD is at a fixed potential and the signal is extracted from the anode of the APD.
• the semiconductor region of the first conductivity type is a P-type semiconductor region
• the semiconductor region of the second conductivity type is an N-type semiconductor region.
• a configuration in which the potentials of both nodes vary may be used.
• FIG. 4 is a schematic block diagram showing an example of the configuration of the circuit board 21.
• the circuit board 21 has a first circuit area 22 in which a plurality of pixel signal processing units 103 are arranged in a plurality of rows and a plurality of columns.
• a vertical scanning circuit 110 a horizontal scanning circuit 111, a readout circuit 112, a pixel output signal line 113, an output circuit 114, and a control signal generation section 115 are arranged.
• the plurality of photoelectric conversion units 102 shown in FIG. 3 and the plurality of pixel signal processing units 103 shown in FIG. 4 are electrically connected via connection wiring provided for each pixel 101. ing.
• the control signal generation section 115 is a control circuit that generates control signals for driving the vertical scanning circuit 110, horizontal scanning circuit 111, and readout circuit 112, and supplies them to each of these sections. Thereby, the control signal generation section 115 controls the drive timing of each section, etc.
• the vertical scanning circuit 110 supplies a control signal to each of the plurality of pixel signal processing units 103 based on the control signal supplied from the control signal generation unit 115.
• the vertical scanning circuit 110 supplies a control signal for each row to each pixel signal processing section 103 via a drive line provided for each row of the first circuit area 22 .
• each row may have a plurality of drive lines.
• Logic circuits such as a shift register and an address decoder may be used for the vertical scanning circuit 110. Thereby, the vertical scanning circuit 110 selects a row from which the pixel signal processing unit 103 outputs a signal.
• the signal output from the photoelectric conversion unit 102 of the pixel 101 is processed by the pixel signal processing unit 103.
• the pixel signal processing unit 103 acquires and holds a digital signal having a plurality of bits by counting the number of pulses output from the APD included in the photoelectric conversion unit 102.
• the pixel signal processing unit 103 does not necessarily have to be provided for every pixel 101.
• one pixel signal processing unit 103 may be shared by a plurality of pixels 101.
• the pixel signal processing unit 103 provides a signal processing function to each pixel 101 by sequentially processing the signals output from each photoelectric conversion unit 102.
• the horizontal scanning circuit 111 supplies a control signal to the readout circuit 112 based on the control signal supplied from the control signal generation section 115.
• the pixel signal processing section 103 is connected to the readout circuit 112 via a pixel output signal line 113 provided for each column of the first circuit area 22.
• a pixel output signal line 113 in one column is shared by a plurality of pixel signal processing units 103 in the corresponding column.
• the pixel output signal line 113 includes a plurality of wiring lines, and has at least the function of outputting a digital signal from each pixel signal processing unit 103 to the readout circuit 112, and the function of outputting a control signal for selecting a column to output a signal to the pixel. It has a function of supplying signals to the signal processing section 103.
• the readout circuit 112 outputs a signal to a storage section or a signal processing section outside the photoelectric conversion device 100 via the output circuit 114 based on the control signal supplied from the control signal generation section 115.
• the photoelectric conversion units 102 in the pixel area 12 may be arranged one-dimensionally. Further, the function of the pixel signal processing unit 103 does not necessarily have to be provided in every pixel 101. For example, one pixel signal processing unit 103 may be shared by a plurality of pixels 101. In this case, the pixel signal processing unit 103 provides a signal processing function to each pixel 101 by sequentially processing the signals output from each photoelectric conversion unit 102.
• a first circuit area 22 in which a plurality of pixel signal processing units 103 are arranged is arranged in an area that overlaps with the pixel area 12 in plan view.
• a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control signal generation section 115 are arranged so as to overlap between the edge of the sensor substrate 11 and the edge of the pixel area 12. be done.
• the sensor substrate 11 has a pixel area 12 and a non-pixel area arranged around the pixel area 12.
• a second circuit area 23 in which a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control signal generation unit 115 are arranged is arranged in an area overlapping with a non-pixel area in a plan view. (described above in FIG. 2) are arranged.
• the arrangement of the pixel output signal line 113, the arrangement of the readout circuit 112, and the arrangement of the output circuit 114 are not limited to those shown in FIG. 3.
• the pixel output signal line 113 may be arranged to extend in the row direction, and may be shared by a plurality of pixel signal processing units 103 in a corresponding row.
• the readout circuit 112 may be arranged so that the pixel output signal line 113 of each row is connected.
• FIG. 5 is a schematic block diagram showing a configuration example of one pixel of the photoelectric conversion section 102 and the pixel signal processing section 103 according to the present embodiment.
• FIG. 5 schematically shows a more specific configuration example including the connection relationship between the photoelectric conversion section 102 arranged on the sensor board 11 and the pixel signal processing section 103 arranged on the circuit board 21. Note that in FIG. 5, drive lines between the vertical scanning circuit 110 and the pixel signal processing section 103 in FIG. 4 are shown as drive lines 213, 214, and 215.
• the photoelectric conversion unit 102 has an APD 201.
• the pixel signal processing section 103 includes a quench element 202, a waveform shaping section 210, a counter circuit 211, a selection circuit 212, and a gating circuit 216. Note that the pixel signal processing unit 103 only needs to include at least one of a waveform shaping unit 210, a counter circuit 211, a selection circuit 212, and a gating circuit 216.
• the APD 201 generates charges according to incident light by photoelectric conversion.
• a voltage VL first voltage
• the cathode of the APD 201 is connected to the first terminal of the quench element 202 and the input terminal of the waveform shaping section 210.
• the cathode of the APD 201 is supplied with a voltage VH (second voltage) higher than the voltage VL supplied to the anode.
• VH second voltage
• a reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs an avalanche multiplication operation.
• the operation modes when a reverse bias voltage is supplied to the APD 201 include Geiger mode and linear mode.
• Geiger mode is a mode in which the anode and cathode potential difference is greater than the breakdown voltage
• linear mode is a mode in which the anode and cathode potential difference is near or below the breakdown voltage.
• APD that operates in Geiger mode
• the voltage VL first voltage
• VH second voltage
• APD 201 may be operated in linear mode or Geiger mode. In the case of SPAD, the potential difference is larger and the effect of avalanche multiplication becomes more pronounced than in linear mode APD, so SPAD is preferable.
• the quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication.
• the quench element 202 suppresses the voltage supplied to the APD 201 to suppress avalanche multiplication (quench operation). Furthermore, the quench element 202 returns the voltage supplied to the APD 201 to the voltage VH by passing a current corresponding to the voltage drop due to the quench operation (recharge operation).
• Quench element 202 may be, for example, a resistive element.
• the waveform shaping section 210 shapes the potential change at the cathode of the APD 201 obtained during photon detection and outputs a pulse signal.
• an inverter circuit is used as the waveform shaping section 210.
• FIG. 5 shows an example in which one inverter is used as the waveform shaping section 210
• the waveform shaping section 210 may also be a circuit in which a plurality of inverters are connected in series, or other circuits having a waveform shaping effect. It may be a circuit.
• the gating circuit 216 is a circuit that performs gating to allow the pulse signal output from the waveform shaping section 210 to pass for a predetermined period. During a period in which the pulse signal can pass through the gating circuit 216, photons incident on the APD 201 are counted by the counter circuit 211 at the subsequent stage. Therefore, the gating circuit 216 controls the exposure period during which signal generation is performed in the pixel 101 based on the incident light. The period during which the pulse signal is passed is controlled by a control signal supplied from the vertical scanning circuit 110 via the drive line 215.
• FIG. 5 shows an example in which one AND circuit is used as the gating circuit 216. A pulse signal and a control signal are input to two input terminals of the AND circuit.
• the AND circuit outputs these logical products to the counter circuit 211.
• the gating circuit 216 only needs to realize gating, and may have a circuit configuration other than an AND circuit.
• the waveform shaping section 210 and the gating circuit 216 may be integrated by using a logic circuit such as a NAND circuit.
• the counter circuit 211 counts the pulse signals output from the waveform shaping section 210 via the gating circuit 216 and holds a digital signal indicating the count value. Furthermore, when a control signal is supplied from the vertical scanning circuit 110 via the drive line 213, the counter circuit 211 resets the held signal.
• a control signal is supplied to the selection circuit 212 from the vertical scanning circuit 110 shown in FIG. 4 via the drive line 214 shown in FIG.
• the selection circuit 212 switches between electrical connection and disconnection between the counter circuit 211 and the pixel output signal line 113.
• the selection circuit 212 includes, for example, a buffer circuit for outputting a signal according to the value held in the counter circuit 211.
• the selection circuit 212 switches between electrical connection and disconnection between the counter circuit 211 and the pixel output signal line 113, it also controls the signal output to the pixel output signal line 113.
• the method to do this is not limited to this.
• a switch such as a transistor at a node between the quench element 202 and the APD 201, or between the photoelectric conversion unit 102 and the pixel signal processing unit 103, and switching between electrical connection and disconnection
• the pixel output may also be controlled.
• the signal output to the pixel output signal line 113 may be controlled by changing the value of the voltage VH or voltage VL supplied to the photoelectric conversion unit 102 using a switch such as a transistor.
• FIGS. 6A, 6B, and 6C are diagrams explaining the operation of the APD 201 according to this embodiment.
• FIG. 6A is a diagram showing the APD 201, quench element 202, and waveform shaping section 210 extracted from FIG. 5.
• the connection node between the input terminals of the APD 201, the quench element 202, and the waveform shaping section 210 is designated as nodeA.
• the output side of the waveform shaping section 210 is designated as nodeB.
• FIG. 6B is a graph showing temporal changes in the potential of node A in FIG. 6A.
• FIG. 6C is a graph showing a temporal change in the potential of nodeB in FIG. 6A.
• a voltage of VH-VL is applied to the APD 201 in FIG. 6A.
• avalanche multiplication occurs in the APD 201.
• an avalanche current flows through the quench element 202, and the potential of node A drops. After that, the amount of potential drop further increases, and the voltage applied to APD 201 gradually decreases.
• avalanche multiplication in the APD 201 stops at time t2. This prevents the voltage level of node A from dropping below a certain constant value. Thereafter, during the period from time t2 to time t3, a current that compensates for the voltage drop flows from the node of voltage VH to node A, and at time t3, node A settles to the original potential.
• the potential of node B becomes high level during a period in which the potential of node A is lower than a certain threshold value.
• the waveform of the drop in the potential of node A caused by the incidence of photons is shaped by the waveform shaping section 210 and output as a pulse to node B.
• FIG. 7 is a functional block diagram showing a schematic configuration example of the distance image generation device 30 according to the present embodiment.
• FIG. 7 shows a more detailed configuration of the light emitting device 31, light receiving device 32, and signal processing circuit 33 described in FIG.
• the light emitting device 31 has a pulsed light source 311 and a light source control section 312.
• the pulsed light source 311 is a light source such as a semiconductor laser device that emits pulsed light throughout the distance measurement range.
• the pulsed light source 311 may be a surface light source such as a surface emitting laser.
• the light source control unit 312 is a control circuit that controls the light emission timing of the pulsed light source 311.
• the light receiving device 32 includes an imaging section 321, a gate pulse generation section 322, a microframe reading section 323, a microframe addition section 324, an addition number control section 325, and a subframe output section 326.
• the imaging unit 321 may be a photoelectric conversion device including a pixel array in which pixel circuits including the APD 201 are arranged in a two-dimensional manner. Thereby, the distance image generation device 30 can acquire a two-dimensional distance image.
• the gate pulse generation unit 322 is a control circuit that outputs a control signal that controls the drive timing of the imaging unit 321. Furthermore, the gate pulse generation section 322 synchronizes the pulsed light source 311 and the imaging section 321 by transmitting and receiving control signals to and from the light source control section 312 . Thereby, imaging can be performed while controlling the time difference from the time when light is emitted from the pulsed light source 311 to the time when the light is received by the imaging unit 321. In this embodiment, the gate pulse generation section 322 drives the imaging section 321 as a global gate.
• Global gate driving is a driving method that simultaneously detects incident light in several pixels (pixel group) in the imaging unit 321 during the same exposure period based on the emission time of pulsed light from the pulsed light source 311.
• incident light is repeatedly detected while the batch exposure timing is sequentially shifted.
• each pixel of the imaging unit 321 simultaneously generates a 1-bit signal indicating the presence or absence of an incident photon in each of the plurality of exposure periods.
• This global gate drive is realized by inputting a high-level signal to the input terminal of the gating circuit 216 of the plurality of pixels 101 during the gating period based on the control signal from the gate pulse generation unit 322. Ru. Note that in the process described later, one group of the plurality of pixels 101 detects incident light during the same exposure period, and another one group of the plurality of pixels 101 detects incident light during a different exposure period. Although incident light is detected, such a driving method is also included in global gate driving.
• the microframe readout section 323, the microframe addition section 324, and the addition number control section 325 are signal processing circuits that read out a 1-bit signal forming a microframe from the imaging section 321 and perform predetermined signal processing. Details of the operation of each of these parts will be described later with reference to FIG.
• the subframe output unit 326 is an interface that outputs a signal from the light receiving device 32 to the signal processing circuit 33 in accordance with a predetermined standard.
• the subframe output unit 326 transmits a signal from the memory in the light receiving device 32 to the memory in the signal processing circuit 33.
• the functions of these parts can be realized by the counter circuit 211, the selection circuit 212, the gating circuit 216 in FIG. 5, the readout circuit 112 and the output circuit 114 in FIG. 4, and the like.
• the signal processing circuit 33 has a subframe group storage section 331 and a distance image generation section 332.
• the signal processing circuit 33 is a computer that includes a processor that operates as a distance image generation section 332, a memory that operates as a subframe group storage section 331, and the like. The operation of each of these parts will also be described later with reference to FIG.
• FIG. 8 shows a schematic diagram of the acquisition period of a distance measurement frame corresponding to a distance image, a subframe used for generating the distance measurement frame, and a microframe used for generating the subframe by arranging blocks in the horizontal direction. is shown.
• the horizontal direction in FIG. 8 indicates the passage of time, and one block indicates the acquisition period of one ranging frame, subframe, or microframe.
• the distance measurement frame F1 corresponds to one distance image. That is, the distance measurement frame F1 has information corresponding to the distance to the target object X calculated from the time difference between light emission and light reception for each of the plurality of pixels. In this embodiment, it is assumed that the distance image is acquired as a moving image, and one distance measurement frame F1 is repeatedly acquired every time one distance measurement frame period T1 elapses.
• One ranging frame F1 is generated from multiple subframes F2.
• One ranging frame period T1 includes a plurality of subframe periods T2.
• One subframe F2 is repeatedly acquired every time one subframe period T2 elapses.
• the subframe F2 is composed of a multi-bit signal corresponding to the amount of light incident on the subframe period T2.
• One subframe F2 is generated from multiple microframes F3.
• One subframe period T2 includes multiple microframe periods T3.
• One microframe F3 is repeatedly acquired every time one microframe period T3 elapses.
• the microframe F3 is composed of a 1-bit signal indicating the presence or absence of light incident on the photoelectric conversion element during the microframe period T3.
• one subframe F2 of multi-bit signals is generated. Accordingly, one subframe F2 can include a multi-bit signal corresponding to the number of microframes in which incident light was detected within the subframe period T2.
• a plurality of subframes F2 with different periods for acquiring incident light are acquired.
• This signal acquisition time can be associated with the distance from the distance image generation device to the distance measurement target.
• the signal acquisition time at which the signal value is maximum can be determined from the distribution of the signal acquisition time and the signal values of the plurality of subframes F2. Since it is estimated that the reflected light is incident on the imaging unit 321 at the time when the signal value is maximum, the distance is calculated by converting the signal acquisition time at which the signal value is maximum into the distance to the object X. be able to.
• a distance image can be generated by calculating the distance for each pixel and obtaining a two-dimensional distribution of distances.
• the length of the distance measurement frame period T1 required to obtain one distance measurement frame F1 depends on the number of subframes F2. Since the number of subframes F2 is a parameter corresponding to the number of ranging points, there is a trade-off relationship between distance resolution and frame rate.
• FIG. 9 shows a method of driving the distance image generation device during one distance measurement frame period T1. The driving method of this embodiment will be explained along the flowchart of FIG.
• the processing from "start” to “end” indicates the processing performed during the ranging frame period T1 in which one ranging frame F1 in FIG. 8 is acquired.
• One cycle of processing in the loop from step S11 to step S15 indicates processing performed during subframe period T2 for acquiring one subframe F2 in FIG.
• One cycle of processing in the loop from step S11 to step S13 indicates processing performed during microframe period T3 for acquiring one microframe F3 in FIG.
• step S11 the light source control unit 312 controls the pulsed light source 311 to emit pulsed light within a predetermined distance measurement range.
• the gate pulse generation section 322 controls the imaging section 321 to start detecting the incident light by global gate driving.
• step S12 the microframe reading unit 323 reads microframes from the imaging unit 321 every time the microframe period elapses.
• the read microframe is held in the memory of the microframe adder 324.
• This memory has a storage capacity that can hold multiple bits of data for each pixel.
• the microframe addition unit 324 sequentially adds the value of the microframe to the value held in the memory every time the microframe is read. Thereby, the microframe addition unit 324 adds a plurality of microframes within the subframe period to generate a subframe.
• the number of additions in the microframe addition section 324 is controlled by an addition number control section 325. For this control, the addition number control unit 325 holds information on the preset number of additions.
• the microframe readout unit 323 functions as an acquisition unit that acquires a microframe composed of a 1-bit signal based on light incident on the photoelectric conversion element.
• the microframe addition unit 324 functions as a synthesis unit that synthesizes a plurality of microframes acquired in different periods. For example, if the number of additions is 64, a subframe signal having 6-bit gradation can be generated by combining 64 microframes.
• step S13 the microframe addition unit 324 determines whether or not the addition of microframes a preset number of times has been completed. If the addition of the set number of microframes has not been completed (NO in step S13), the process moves to step S11, and the next microframe is read. If the addition of the set number of microframes has been completed (YES in step S13), the process moves to step S14.
• step S14 the subframe output unit 326 reads out the subframes for which addition has been completed from the memory of the microframe addition unit 324, and outputs them to the subframe group storage unit 331.
• the subframe group storage section 331 stores the subframes output from the subframe output section 326.
• the subframe group storage unit 331 is configured to be able to individually store a plurality of subframes used to generate one ranging frame for each subframe period.
• step S15 the signal processing circuit 33 determines whether the subframe group storage unit 331 has acquired a predetermined number of subframes (ie, the number of ranging points). If the acquisition of subframes for the number of distance measurement points has not been completed (NO in step S15), the process moves to step S11, where multiple microframes are acquired and added again in order to read the next subframe. be exposed. In this case, similar processing is performed by shifting the start time of global gate drive with respect to the light emission time by one subframe period (gate shift). If the acquisition of subframes for the number of distance measurement points has been completed (YES in step S15), the process moves to step S16. Through the loop from step S11 to step S15, subframes corresponding to the number of distance measurement points are acquired.
• a predetermined number of subframes ie, the number of ranging points.
• the distance image generation unit 332 acquires a plurality of subframes in one distance measurement frame period from the subframe group storage unit 331.
• the distance image generation unit 332 generates a distance image showing a two-dimensional distribution of distances by calculating the distance corresponding to the subframe with the maximum signal value for each pixel.
• the distance image generation unit 332 then outputs the distance image to a device external to the signal processing circuit 33. This distance image can be used, for example, to detect the surrounding environment of the vehicle.
• the distance image generation unit 332 may store the distance image in a memory inside the distance image generation device.
• FIG. 10A is a schematic diagram showing the arrangement of pixel groups in the pixel array according to this embodiment
• FIG. 10B is a drive timing chart showing the timing of gate pulses according to this embodiment.
• the pixel array of this embodiment includes a first pixel group 327A ("A" in FIG. 10A), a second pixel group 327B ("B" in FIG. 10A), a third pixel group 327C ("C” in FIG. 10A), and a third pixel group 327C (“C” in FIG. 10A). It includes a fourth pixel group 327D ("D" in FIG. 10A). As shown in FIG. 10A, the first pixel group 327A, the second pixel group 327B, the third pixel group 327C, and the fourth pixel group 327D form a two-dimensional repeating array with four pixels as one block. There is.
• Light emission in FIG. 10B indicates the light emission timing of the pulsed light source 311. As shown in FIG. 10B, the pulsed light source 311 emits light at regular intervals under the control of the light source control section 312. This cycle corresponds to the length of one microframe period during which microframe acquisition is performed once.
• PG_A ” to “ PG_D ” in FIG. 10B indicate input timings of multiple types of gate pulses input from the gate pulse generation unit 322 to the imaging unit 321.
• PG_A ", " PG_B “, “ PG_C “, and “ PG_D” are used for controlling the first pixel group 327A, the second pixel group 327B, the third pixel group 327C, and the fourth pixel group 327D, respectively.
• Different gate pulses are shown.
• the gate pulses synchronized with the light emission timing L01 corresponding to the first pixel group 327A, the second pixel group 327B, the third pixel group 327C, and the fourth pixel group 327D are respectively gate pulses G01, G02, G03, and G04. do.
• the gate pulses G01, G02, G03, and G04 become high level after a predetermined time has elapsed since the pulsed light source 311 emits light, but the periods in which they become high level within one microframe period are different from each other.
• the gate pulses G01, G02, G03, and G04 go to a high level at times when a first time, a second time, a third time, and a fourth time, which are different from each other, have passed since the pulsed light source 311 emitted light. All of the first to fourth times are shorter than the length of the microframe period.
• the exposure time in each of the first pixel group 327A, second pixel group 327B, third pixel group 327C, and fourth pixel group 327D can be changed. can be made different. Therefore, four types of distance measurement points can be measured within one microframe period.
• one subframe is generated by adding multiple microframes. That is, the first pixel group 327A, the second pixel group 327B, the third pixel group 327C, and the fourth pixel group 327D each output signals for generating subframes of different ranging points.
• the timing of the gate pulse in the first pixel group 327A is gate shifted by a predetermined time interval from the timing of the gate pulse of the current subframe period. be done. Further, the gate shift is similarly performed for the second pixel group 327B, the third pixel group 327C, and the fourth pixel group 327D. By such a gate shift, a predetermined period corresponding to each subframe period can be set as an exposure period.
• the frame rate can be improved without reducing the number of distance measurement points (range resolution).
• a photoelectric conversion device with an improved frame rate while ensuring appropriate distance resolution is provided.
• the number of pixel group types is set to four, but it is sufficient that the number of pixel group types is at least two, and even if the number is other than four, the effect of improving the frame rate can be achieved. can get.
• the number of photoelectric conversion elements included in one pixel group may be at least one. That is, for two photoelectric conversion elements (a first photoelectric conversion element and a second photoelectric conversion element), in one microframe period, the first exposure period of the first photoelectric conversion element and the second exposure period of the second photoelectric conversion element If they are different from each other, the above effect can be obtained.
• the arrangement of the pixel groups is not limited to that shown in FIG. 10A, and can be changed as appropriate.
• the in-plane resolution of the distance measurement image may be reduced. Therefore, a method may be further applied to reduce the influence on the in-plane resolution by complementing pixels with missing information in the distance measurement image with surrounding pixels.
• FIG. 11A is a schematic diagram showing the arrangement of pixel groups in the pixel array according to this embodiment
• FIG. 11B is a drive timing chart showing the timing of gate pulses according to this embodiment.
• the pixel array of this embodiment includes a first pixel group 328A ("A" in FIG. 11A), a second pixel group 328B ("B" in FIG. 11A), a third pixel group 328C ("C” in FIG. 11A), and It includes a fourth pixel group 328D ("D" in FIG. 11A).
• the first pixel group 328A and the second pixel group 328B are configured to be sensitive to light of the first wavelength.
• the third pixel group 328C and the fourth pixel group 328D (second photoelectric conversion element) are configured to be sensitive to light of a second wavelength different from the first wavelength.
• the first pixel group 328A and the second pixel group 328B are provided with a first color filter that transmits light of the first wavelength
• the third pixel group 328C and the fourth pixel group 328D are provided with a first color filter that transmits light of a first wavelength
• a second color filter that transmits light of a second wavelength is arranged. Note that the ranges of the first wavelength and the second wavelength in this embodiment may be, for example, both in the infrared region.
• the pulsed light source 311 of this embodiment is configured to be able to emit light of the first wavelength and light of the second wavelength individually at different periods.
• “Light emission (first wavelength)” in FIG. 11B indicates the timing of light emission of the first wavelength by the pulsed light source 311.
• “Light emission (second wavelength)” in FIG. 11B indicates the timing of light emission of the second wavelength by the pulsed light source 311.
• the light of the first wavelength and the light of the second wavelength have different emission period lengths.
• the length of the light emission period of the second wavelength light is twice the length of the light emission period of the first wavelength light.
• gate pulses G05 and G06 are input at timings synchronized with the light emission timing L02 of the first wavelength light.
• gate pulses G07 and G08 are inputted at a timing delayed by one cycle of light emission of the first wavelength from the light emission timing L03 of the second wavelength light. Therefore, the microframe acquisition period for the first pixel group 328A and the second pixel group 328B is different from the microframe acquisition period for the third pixel group 328C and the fourth pixel group 328D.
• the first pixel group 328A and the second pixel group 328B are used as a pixel group for short distance (first distance range) distance measurement
• the third pixel group 328C and the fourth pixel group 328D are used as a pixel group for distance measurement at a long distance (second distance range).
• photoelectric conversion allows multiple different ranging ranges to be acquired at once without reducing the frame rate.
• Equipment is provided.
• the bit depth of the subframe signal obtained from the pixel group for short distance measurement may be different from the bit depth of the subframe signal obtained from the pixel group for long distance measurement.
• the number of additions of the short-range ranging signal is 64
• a 6-bit deep short-range subframe is obtained.
• the number of times the light with the second wavelength is emitted is half the number of times the light with the first wavelength is emitted, so the number of additions of the long-distance ranging signal is 32 at most. Therefore, the far-field subframe is 5 bits deep. In this way, when the bit depths of two signals are different, the bit depths may be adjusted.
• the pulsed light source 311 there are two types of wavelengths of light emitted by the pulsed light source 311, but there may be three or more types of wavelengths. Further, the ratio of the periods of light having different wavelengths is not limited to twice, but can be set as appropriate.
• FIG. 12A is a schematic diagram showing the arrangement of pixel groups in the pixel array according to this embodiment
• FIG. 12B is a drive timing chart showing the timing of gate pulses according to this embodiment.
• the pixel array of this embodiment includes a first pixel group 329R ("R" in FIG. 12A), a second pixel group 329G ("G" in FIG. 12A), a third pixel group 329B ("B" in FIG. 12A), and It includes a fourth pixel group 329Z ("Z" in FIG. 12A).
• the first pixel group 329R, the second pixel group 329G, the third pixel group 329B, and the fourth pixel group 329Z are configured to be sensitive to light of different wavelengths. More specifically, a first color filter that transmits red light with a wavelength of 700 nm (first wavelength) is arranged in the first pixel group 329R (first photoelectric conversion element).
• a second color filter that transmits green light with a wavelength of 550 nm (third wavelength) is arranged in the second pixel group 329G (third photoelectric conversion element).
• a third color filter that transmits blue light with a wavelength of 430 nm (fourth wavelength) is arranged in the third pixel group 329B (fourth photoelectric conversion element).
• a fourth color filter that transmits infrared light having a wavelength of 1000 nm (second wavelength) is arranged in the fourth pixel group 329Z (second photoelectric conversion element).
• the pulsed light source 311 of this embodiment is configured to be able to emit light of infrared wavelength.
• Light emission (infrared wavelength) indicates the timing of light emission of infrared wavelength by the pulsed light source 311.
• PG_Z indicates a gate pulse for controlling the fourth pixel group 329Z.
• PG_RGB indicates a control gate pulse common to the first pixel group 329R, the second pixel group 329G, and the third pixel group 329B.
• the gate pulse G09 is input at a timing synchronized with the light emission timing L04 of the infrared wavelength light.
• the gate pulse G10 is input to the first pixel group 329R, the second pixel group 329G, and the third pixel group 329B over a predetermined exposure period during which imaging is performed.
• signals including information of each color of red, green, and blue are obtained in the first pixel group 329R, the second pixel group 329G, and the third pixel group 329B, so that a color image can be generated using these signals. Can be done.
• the three colors red, green, and blue are just examples, and if the acquired signal includes information on a plurality of colors in the visible region, a color image can be generated.
• distance information is obtained in the fourth pixel group 329Z by inputting the gate pulse G09 at a timing synchronized with light emission.
• a distance measurement signal and a color image generation signal can be acquired within the same subframe period. Therefore, a photoelectric conversion device is provided that can acquire color images and distance images while maintaining distance resolution and frame rate.
• FIG. 13 is a schematic block diagram of pixels according to this embodiment.
• a transistor 217 is arranged as a specific example of the quench element 202 in FIG.
• Transistor 217 is an NMOS transistor.
• a source of the transistor 217 is connected to a connection node between the APD 201 and the input terminal of the waveform shaping section 210.
• a voltage VH is supplied to the drain of the transistor 217.
• a recharge pulse is input to the gate of the transistor 217 from the vertical scanning circuit 110 via a drive line 218.
• the transistor 217 is turned on, and the cathode potential of the APD 201 is reset to a potential that allows avalanche multiplication. This operation is called a recharge operation.
• FIG. 14 is a drive timing diagram showing the timing of gate pulses and recharge pulses according to this embodiment. It is assumed that the configuration of the pixel group in the pixel array of this embodiment is the same as that of the third embodiment. Further, the pulsed light source 311 of this embodiment is also configured to emit light of infrared wavelength, similarly to the third embodiment. " PR_Z " indicates a recharge pulse for controlling the fourth pixel group 329Z. “ PR_RGB ” indicates a control recharge pulse common to the first pixel group 329R, the second pixel group 329G, and the third pixel group 329B.
• a recharge pulse corresponding to the fourth pixel group 329Z and synchronized with the light emission timing L05 is referred to as a recharge pulse R01.
• a recharge pulse synchronized with the light emission timing L05 corresponding to the first pixel group 329R, the second pixel group 329G, and the third pixel group 329B is referred to as a recharge pulse R02.
• the recharge pulses R01 and R02 become high level after a predetermined time has elapsed since the pulsed light source 311 emits light, but the periods in which they become high level within one microframe period are different from each other.
• the gate pulse G11 corresponding to the fourth pixel group 329Z becomes high level. Further, after the recharge pulse R02 falls, the gate pulse G12 corresponding to the first pixel group 329R, the second pixel group 329G, and the third pixel group 329B becomes high level.
• the period from the fall of the recharge pulse R02 to the fall of the gate pulse G12 is a photon detection period for capturing a color image. . That is, in the configuration of this embodiment, by appropriately setting the timings of the recharge pulse R02 and the gate pulse G12, it is possible to control the charge accumulation time during color image capturing.
• the gate pulse G10 for capturing color images is maintained at a high level, when the subject is a moving object, image blur or mixing of signals from different subjects may occur. There is. However, in the configuration of this embodiment, the influence of these factors can be reduced by appropriately setting the charge accumulation time when capturing a color image. Therefore, according to this embodiment, in addition to obtaining the same effects as the third embodiment, it is possible to further improve image quality.
• FIG. 15 is a functional block diagram showing a schematic configuration example of a distance measuring device according to this embodiment.
• the light receiving device 32 is further provided with a flag calculation section 327
• the signal processing circuit 33 is further provided with a map generation section 333.
• the other configurations are similar to any of the first to fourth embodiments. In the following, operations of the flag calculation section 327 and map generation section 333 of this embodiment will be explained based on the configuration of the first embodiment.
• the flag calculation unit 327 performs logical operations such as AND on the 1-bit signals for microframes output from a plurality of pixel groups, and outputs the result as an external light flag.
• the external light flag is a 1-bit signal indicating the presence or absence of environmental light (external light) emitted from sources other than the light emitting device 31.
• the flag calculation unit 327 calculates the AND of four 1-bit signals output from each of the first pixel group 327A, the second pixel group 327B, the third pixel group 327C, and the fourth pixel group 327D. In this case, when the logical values of the four 1-bit signals are all "1", the logical value of the calculation result is "1".
• the pixel group to be added may be selected from a plurality of pixels adjacent to each other, such as the four pixels labeled with symbols in FIG. 10A.
• the flag calculation unit 327 generates a plurality of external light flags by performing similar logical operation processing on each of a plurality of regions selected from the entire pixel array.
• the map generation unit 333 acquires the plurality of external light flags generated by the flag calculation unit 327, generates an external light map in which the positions of the pixel array are associated with the logical values of the external light flags, and Output to.
• FIG. 16 is a schematic diagram showing an example of the external light map according to the present embodiment.
• FIG. 16 shows an image captured by a photoelectric conversion device mounted on a vehicle, with boxes marked with "1" superimposed only on areas where the logical value of the external light flag is "1". It is. As shown in FIG. 16, the logical value of the external light flag is "1" in a blue sky region where brightness is often high due to the influence of sunlight.
• accuracy in methods that measure distance by emitting light from a light source and detecting the light reflected by an object, accuracy can be affected by external light such as sunlight that exists independently of the light from the light source.
• external light such as sunlight that exists independently of the light from the light source.
• external light that is not reflected light from an object may serve as a false signal indicating the object.
• Detection values based on external light often depend on position. For example, the area that receives external light from the blue sky as shown in FIG. 16 extends over a wide area at the top of the image.
• an external light flag by logical operation of 1-bit signals output from multiple pixels, pixels that are likely to be exposed to external light that has a certain tendency as described above are identified. can be detected.
• information indicating areas where external light is likely to be incident can be output to the outside.
• the external light map can be used in subsequent image processing.
• a photoelectric conversion device that can output information indicating signal accuracy to the outside is provided.
• logical product is used as an example of the logical operation performed in the flag calculation unit 327, but the present invention is not limited to this, and any process that outputs a value based on signals from a plurality of pixels may be used.
• the logical operation may be a logical sum.
• the device 80 includes a distance measuring unit 803, which is an example of the distance image generating device of the embodiment described above, and a signal processing device (processing device) that processes a signal from the distance measuring unit 803.
• the device 80 includes a distance measurement unit 803 that measures the distance to the object, and a collision determination unit 804 that determines whether there is a possibility of a collision based on the measured distance.
• the distance measurement unit 803 is an example of distance information acquisition means that acquires distance information to the target object. That is, the distance information is information regarding the distance to the target object, etc.
• the collision determination unit 804 may determine the possibility of collision using distance information.
• the device 80 is connected to a vehicle information acquisition device 810, and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. Further, a control ECU 820 is connected to the device 80, which is a control device that outputs a control signal for generating a braking force on the vehicle based on the determination result of the collision determination unit 804. The device 80 is also connected to a warning device 830 that issues a warning to the driver based on the determination result of the collision determination unit 804. For example, if the collision determination unit 804 determines that there is a high possibility of a collision, the control ECU 820 performs vehicle control to avoid the collision and reduce damage by applying the brakes, releasing the accelerator, or suppressing engine output.
• the alarm device 830 warns the user by sounding an alarm, displaying alarm information on the screen of a car navigation system, etc., or applying vibration to the seat belt or steering wheel.
• These devices of the device 80 function as a mobile object control unit that controls the operation of controlling the vehicle as described above.
• the distance around the vehicle is measured using the device 80.
• FIG. 17B shows equipment for measuring distance in front of the vehicle (distance measurement range 850).
• Vehicle information acquisition device 810 as distance measurement control means sends an instruction to device 80 or distance measurement unit 803 to perform a distance measurement operation. With such a configuration, the accuracy of distance measurement can be further improved.
• control to avoid collision with other vehicles has been described, but it is also applicable to control to automatically drive while following other vehicles, control to automatically drive to avoid running out of the lane, etc.
• the device is not limited to vehicles such as automobiles, but can be applied to moving objects (mobile devices) such as ships, aircraft, artificial satellites, industrial robots, and consumer robots.
• mobile devices such as ships, aircraft, artificial satellites, industrial robots, and consumer robots.
• present invention can be applied not only to mobile objects but also to a wide range of devices that utilize object recognition or biometric recognition, such as intelligent transportation systems (ITS) and monitoring systems.
• ITS intelligent transportation systems
• the present invention provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in a computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
• a circuit for example, ASIC
• Imaging section 322 Gate pulse generation section 323 Microframe reading section 324 Microframe addition section

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Transforming Light Signals Into Electric Signals (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=90191084

Family Applications (1)

Country Status (6)

Families Citing this family (1)

Citations (5)

• 2022
  • 2022-09-08 JP JP2022142845A patent/JP2024038655A/ja active Pending
• 2023
  • 2023-08-23 EP EP23862939.8A patent/EP4586633A1/en active Pending
  • 2023-08-23 CN CN202380063898.4A patent/CN119817110A/zh active Pending
  • 2023-08-23 WO PCT/JP2023/030359 patent/WO2024053400A1/ja not_active Ceased
  • 2023-08-23 KR KR1020257010535A patent/KR20250057879A/ko active Pending
• 2025
  • 2025-02-26 US US19/064,353 patent/US20250203238A1/en active Pending

Patent Citations (5)

Also Published As

Similar Documents

Legal Events