JP7401986B2 - Time-resolved image sensor and imaging unit for grayscale imaging, and grayscale image generation method - Google Patents

Description

The present invention relates to an image sensor, and more particularly to a time-of-flight (TOF) image sensor and imaging unit that generates a grayscale image from accumulated photon detection events and a method for generating a grayscale image of an object. .

Three-dimensional (3D) imaging systems are increasingly being used in a wide range of diverse applications such as industrial manufacturing, video games, computer graphics, robotic surgery, consumer displays, surveillance video, 3D modeling, real estate sales, etc. Used a lot. Existing 3D imaging technologies include, for example, time-of-flight (TOF)-based distance imaging, stereo vision systems, and structured light (SL) methods.

In the TOF method, the distance to the 3D object is measured based on the known speed of light and the round trip time required for the light signal to travel between the camera and the 3D object for each point in the image. Determined by TOF cameras can use a scannerless approach to capture the entire scene with each laser or light pulse. Some exemplary applications of TOF methods are improved vehicle applications such as active pedestrian protection and pre-collision detection based on real-time range images, interaction with games on video game consoles, etc. This includes tracking people's movements, identifying objects such as items on a conveyor belt in industrial machine vision, and helping robots locate items.

In stereoscopic-imaging or stereo vision systems, two cameras placed horizontally to each other are used to obtain two different views of a scene or a 3D object within a scene. By comparing these two images, relative depth information for the 3D object is obtained. Stereo vision is particularly important in fields such as robotics, where it extracts information about the relative position of 3D objects in the vicinity of an autonomous system/robot. Other applications for robotics include object recognition. Here, the stereoscopic depth information allows the robot system to separate the hidden image components. Without this, the robot would not be able to distinguish between two objects, with one object positioned in front of the other and partially or completely obscuring the other. 3D stereo displays may also be used in entertainment and automated systems.

In the SL method, the 3D shape of an object is measured using an illuminated light pattern and a camera for imaging. In SL methods, a known pattern of light (often a grid or pattern of horizontal lines or parallel stripes) is illuminated onto a scene or a 3D object within the scene. The illuminated pattern is deformed or transformed upon impacting the surface of the 3D object. Such a transformation allows the SL vision system to calculate depth and surface information of the object. Therefore, illuminating a 3D surface with a narrow band of light produces illumination lines that appear distorted from a different perspective than that of the projector and are used for geometrical reconstruction of the illuminated surface morphology. SL-based 3D imaging can be used by police to capture fingerprints in a 3D scene, inline inspection of components during manufacturing processes, and in healthcare for live measurements of human morphology and/or skin microstructure. used in a variety of applications such as

US Patent No. 9516244 US Patent Application Publication No. 2010/0127160 US Patent Application Publication No. 2017/0052065

NICLASS, Cristiano, et al. , “Cmos Imager Based on Single Photon Avalanche Diodes,”The 13th International Conference on Solid-State Sensors, Actuators and Microsystems, 2005, Digest of Technical Papers, TRANSDUCERS '05, August 22, 2005, 5 pages. NICLASS, Cristiano, et al. , “TOWARD A 3 -D Camera Basle ON SINGLE PHOTON AVALANCHE DIODES,“ IEEE JOURNAL OF SELECTED TOPICS in QUANTICS in QUANTUMS ELECTRONICS, VOL . 10, No. 4, July/August 2004, pp. 796-802. STOPPA, David, et al. , “A CMOS 3-D Imager Based on Single Photon Avalanche Diode,” IEEE Transactions on Circuits and Systems, I: Regular Papers, V ol. 54, No. 1, January 2007, pp. 4-12.

The present invention has been made in view of the above-mentioned prior art, and an object of the present invention is to provide an image sensor and an imaging sensor that senses light pulses reflected from an object and generates both a 3D image and a 2D image of the object. The object of the present invention is to provide a method for generating grayscale images of units and objects.

In order to achieve the above object, an image sensor according to an aspect of the present invention detects, by at least one pixel, one or more photons reflected from an object in response to a light pulse directed toward the object. a time-resolved sensor including the at least one pixel that outputs a first signal and a second signal pair in response to the light pulse being reflected; a first ratio of the amplitude of the first signal of the pair to the sum of the amplitude of the first signal of the pair and the amplitude of the second signal of the pair; , a second ratio of the amplitude of the second signal of the pair to the sum of the amplitude of the first signal and the second signal of the pair, proportional to the time of flight of the detected one or more photons; is proportional to the time of flight of the detected one or more photons.

The processor may further determine a distance to the object based on the first signal and second signal pair.
The time-resolved sensor is responsive to detecting by the pixel one or more photons reflected from the object in response to a plurality of light pulses directed toward the object. the processor outputs a plurality of first and second signal pairs, one pair of signals corresponding to each light pulse; may determine a surface reflectance of the object on which is reflected.

An imaging unit according to an aspect of the present invention made to achieve the above object includes: a light source that illuminates an object with a series of light pulses directed toward a surface of the object; a time-resolved sensor comprising said at least one pixel that outputs a first signal and a second signal pair in response to detecting one or more photons corresponding to a light pulse reflected from a surface of said determining a distance to the object based on a first signal and a second signal pair; and determining a surface reflectance of the object from which the light pulse was reflected based on the first signal and second signal pair. a processor for determining, a first ratio of the amplitude of the first signal of the pair to the sum of the amplitude of the first signal and the amplitude of the second signal of the pair, a second ratio of the amplitude of the second signal of the pair to the sum of the amplitude of the first signal and the amplitude of the second signal of the pair is proportional to the time of flight of a photon of the detected one. It is proportional to the flight time of the photon.

The time-resolved sensor is responsive to detecting by the pixel one or more photons reflected from the object such that each of the first and second signal pairs is directed toward the object. outputting a plurality of first and second signal pairs corresponding to each of a plurality of light pulses, the processor detecting a plurality of surfaces of the object based on the corresponding first and second signal pairs; Reflectance may further be determined.
The processor may further generate a grayscale image of the object based on the plurality of surface reflectances.

A method of generating a grayscale image of an object according to the present invention, which has been made to achieve the above object, comprises the steps of: directing a series of light pulses from a light source toward the surface of the object; detecting one or more photons corresponding to a reflected light pulse; and in response to detecting the one or more photons by a time-resolved sensor synchronized to the light source, a first signal and a generating a pair of second signals; determining, by a processor, a distance to the object based on the pair of first and second signals; determining a surface reflectance of the object based on a pair of signals, the first signal of the pair relative to the sum of the amplitude of the first signal and the amplitude of the second signal of the pair. is proportional to the time-of-flight of the detected one or more photons, and the first ratio of the amplitude of the first signal of the pair to the amplitude of the second signal of the pair is proportional to the time of flight of the detected one or more photons. A second ratio of signal amplitude is proportional to the time of flight of the detected one or more photons.

The method includes, at the pixel, a plurality of light pulses directed toward the object, each of a plurality of pairs of first and second signals corresponding to a respective light pulse of the series of light pulses; detecting one or more photons reflected from the object; and determining, by the processor, a surface reflectance of the object based on at least one of the plurality of first and second signal pairs. The method may further include a step.
The method may further include generating, by the processor, at least one histogram of arrival times of photons detected by predetermined pixels of the plurality of pixels to generate the grayscale image.

According to the present invention, from the time of illuminating a light pulse toward an object until the light pulse reflected from the object is sensed, the amount of charged charge transferred is converted into a first signal, and the remaining charge of the charged charge is converted into a first signal. The quantity is converted into a second signal. A time of flight of the light pulse is calculated based on the first signal and the second signal, and a distance is calculated from the time of flight to generate a 3D image. Based on the first signal and the second signal, a power of the reflected light pulse is calculated, and a reflectance is calculated from the calculated power or the calculated distance to generate a 2D grayscale image of the object.

FIG. 1 is a schematic configuration diagram of an image sensor system. 2 is a configuration diagram for explaining an example of the operation of the image sensor system of FIG. 1. FIG. 3 is a flowchart illustrating an example of how 3D depth measurement is performed. FIG. 3 is a diagram illustrating an example of how points are scanned to measure 3D depth; 3 is a block diagram illustrating an example of the pixel of FIG. 2. FIG. FIG. 3 is a diagram showing an example of a pixel array structure. FIG. 7 is a diagram showing another example of a pixel array structure. FIG. 7 is a diagram showing still another example of a pixel array structure. FIG. 2 is a circuit diagram showing an example of a pixel. 8 is a timing diagram for explaining an example of an outline of a transfer mechanism of charges modulated by the pixels of FIG. 7; FIG. 8 is a timing diagram illustrating an example of different signals in the image sensor system of FIGS. 1 and 2 when the pixels of FIG. 7 are used in a pixel array for measuring TOF values; FIG. FIG. 2 is a diagram illustrating how a logical unit is implemented using pixels. 3 is a flowchart illustrating an example of how the TOF value is determined in the image sensor system of FIGS. 1 and 2. FIG. FIG. 2 is a layout diagram showing an example of a portion of an image sensor unit. FIG. 7 is a circuit diagram showing another example of pixels. 14 is a timing diagram illustrating another example of different signals of the image sensor system of FIGS. 1 and 2 when the pixels of FIG. 13 are used in a pixel array for measuring TOF values; FIG. FIG. 2 is a block diagram showing an example of a time-resolved sensor. 16 is a circuit diagram showing an example of a SPAD circuit of the time-resolved sensor of FIG. 15. FIG. 16 is a circuit diagram showing an example of a logic circuit of the time-resolved sensor of FIG. 15. FIG. 16 is a circuit diagram showing an example of a PPD circuit of the time-resolved sensor of FIG. 15. FIG. 16 is a relative signal timing diagram illustrating an example for the time-resolved sensor of FIG. 15; FIG. FIG. 3 is a block diagram showing another example of a time-resolved sensor. 21 is a circuit diagram showing an example of a second PPD circuit of the time-resolved sensor of FIG. 20. FIG. 21 is a relative signal timing diagram illustrating another example of the time-resolved sensor of FIG. 20; FIG. FIG. 7 is a circuit diagram showing still another example of a time-resolved sensor. 24 is a relative signal timing diagram showing still another example of the time-resolved sensor of FIG. 23. FIG. 24 is a flowchart illustrating an example of a method for resolving time using the time-resolving sensor of FIG. 23. FIG. It is a figure which shows an example of the trigger waveform output from SPAD. It is a histogram showing an example of the light detection time of the formed pixel. FIG. 6 is a diagram showing, as an example of a histogram, a window width indicating the FWHM of an emitted pulse (not shown), and for explaining that the maximum event count is determined. FIG. 26 is a diagram showing an example of a histogram and explaining that the trigger waveform output from the SPAD (FIG. 26) is convolved with the histogram to determine the maximum event count. 2 is a histogram showing an example of pixels. 1 is a flowchart of an example method for generating scene depth or extent, maps, and grayscale images. This is an image showing an example of a scene. 29a is a depth map showing an example of the scene shown in FIG. 29a. 29a is a grayscale image showing an example of the scene shown in FIG. 29a; FIG. FIG. 3 is a block diagram showing an example of the overall layout of the imaging system shown in FIGS. 1 and 2. FIG.

Hereinafter, specific examples of modes for carrying out the present invention will be described in detail with reference to the drawings. However, it will be understood by those skilled in the art that the invention may be practiced without being limited to these specific embodiments. Well-known methods, procedures, components, and circuits have not been described in detail in order not to obscure the present invention. Additionally, the present invention can be implemented to perform low power, 3D depth measurements on any imaging device or system, including, but not limited to, a smartphone, a user equipment (UE), and/or a laptop computer. be done.

References herein to "an embodiment" or "an embodiment" include references to "an embodiment" or "an embodiment" in which a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment described herein. It means to get. Therefore, appearances of the phrases "in one embodiment,""in an embodiment," or "according to one example" (or other phrases of similar meaning) in various parts of this specification all refer to the same implementation. It should not be used as a reference to examples. Note that the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In this context, the word "exemplary," as used herein, means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not to be construed as preferred or advantageous over other embodiments. Also, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Also, depending on the context described herein, singular terms may include corresponding plural terms, and plural terms may include corresponding singular terms. Similarly, hyphenated terms (e.g., "2-dimensional,""pre-defined,""pixel-specific," etc.) sometimes refer to corresponding unhyphenated versions (e.g., "2-dimensional,""pre-defined,""pixel-specific," etc.) ”, “predetermined”, “pixel specific”, etc.), and are used instead of items with English capital letters (e.g., “Counter Clock”, “Row Select”, "PIXOUT," etc.) may be used in place of the corresponding non-capitalized versions (eg, "counter clock,""rowselect,""pixout," etc.). Such occasional alternative uses shall not be deemed mutually exclusive.
It should be noted that, depending on the context described herein, singular terms may include corresponding plural terms, and plural terms may include corresponding singular terms. The various drawings (including component views) shown and described herein are for illustrative purposes only and are not limited to scale. Similarly, various waveforms and timing diagrams are shown for illustrative purposes only. For example, some dimensions of an element may be emphasized more than others for clarity. Also, where appropriate, reference signs may be repeated in the drawings to refer to corresponding and/or similar elements.

The terminology used herein is for the purpose of describing some embodiments only and is not intended as a limitation of the invention. As used herein, the singular term also includes the plural term, unless the context clearly dictates otherwise. As used herein, the terms "comprise" and/or "comprising" indicate the presence of the referenced feature, integer, step, act, element, and/or component. , one or more other features, integers, steps, acts, elements, components, and/or groups thereof. As used herein, terms such as "first", "second", etc. are used as labels for the nouns they follow and are not explicitly defined as such in any form of ordering (e.g. , spatial, temporal, logical, etc.). Also, the same reference numeral may be used in more than one drawing to indicate a part, component, block, circuit, unit, or module having the same or similar function. However, such use is for simplicity of explanation and ease of explanation only, and does not imply that the structure of such components, units, or structural details are the same across all embodiments. There is no suggestion that such commonly referenced components/modules are the only ways to implement some of the embodiments described herein.

When an element or hierarchy is referred to as being on top of, connected to, or combined with another element or hierarchy, that element or hierarchy is directly above or directly above another element or hierarchy. It is understood that they may be concatenated or directly coupled and that there are interrupting elements or hierarchies. Conversely, if an element is directly above, directly connected to, or referenced by being directly coupled to another element or hierarchy, there is no intervening element or hierarchy. Like reference numbers refer to similar elements throughout. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

As used herein, terms such as "first", "second", etc. are used as labels for the nouns that follow them, and when not explicitly defined as such, any form of ordering (e.g. , spatial, temporal, logical, etc.). Also, the use of the same reference numerals in two or more drawings refers to parts, components, blocks, circuits, units, or modules that have the same or similar functionality. However, such use is for simplicity of explanation and ease of explanation only, and does not imply that the structure of such components, units, or structural details are the same across all embodiments. , or that such commonly referenced components/modules are not the only ways to implement some of the embodiments described herein.

As shown in the drawings, spatially relative terms such as "below", "below", "below", "above", "superior", etc. are used here. is used for ease of explanation to describe the relationship of one element or feature to another element(s) or feature(s). It is understood that spatially relative terms are intended to include other orientations of the device in use or operation in addition to the orientation shown in the drawings. For example, when the device in the drawings is turned over, elements described as "below" or "beneath" other elements or features may be oriented "above" other elements or features. Accordingly, the term "below" includes both upward and downward directions. If the device is oriented differently (rotated 90 degrees or in other directions), the spatially relative descriptions used herein will be interpreted accordingly.

Unless defined differently, all terms (including technical and scientific terms) used herein are the same as commonly understood by one of ordinary skill in the art to which this invention pertains. have meaning. The same terms as defined in a commonly used dictionary should be construed to have meanings consistent with their meaning in the context of the relevant field, and are not expressly defined as such herein. should not be idealized or overly formalized.

As used herein, the term "module" refers to any combination of software, firmware, and/or hardware that is associated with the module and configured to provide the functionality described herein. Point. Software is implemented in software packages, code, and/or command sets or commands, and the term "hardware" as used in any implementation described herein includes, for example, programmable circuits, alone or in any combination. The hardware may include embedded hardwired circuitry, programmable circuitry, state machine circuitry, and/or firmware that stores commands executed by the hardware. Modules may be implemented collectively or individually as circuits forming part of a larger system, such as, but not limited to, an integrated circuit (IC), a system on a chip (SoC), etc. .

The 3D technology described above has many problems. For example, TOF-based 3D imaging systems require power to operate optical or electrical shutters. Such systems typically operate over ranges of a few meters to tens of meters, but the resolution of such systems decreases in short distance measurements and it is rarely possible to create 3D images at distances of about 1 meter. It's impossible. Therefore, TOF systems are not preferred for mobile phone-based camera applications where photos are taken mostly at close range. TOF sensors also require special pixels with pixel sizes typically larger than 7 μm. Such pixels are also sensitive to ambient light.

Stereoscopic imaging methods generally operate only on textured surfaces. This has high computational complexity due to the need to match features and find correspondences between images of stereo pairs of objects. This requires high system power and is not an appropriate characteristic for applications that require power savings, such as smartphones. Stereo imaging also requires two permanently mounted high-bit resolution sensors along with two lenses, making it suitable for applications on portable devices such as mobile phones or tablets where device real estate (device space) is at a premium. However, the entire assembly does not fit.

SL methods exhibit distance ambiguity and require high system power. For 3D depth measurements, SL methods may require multiple images with multiple patterns. All of these increase computational complexity and power consumption. Note that SL imaging requires a permanent image sensor with high bit resolution. Therefore, structured light-based systems may not be suitable for low-cost, low-power, compact image sensors in smartphones.

In contrast to 3D technology, some embodiments described herein provide for implementing low power 3D imaging systems for portable electronic devices such as smartphones, tablets, and user equipment (UE). A 2D imaging sensor according to some embodiments described herein captures 3D depth measurements with 2D RGB (red, green, blue) images and visible light laser scans, and captures 3D depth measurements with a 2D RGB (red, green, blue) image and a visible light laser scan, and captures ambient light during 3D depth measurements. can be blocked. Although the following techniques frequently refer to visible light lasers as light for point scanning and frequently refer to 2D RGB sensors as image/light capture devices, such references are for purposes of descriptive consistency. It is only for the purpose. The visible-light laser and RGB sensor-based examples described below may find applications in low-power, consumer-grade mobile electronic devices with cameras such as smartphones, tablets, and user equipment (UE). I can do it. However, the invention is not limited to the visible light RGB sensor-based examples mentioned below. Alternatively, in accordance with some embodiments described herein, point scan-based 3D depth measurement and ambient light blocking methods include, but are not limited to, (i) the laser source is red light (R); a 2D color (RGB) sensor with a green light (G) or blue light (B) laser or a visible light laser source producing a combination of such light; (ii) an infrared (IR) blocking filter; (iii) a near-infrared (NIR) laser with a 2D IR sensor; (iv) a NIR laser with a 2D NIR sensor; (v) a 2D RGB sensor (with IR blocking); (vi) NIR laser with 2D RGB sensor (no NIR blocking filter), (vii) 2D RGB-IR sensor with visible light or NIR laser, (viii) visible It can be accomplished using a number of different combinations of 2D sensors and laser light sources (for point scanning), such as 2D RGBW (red, green, blue, white) sensors with optical or NIR lasers.

For 3D depth measurements, all sensors operate as binary sensors in conjunction with laser scanning to reconstruct the 3D content. In some embodiments, the sensor pixel size can be as small as 1 μm. Note that due to the low bit resolution, the analog-to-digital converter (ADC) unit of the image sensor according to some embodiments described herein may be smaller than that required for a high bit resolution sensor in a typical 3D imaging system. may also require very little processing power. Due to the need for low processing power, the 3D imaging module according to the present invention requires low system power consumption and is therefore well suited for inclusion in low power consumption devices such as smartphones.

In some embodiments, the invention uses triangulation and point scanning with a laser light source for 3D depth measurement with a group of line sensors. The laser scanning plane and imaging plane are oriented using epipolar geometry. An image sensor according to one embodiment described herein uses time stamps to disambiguate the method of triangulation. Thus reducing the amount of depth computation and system power. The same image sensor (ie, each pixel of the image sensor) is typically used in 2D (RGB color or non-RGB) imaging mode, as well as in 3D laser scanning mode. However, in laser scanning mode, the resolution of the image sensor's ADC is reduced to a binary output (1 bit resolution). Readout speeds are thus increased and power consumption (e.g. due to switching within the ADC unit) is reduced in chips containing processing units associated with image sensors. The point scan method reduces latency for depth measurements and reduces motion blur by having the system take all measurements at once.

As mentioned above, in some embodiments the entire image sensor is used not only for 3D depth imaging using visible light laser scanning, but also for regular 2D RGB color imaging using e.g. ambient light. Ru. Such dual use of the same camera unit can save space and cost of the mobile device. In certain applications, visible light lasers for 3D applications may be better for user eye safety compared to near-infrared (NIR) lasers. The sensor is in the visible spectrum rather than the NIR spectrum, has higher quantum efficiency, and can derive lower power consumption from the light source. In one example, the dual-use image sensor typically operates as a 2D sensor in a linear mode of operation for 2D imaging. However, for 3D imaging, the sensor operates in linear mode under normal light conditions and in logarithmic mode under strong ambient light, blocking strong ambient light and using visible laser light sources. enable sustainable use. Ambient light blocking may also be necessary in NIR lasers, for example if the passband bandwidth of the IR blocking filter employed in the RGB sensor is not wide enough.

In summary, the present description describes time-to-charge conversion (TCC) with amplitude-modulated charge-transfer operation controlled by outputs from multiple adjacent SPADs at a pixel. TOF is determined using a pinned photodiode (PPD) at the pixel as a to-charge converter. When ambient light is high, there is a high probability that SPAD will be triggered by ambient photons instead of reflected photons (eg, within reflected light pulse 37). Reliance on such triggers can induce distance measurement errors. Therefore, in the description herein, if two or more SPADs are triggered in a very short, predefined time interval, such as when the electronic shutter is on, PPD charge transfer is stopped and the TOF only are recorded. As a result, all-weather autonomous driving systems in accordance with the teachings described herein provide improved visibility to drivers in difficult driving conditions, such as low light, fog, adverse weather, strong ambient light, etc. can do. In some embodiments, a driving system according to the teachings described herein can block up to 100 kLux of high ambient light. In some embodiments, a smaller pixel size and higher spatial resolution pixel structure is provided as a 1:1 ratio of SPAD to PPD. In some embodiments, the SPAD is biased below a breakdown voltage and used in an avalanche photodiode (APD) mode.

FIG. 1 is a schematic configuration diagram of an imaging system 150, and FIG. 2 is a configuration diagram for explaining an example of the operation of the image sensor system of FIG. 1. System 15 includes an imaging module 17 coupled to and in communication with a processor module (or simply processor) 19 or host. System 15 also includes a memory module (or simply memory) 20 coupled to processor module 19 for storing content such as image data received from imaging module 17 . In some examples, the entire system may be encapsulated in a single integrated circuit (IC) or chip. Additionally, each of the modules (17, 19, 20) may be implemented as a separate chip. Memory module 20 may include one or more memory chips, and processor module 19 may also include multiple processing chips. Details of the packaging of the module of FIG. 1, details of how the module is manufactured or implemented (single chip or multiple separated chips) are not relevant to this description, so such details are not included. are not provided herein.

System 15 may be any low power electronic device configured for 2D and 3D camera applications according to the present invention. System 15 may be portable or non-portable. Some examples of portable versions of system 15 include, but are not limited to, mobile devices, cell phones, smart phones, user equipment (UE), tablets, digital cameras, laptop or desktop computers, electronic smart watches, machine-to-machine (M2M) communication units, virtual reality (VR) devices or modules, robots, etc. On the other hand, some examples of non-portable versions of system 15 include video arcade game consoles, interactive video terminals, automobiles, machine viewing systems, industrial robots, VR devices, driver-side mounted cameras of automobiles (e.g., (monitoring whether the person is dozing off or not). The 3D imaging capabilities described herein are useful for automotive applications such as, but not limited to, all-weather autonomous driving and driver assistance in low-light or adverse weather conditions, human-machine interface and gaming applications, and machine vision and robotics applications. can be used in many applications such as

In some embodiments described herein, imaging module 17 includes a projector module (or light source module 22) and an image sensor unit 24. The light source of the projector module 22 is an infrared (IR) laser, such as a Near Infrared (NIR), Short Wave Infrared (SWIR) laser, making the illumination invisible. In other embodiments, the light source can be a visible light laser. Image sensor unit 24 includes a pixel array and auxiliary processing circuitry, as shown in FIG.

In one embodiment, processor module 19 is a central processing unit (CPU) that is a general purpose microprocessor. As used herein, the terms "processor" and "CPU" may be used interchangeably. However, instead of or in addition to a CPU, processor module 19 may include, but is not limited to, a microcontroller, a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC) processor. Including any other type of processor such as. In one embodiment, processor module 19 (or host) includes one or more CPUs operating in a distributed processing environment. The processor module 19 executes commands and uses an x86 instruction set architecture (32-bit or 64-bit version) that relies on the Reduced Instruction Set Computer (RISC) It may be configured to process data based on a special ISA, such as a PC® ISA or a Microprocessor without Interlocked Pipeline Stages (MIPS) instruction set architecture. In one embodiment, processor module 19 is a system-on-chip (SoC) that has functionality additional to that of a CPU.

In some embodiments, the memory module 20 is a DRAM-based three-dimensional memory module, such as, but not limited to, a synchronous dynamic random access memory (SDRAM), a high bandwidth memory (HBM) module, or a hybrid memory cube (HMC) memory module. It may be a dynamic random access memory (DRAM) such as a stacked (3DS) memory module. In other embodiments, the memory module 20 may include, but is not limited to, a solid state drive (SSD), a non-3DS DRAM module, a static random access memory (SRAM), a phase change random access memory (PRAM or PCRAM), a resistive It can be any other semiconductor-based storage system, such as random access memory (ReRAM), conductive bridge RAM (CBRAM), magnetic RAM (MRAM), or spin-transfer torque MRAM (STT-MRAM).

FIG. 2 shows an exemplary operational configuration of the imaging system 15 of FIG. 1 in accordance with the present invention. The system 15 is used to obtain distance or depth information (along the Z-axis) for an object, such as an object (or 3D object) 26, either an individual object or an object within a scene (not shown). be done. System 15 may be a direct TOF imager where a single pulse per image frame (of a pixel array) is used. In some embodiments, multiple short pulses may be transmitted to object 26. In one embodiment, distance/depth information is determined by processor module 19 based on scan data received from image sensor unit 24 . In other embodiments, the distance/depth information is determined by image sensor unit 24. In some embodiments, the depth information is used by processor module 19 as part of a 3D user interface to allow a user of system 15 to communicate with a 3D image of an object, or to display a 3D image of an object executed by system 15. It can be used as part of other applications, such as games or autonomous driving applications. 3D imaging according to the invention may also be used for other purposes or applications and applied to virtually any scene or 3D object.

In FIG. 2, the X-axis is horizontal along the front surface of system 15, the Y-axis is vertical (outside the page in view), and the Z-axis is the general direction of the object 26 that will be imaged from system 15. This is the direction in which it is stretched. For depth measurement, the optical axes of the light source module 22 and image sensor unit 24 are parallel to the Z axis. Other optical arrangements may be used to implement the principles described herein, and such alternative arrangements are considered within the scope of the invention.

Arrows associated with corresponding dotted lines (30, 31) indicating the illumination path of the light beam or optical radiation used to point-scan the object 26 within the field of view (FOV). As shown at (28, 29), the projector (or light source) module 22 illuminates the object 26. Line-by-line point scanning of the object surface is accomplished using an optical radiation source, which in one embodiment is a laser source (or simply a laser) 33 operated and controlled by a laser controller 34 . The light beam from the laser light source 33 is point-scanned in the XY direction on the surface of the object 26 through the lens 35 of the projection device under the control of the laser controller 34 . Point scanning projects points of light onto the surface of an object along a scan line, as described in more detail with reference to FIG. The lens 35 of the projection device may be a focusing lens, a glass/plastic surface, or other cylindrical optical element that focuses the laser beam from the laser light source 33 onto a point or points on the surface of the object 26. In the embodiment shown in FIG. 2, a convex structure is illustrated as the lens 35 of the projection device. However, any other suitable lens may be selected in the projection device. The object 26 is placed in a concentrated position where the illumination light from the laser light source 33 is focused into a light spot by the lens 35 of the projection device. Thus, in a point scan, points or narrow areas/spots on the surface of the object 26 are sequentially illuminated by a focused light beam from the lens 35 of the projection device.

In some embodiments, the laser light source 33 (or illumination source) may include a diode laser or light emitting diode (LED) that emits visible light, a NIR laser, a point light source, a monochromatic illumination source within the visible light spectrum (white lamps and (such as a combination of monochromators), or any other type of laser light source. Laser 33 is fixed in one position within the housing of system 15, but is rotatable in the XY directions. Laser light source 33 is XY addressable (eg, by laser controller 34) and can perform a point scan of 3D object 26. In one example, the visible light may be substantially green light. Visible light illumination from the laser light source 33 is projected onto the surface of the 3D object 26 using a mirror (not shown), and point scanning can be accomplished completely without a mirror. In some embodiments, light source module 22 may include more or fewer components than the embodiment illustrated in FIG.

In the example of FIG. 2, the light reflected from the point scan of object 26 travels along the collection path shown by arrows (36, 37) and dotted lines (38, 39). The light collection path carries photons reflected or scattered from the surface of the object 26 as light from the laser light source 33 is received. Here, the representation of various transmission paths using solid arrows and dotted lines in FIG. 2 (and also applicable to FIG. 4) is for illustrative purposes only; It shall not be considered as indicating a transmission path. In fact, the illumination and collection signal paths may differ from those shown in FIG. 2 and may not be as clearly defined as shown in FIG.

Light received from the illuminated object 26 is focused through a collector lens 44 of the image sensor unit 24 onto any one or more pixels of a 2D pixel array (or simply pixel array) 42 . Similar to the projection device lens 35, the collection device lens 44 may be a concentrating lens, glass/plastic surface, or other material that focuses the reflected light received from the object 26 onto one or more pixels within the 2D pixel array 42. It is a cylindrical optical element. In the embodiment shown in FIG. 2, a convex structure is shown as the lens 44 of the collecting device. However, any other suitable lens design may be selected as the collection device lens 44. Although pixel array 42 is illustrated in FIG. 2 as simply being a 3x3 pixel array, it should be understood that modern pixel arrays include thousands or even millions of pixels. . Pixel array 42 may be an RGB pixel array where different pixels collect light signals of different colors. In some embodiments, pixel array 42 may include any sensor, such as, but not limited to, a 2D RGB sensor with an IR blocking filter, a 2D IR sensor, a 2D NIR sensor, a 2D RGBW sensor, or a 2D RGB-IR sensor. 2D sensor. System 15 may use the same pixel array 42 for 3D imaging (including depth measurements) of object 26 as well as 2D RGB color imaging of object 26.

Pixel array 42 converts the received photons into corresponding electrical signals, which are subsequently processed by associated pixel processing unit 46 to determine a 3D depth image of object 26. In one embodiment, pixel processing unit 46 utilizes triangulation for depth measurements. The method of triangulation will be described later with reference to FIG. Pixel processing unit 46 includes circuitry for controlling operation of pixel array 42.

The processor 19 controls the operation of the light source module 22 and the image sensor unit 24. For example, system 15 can have a mode switch (not shown) controllable by the user to switch from 2D imaging mode to 3D imaging mode. When the user selects the 2D imaging mode using the mode switch, the processor 19 can activate the image sensor unit 24, but cannot activate the light source module 22 because the 2D imaging mode uses ambient light. Sometimes. On the other hand, when the user selects the 3D imaging mode using the mode switch, the processor 19 activates both the light source module 22 and the image sensor unit 24, and also changes the level of the reset (RST) signal in the pixel processing unit 46. trigger. For example, if the ambient light is too strong and linear mode is rejected (as described further below), processor 19 can change from linear mode to logarithmic mode. Processed image data received from pixel processing unit 46 is stored in memory 20 by processor 19 . Processor 19 may also display 2D or 3D images selected by the user on a display screen (not shown) of system 15. Processor 19 can be programmed with software or firmware to perform various processing tasks described herein. Alternatively or in addition, processor 19 may include programmable hardware logic to perform some or all of the functions of processor 19. In some embodiments, memory 20 stores program code, lookup tables, and/or intermediate operation results that provide processor 19 with the functionality of processor 19 .

FIG. 3 is a flowchart 50 illustrating an example of how 3D depth measurements are performed. The various operations illustrated in FIG. 3 may be performed by a single module within system 15, multiple modules, or a combination of system components. Specific tasks are described, by way of example only, as being performed by specific modules or components of the system. Other modules or system components are suitably configured to perform such tasks.

3, at step 52, system 15 (more specifically, processor module 19) utilizes a light source, such as light source module 22, to scan a 3D object, such as object 26 of FIG. 2, along a scan line. Perform a dimensional (1D) point scan. As part of a point scan, the light source module 22 is configured to project a series of light points onto the surface of the 3D object 26 in a line-by-line manner, for example by the processor 19. At step 54, pixel processing unit 46 of system 15 selects a row of pixels of an image sensor, such as 2D pixel array 42. 2D pixel array 42 includes a plurality of pixels arranged in a 2D array forming an image plane, with selected rows of pixels forming epipolar lines of the scanning line (at step 52) on the image plane. Form. A brief explanation of epipolar line geometry is provided with reference to FIG. At step 56, pixel processing unit 46 is cooperatively configured by processor 19 to detect each light spot utilizing a corresponding pixel in the row of pixels. The light reflected from a point of illumination light is concentrated into a single pixel or one or more adjacent pixels in the same way that the light reflected from a point of illumination is focused by the lens 44 of the collection device onto two or more adjacent pixels. It must be noted that this is detected by Light reflected from more than one light spot is also collected at a single pixel within the 2D pixel array 42. Timestamp-based methods eliminate ambiguities associated with depth calculations due to two different points being imaged by the same pixel or a single point being imaged by two different pixels. used for. At step 58, pixel processing unit 46 (as suitably configured by processor 19) is responsive to pixel-specific detection (at step 56) of a corresponding light spot of the series of light spots (of the point scan of step 52). to produce pixel-specific output. Consequently, in step 60, the pixel processing unit 46 performs a 3D process based on the scan angle used by the light source to project at least a pixel-specific output (in step 58) and a corresponding light spot (in step 52). Determine the 3D distance (or depth) of the corresponding light spot on the surface of the object. Depth identification will be explained in more detail with reference to FIG. 4.

FIG. 4 is a diagram illustrating an example of how points are scanned to measure 3D depth. The XY rotational capability of the laser source 33 is illustrated in FIG. 4 by arrows (62, 64) indicating the angular movement of the laser in the X direction (with angle β) and the Y direction (with angle α). In one embodiment, laser controller 34 controls the XY rotation of laser light source 33 based on scan commands/inputs received from processor 19. For example, when a user selects a 3D image mode, processor 19 configures and controls laser controller 34 to begin measuring the 3D depth of the surface of the object facing lens 35 of the projection device. In response, laser controller 34 begins scanning 1D XY points on the object surface through XY movement of laser light source 33 . As shown in FIG. 4, the laser 33 point-scans the surface of the object 26 by projecting a light spot along a 1D scanning line. Two scanning lines (S _R 66 and S _R+1 68) are shown in dotted lines in FIG. Due to the curvature of the surface of the object 26, the light spots 70-73 form a scanning line S _R 66 in FIG. The light spot forming the scanning line S _R+1 68 is not indicated by reference numerals. The laser 33 scans the object 26 one spot at a time from left to right along the row (R, R+1), for example. The value of row (R, R+1) can be found by reference to the row of pixels of the 2D pixel array 42. For example, in the 2D pixel array 42 of FIG. 4, a row (R) of pixels is designated using the reference number "75" and a row (R+1) is designated using the reference number "76." It is understood that row (R, R+1) is selected from the plurality of rows for illustrative purposes only.

The plane containing the rows of pixels of the 2D pixel array 42 is called the image plane, and the plane containing the scan lines, such as lines (S _R , S _R+1 ), is called the scan plane. In the embodiment shown in FIG. 4, the image plane and the scan plane are epipolar lines such that each row (R, R+1) forms an epipolar line of a corresponding scan line (S _R , S _R+1 ) in the 2D pixel array 42. Oriented using geometry. If the projection of the illumination spot on the image plane (at the scan line) forms a clear spot along the row (R), then the row (R) of pixels is considered to be an epipolar line corresponding to the scan line. For example, in FIG. 4, arrow 68 indicates the illumination of light spot 71 by laser light source 33, and arrow 80 indicates that light spot 71 is imaged or projected along a row (R) by lens 44 of the collection device. shows. Although not shown in FIG. 4, all of the light spots (70-73) would be imaged with corresponding pixels in row (R). Thus, in one embodiment, the physical arrangement of the laser light source 33 and the pixel array 42, such as the position and direction, is such that the illuminated light spot of a scan line on the surface of the object 26 is determined by the pixels of the corresponding row of the pixel array 42. The relevant row of pixels that are captured or detected form the epipolar line of the scan line.

The pixels of 2D pixel array 42 are arranged in rows and columns. Illuminated light spots are referenced by corresponding rows and columns of pixel array 42. For example, in FIG. 4, light spot 71 of scan line (S _R ) is designated as X _R,i , indicating that light spot 71 is imaged by row R and column i (C _i ) of pixel array 42 . Column (C _i ) is indicated by a dotted line 82 . Other illuminated spots are similarly identified. As mentioned above, light reflected from two or more points of light is received by a single pixel in a row, or light reflected from a single point of light is received by one or more of a row of pixels. received by the pixel. Timestamp-based methods can be used to remove ambiguities in depth calculations arising from such multiple or overlapping projections.

In FIG. 4, the arrow with the reference number "84" indicates the depth or distance Z (along the Z-axis) of the light spot 71 from the X-axis, such as the X-axis shown in FIG. 2, along the front surface of the system 15. shows. In FIG. 4, the dotted line with the reference number "86" indicates an axis that is visualized as being contained in a vertical plane that also includes the lens 35 of the projection device and the lens 44 of the collection device. However, to facilitate the triangulation-based explanation, the laser light source 33 in FIG. 4 is shown as being at the X-axis 86 instead of the projection device lens 35. In a triangulation-based method, the value of Z is determined using Equation 1 below.

In Equation 1, "h" is the distance along the Z-axis between the collector lens 44 and the pixel array 42, considered to be in the vertical plane of the back surface of the collector lens 44. "d" is the offset distance between the laser light source 33 associated with the image sensor unit 24 and the collection device lens 44. "q" denotes the pixel that detected the light spot corresponding to the lens 44 of the collection device (in the example of FIG. 4, detecting/imaging pixel ( _i )) in the column ( C _i )). "θ" is the scan angle or beam angle of the light source with respect to the associated light spot (light spot 71 in the example of FIG. 4). Alternatively, "q" can also be considered to be the offset of the light spot within the field of view of the pixel array 42. The parameters of Equation 1 are also shown in FIG.

From Equation 1, the variables h and d for scanning a point with only parameters (θ, q) given are in principle predetermined or fixed based on the physical geometric information of the system 15. I can see what will happen. Since the row (R) 75 is an epipolar line of the scan line (SR), the depth difference or depth profile of the object 26 is imaged, horizontally as shown as the value of q for different light points. The movement of is reflected. A timestamp-based method is used to find the correspondence between the pixel position of the captured light spot and the corresponding scan angle of the laser light source 33. That is, the timestamp can indicate the relationship between the values of q and θ. Therefore, from a known value of the scan angle (θ) and the corresponding position of the imaged light spot (as shown in q), the distance Z to the light spot can be determined using triangulation in Equation 1. can do. Triangulation for distance measurement is also described in related literature, including, for example, US Patent Application Publication No. 2011/0102763A1 filed by Brown et al. Accordingly, the descriptions of Brown's published documents relating to triangulation-based distance measurements are incorporated herein by reference in their entirety.

FIG. 5 is a block diagram illustrating an example of a pixel, such as pixel 43 of pixel array 42 of FIG. For TOF measurements, pixel 43 can operate as a time-resolved sensor. As shown in FIG. 5, pixel 43 includes a SPAD core portion (or simply SPAD core) 501 that is electrically coupled to a PPD core portion (or simply PPD core) 502. As shown in FIG. Other exemplary configurations of SPAD and PPD core arrays within a pixel, as described herein, are shown in FIGS. 6a-6c. SPAD core portion 501 includes two or more SPADs 503 that are cooperatively coupled to a first control circuit 504 . One or more SPADs 503 receive incident light 505 and generate corresponding SPAD-specific electrical signals, which are processed by a first control circuit 504 to generate SPAD-specific digital outputs. All such SPAD-specific digital outputs are collectively and symbolically indicated by arrow 506 in FIG. PPD core 502 includes a second control circuit 507 coupled to PPD 508 . A second control circuit 507 receives the SPAD output 506 and controls charge transfer from the PPD 508 in response to the SPAD output 506 to generate a pixel specific analog output signal (or simply pixel output) (PIXOUT 510). More specifically, as described in more detail below, two or more adjacent SPADs 503 within pixel 43 detect photons (reflected) from incident light 505 within a predetermined time interval. Only then is the transfer of charge from the PPD 508 stopped by the second control circuit 507, causing the TOF value and the corresponding distance to the 3D object 26 to be recorded. In other words, the spatial/temporal correlation between the outputs of at least two adjacent SPADs 503 is used to control the operation of the PPD 508. For pixel 43, the light sensing function is performed by SPAD 503 and PPD 508 is used for TCC instead of the light sensing element. The reflected photons (of reflected light pulse 37) are correlated to the transmitted light pulse 28 (compared to uncorrelated ambient light) and control of charge transfer from PPD 508 occurs within a predetermined time interval. and provides improved performance of the image sensor unit 24 in strong ambient light situations by eliminating ambient photons. Therefore, distance measurement errors can be substantially prevented.

Figures 6a-6c are diagrams showing three different examples of pixel array structures, respectively. Any of the pixel array structures shown in FIGS. 6a-6c may be used to implement pixel array 42 of FIG. 2. In FIG. 6a, an exemplary 2×2 pixel array structure 600A is shown, where each pixel (601-604) (in some embodiments, pixel 43 of FIG. 5 is shown) has one pixel-specific PPD core and Contains four pixel specific SPAD cores. For simplicity, the PPD and SPAD cores are identified only for pixel 601, with the PPD core designated by the reference number "605" and the SPAD cores designated by the reference numbers "606-609."

The structure 600A shown in FIG. 6a is a low (spatial) resolution structure due to the physical space occupied by each pixel of a semiconductor die of a given size. As a result, relatively fewer pixels are formed in the die's pixel array compared to the exemplary structure 600B of FIG. 6b, which provides the higher resolution of a 3×3 pixel array structure. In the higher resolution structure 600B of FIG. 6b, one SPAD core is shared by four (2×2) neighboring PPD cores. For example, in FIG. 6b, SPAD core 625 is shown as being shared by PPD cores of adjacent pixels (621-624) (in some implementations, each of these is shown as pixel 43 in FIG. 5). It will be done. For simplicity, other components of pixel array structure 600B in FIG. 6b are not identified with reference numbers. The configuration of pixel array structure 600B of FIG. 6b, in which one SPAD is shared between four adjacent pixels, provides an efficient 1:1 ratio between the PPD within a pixel and the SPAD associated with the pixel. provide.

Such sharing extends to 3x3 sharing or more, as shown by pixel array structure 600C in Figure 6c. The SPAD sharing configuration shown in Figure 6b provides a high (spatial) resolution structure for the pixel array, since more pixels are formed in the pixel array when each SPAD is shared by adjacent pixels of the die. do. Therefore more space on the die can accommodate more pixels. Note that because the pixels of pixel array structure 600B of FIG. 6b have a single PPD core associated with four SPAD cores in a 2x2 configuration, up to four coincident photons can be detected by each pixel (i.e. one photon per SPAD).

6a and 6b illustrate an exemplary pixel array structure in which a PPD and a SPAD are implemented in a single die. That is, SPAD and PPD are at the same level within the die. In contrast, FIG. 6c shows an exemplary 4×4 pixel array structure 600c implemented in a die with stacked pixels. For example, the SPAD core may be implemented on an upper die, and the PPD core (and readout circuitry) may be implemented on a lower die. Thus, the PPD and SPAD are in two different stacked dies, and the circuit elements of the dies (PPD, SPAD, transistor, etc.) may be electrically connected by wiring or metal bumps. Like structure 600B of FIG. 6b, pixel array structure 600C of FIG. 6c also provides a high resolution structure in which a single SPAD core is shared by nine (3×3) adjacent PPD cores. Similarly, as shown in FIG. 6c, a single PPD core, such as PPD core 641, forms a single pixel in conjunction with nine SPAD cores (642-650). SPAD cores (642-650) are also shared by other pixels. For simplicity, other pixels, their PPD cores, and associated SPAD cores are not identified by reference numbers in FIG. 6c. Furthermore, because the pixels of pixel array structure 600C of FIG. one photon per photon).

FIG. 7 is a circuit diagram illustrating an example of a pixel 700. Pixel 700 shown in FIG. 7 may be an example of the more general pixel 43 shown in FIGS. 2 and 5. An electronic shutter signal 701 is provided to each pixel (as described in more detail below with reference to the timing diagrams of FIGS. 8, 9, and 14), and the pixel 700 is triggered by the reflected light pulse 37. Capturing pixel-specific photoelectrons in a time-related manner. More generally, pixel 700 may have a charge transfer trigger portion, a charge generation and transfer portion, and a charge collection and output portion. The charge transfer trigger part includes a SPAD core 501 and a logic unit 702. The charge generation and transfer portion includes a PPD 508, a first NMOSFET (N-channel Metal Oxide Semiconductor Field Effect Transistor) (or first NMOS transistor) (or simply the first transistor) 703, a second NMOS transistor 704, and a third NMSFET. OS transistor 705 include. The charge collection and output section includes a third NMOS transistor 705, a fourth NMOS transistor 706, and a fifth NMOS transistor 707. In some embodiments, the PPD core of pixel 700 of FIG. 7 and the PPD core of pixel 900 of FIG. Formed by a transfer device. It should be noted that portions of pixel 700 described herein are for illustrative purposes only. In some examples, each portion of pixel 700 may include more, fewer, or other circuit elements than those described herein.

PPD 508 can store charge like a capacitor. In one example, PPD 508 may not respond to light by being covered. Therefore, PPD 508 can be used in TCC instead of a light sensing element. However, as mentioned above, the light sensing function is accomplished by the SPAD within the SPAD core 501. In some embodiments, a photogate or other semiconductor device (with appropriate modifications) may be used in place of the PPD in the pixel configurations of FIGS. 7 and 13.

The charge transfer trigger portion generates a transfer active (TXEN) signal 708 under control of the electronic shutter signal 701 to trigger the transfer of the charge stored in the PPD 508 . The SPAD detects photons of a light pulse reflected and transmitted from an object such as object 26 in FIG. It is latched under cooperative control of electronic shutter signal 701 for subsequent processing. Logic unit 702 controls all digital SPAD outputs 506 if, for example, there are outputs 506 received from at least two adjacent SPADs within a predetermined time interval during which electronic shutter signal 701 is activated. It includes logic circuitry to process and generate the TXEN signal 708.

In the charge generation and transfer portion, the PPD 508 is initially set to its full well capacity with the third transistor 705 using a reset signal (RST). The first transistor 703 receives a transmission voltage (VTX) signal 710 from its drain terminal and receives a TXEN signal 708 from its gate terminal. A TX signal 711 is available from the source terminal of the first transistor 703 and applied to the gate terminal of the second transistor 704. As shown, the source terminal of the first transistor 703 is coupled to the gate terminal of the second transistor 704. The VTX signal 710 (or equivalently, the TX signal 711) is used as an amplitude modulation signal to control the charge transferred from the PPD 508 coupled to the source terminal of the second transistor 704. The second transistor 704 transfers the charge of the PPD 508 from the source terminal to the drain terminal of the transistor 704, and the drain terminal is coupled to the gate terminal of the fourth transistor 706 to form a floating diffusion (FD) node/junction 712. It forms a "collection site" of charges referenced by . In some embodiments, the charge transferred from PPD 508 depends on modulation provided by an amplitude modulated signal (VTX signal 710) (or equivalently, TX signal 711). In the embodiments of FIGS. 7 and 13, the charges transferred are electrons. However, the invention is not limited to this description, PPDs with other designs can be used and the transferred charges can be holes.

In the charge collection and output portion, the third transistor 705 receives the RST signal 709 at the gate terminal of the third transistor 705 and the pixel voltage (VPIX) signal 713 from the drain terminal of the third transistor 705 . A source terminal of third transistor 705 is coupled to floating diffusion node/junction 712 . In one embodiment, the voltage level of the VPIX signal 713 is the same as the voltage level of the global power supply voltage (VDD), which may be in the range of 2.5V to 3.0V. The drain terminal of the fourth transistor 706 also receives the VPIX signal 713. In some embodiments, the fourth transistor 706 operates as an NMOS source follower and functions as a buffer amplifier. The source terminal of the fourth transistor 706 is coupled to the drain terminal of the fifth transistor 707, and the drain terminal of the fifth transistor 707 is connected to the gate of the fifth transistor 707, which is in cascode with the fourth transistor 706 of the source follower. The terminal receives a selection (SEL) signal 714. The charge transferred from PPD 508 and collected at floating diffusion node/junction 712 appears at the source terminal of fifth transistor 707 as a pixel specific output (PIXOUT) data line.

The charge transferred from PPD 508 to floating diffusion node/junction 712 is controlled by VTX signal 710 (and TX signal 711). The amount of charge reaching floating diffusion node/junction 712 is modulated by TX signal 711. In one embodiment, transfer voltage (VTX) signal 710 (and TX signal 711) progressively transfers charge from ramped PPD 508 to floating diffusion node/junction 712. Therefore, the amount of charge transferred is a function of the amplitude modulation signal (TX signal 711) and the ramping of the TX signal 711 is a function of time. Therefore, the amount of charge transferred from PPD 508 to floating diffusion node/junction 712 is also a function of time. Generation of TXEN signal 708 by SPAD core 501 due to photon detection events occurring in SPAD core 501 for at least two adjacent SPADs during charge transfer from PPD 508 to floating diffusion node/junction 712. When the second transistor 704 is turned off, the transfer of charge from the PPD 508 to the floating diffusion node/junction 712 stops. As a result, both the amount of charge transferred to floating diffusion node/junction 712 and the amount of charge remaining on PPD 508 is a function of the TOF of the incident photon. The result is a time-to-charge conversion and a single-ended differential signal conversion. Therefore, PPD 508 operates as a time-to-charge converter (TCC). The more charge is transferred to floating diffusion node/junction 712, the more the voltage at floating diffusion node/junction 712 decreases and the more the voltage at PPD 508 increases.

The voltage at the floating diffusion node/junction 712 is then transmitted as a PIXOUT signal through a fifth transistor 707 to an analog-to-digital converter (ADC) unit (not shown) to generate the appropriate digital signal for further processing. /value. More details of the timing and operation of the various signals of FIG. 7 are provided with reference to the description of FIG. 9. In that case, in the embodiment of FIG. 7, the floating diffusion node/junction 712 converts the charge on the PPD 508 into a voltage, and after the charge is completely transferred to the floating diffusion node/junction 712, the fifth transistor 707 The charge transferred to the floating diffusion node/junction 712 upon receiving the SEL signal 714 for selecting is output as the PIXOUT1 (or pixel output 1) voltage, and the charge remaining in the PPD 508 is output as the PIXOUT2 (or pixel output 2) voltage. It can be output as The data line of pixel output 510 sequentially reads out and outputs the PIXOUT1 and PIXOUT2 signals in sequence, as described below with reference to FIG. In other embodiments, one rather than both the PIXOUT1 and PIXOUT2 signals are read.

FIG. 8 is a timing diagram 800 illustrating an example overview of a modulated charge transfer mechanism in pixel 700 of FIG. The waveforms shown in FIG. 8 (also FIGS. 9 and 14) are simplified in nature and are referred to for illustrative purposes only. The actual waveforms may differ not only in form but also in timing depending on the circuit implementation. In FIGS. 7 and 8, common signals are identified using the same reference numerals and include the VPIX signal 713, the RST signal 709, the electronic shutter signal 701, and the VTX signal 710. In FIG. 8, two different waveforms (801, 802) are also shown, the state of charge in PPD 508 and the state of charge in floating diffusion node/junction 712 when VTX signal 710 is applied during charge transfer. are shown respectively. In the example of FIG. 8, VPIX signal 713 initializes pixel 700 during operation of pixel 700 starting from a low logic voltage (e.g., logic 0 or 0V) and then to a high logic voltage (e.g., logic 1 or 3V). The reset (RST) signal 709 starts with a high logic voltage pulse (e.g., a pulse that changes from a logic 0 to a logic 1 and back to a logic 0) during initialization of the pixel 700 to bring the charge in the PPD 508 to a full well. The charge on the floating diffusion node/junction 712 is set to zero Coulombs (0C). The reset voltage level for floating diffusion node/junction 712 is a logic one level. During a TOF measurement operation, the more electrons the floating diffusion node/junction 712 receives from the PPD 508, the lower the voltage at the floating diffusion node/junction 712. During initialization of pixel 700 electronic shutter signal 701 starts at a low logic voltage (e.g., logic 0 or 0V) and increases to a logic one level (e.g., 3V) at times corresponding to the minimum measurement distance during operation of pixel 700. ) to cause the SPAD 503 of the SPAD core 501 to detect photons (seconds) of the reflected light pulse 37, and then change to a logic 0 level (eg, 0V) at a time corresponding to the maximum measurement distance. Therefore, the logic one level interval of the electronic shutter signal 701 provides a predefined time interval/window during which the outputs received from adjacent SPADs have spatial and temporal correlation. The charge in PPD 508 starts fully charged during initialization and decreases as VTX signal 710 is ramped (preferably linearly) from 0V to a higher voltage. The PPD charge level under control of the amplitude modulation signal (VTX signal 710) is illustrated in FIG. 8 by the waveform having the reference number 801. The reduction in PPD charge is a function of how long the VTX signal is ramped, which induces a certain amount of charge transfer from PPD 508 to floating diffusion node/junction 712. Thus, as shown by the waveform having the reference numeral 801 in FIG. increases as the voltage increases, partially transferring a certain amount of charge from PPD 508 to floating diffusion node/junction 712. Charge transfer is a function of how long VTX signal 710 is ramped.

As mentioned above, the pixel specific output to the data line of PIXOUT 510 is derived from the PPD charge transferred to floating diffusion node/junction 712. Therefore, the PIXOUT signal 510 can be amplitude-modulated according to time by an amplitude modulation signal (VTX signal 710) (or equivalently, TX signal 711). In this manner, TOF information is provided through amplitude modulation of pixel specific outputs using VTX signal 710 (or equivalently, TX signal 711). In some embodiments, the modulation function for generating VTX signal 710 is monotonic. In the exemplary embodiments shown in FIGS. 8, 9, and 14, the amplitude modulated signal is generated using a ramp function. Therefore, these appear to have a slope type waveform. However, in other embodiments, other types of analog waveforms/functions may be used as the modulation signal.

In one example, the ratio of the output of one pixel (e.g., PIXOUT1) to the sum of the outputs of two pixels (here, PIXOUT1 + PIXOUT2) is T It is proportional to the time difference between _tof and _Tdly . For example, in the example of pixel 700, the parameter (T _tof ) is the pixel-specific TOF value of an optical signal received by two or more SPADs at 501 in the SPAD core, and the delay time parameter (T _dly ) is the This is the time from when (light pulse 28) is initially transmitted until the VTX signal 710 begins to slope. Typically, if the light pulse 28 is transmitted after the VTX signal 710 generated when the electronic shutter signal 701 is opened begins to ramp, the delay time (T _dly ) is negative. The proportional relationship is expressed by mathematical formula 2.

However, the present invention is not limited to the relationship shown in Formula 2. As explained below, the ratio in Equation 2 is used to calculate the depth or distance of an object and is less sensitive than pixel-to-pixel variation if the sum of PIXOUT1 and PIXOUT2 is not always the same. .

For convenience, as used herein, the term "P1" is used to refer to "PIXOUT1" and the term "P2" is used to refer to "PIXOUT2." The relationship in Equation 2 shows that the pixel-specific TOF value is determined by the ratio of the pixel-specific output values (P1, P2). In some embodiments, once a pixel-specific TOF value is determined, a pixel-specific distance D to an object (such as object 26 of FIG. 2), a range R, or a The specific position is given by Equation 3.

Here, c is the speed of light. Alternatively, in some embodiments, the modulating signal, such as VTX signal 710 (or TX signal 711) of FIG. 7, is linear within the shutter window, for example, and the range (or distance) is calculated by Equation 4. .

Accordingly, a 3D image of an object, such as object 26, may be generated by TOF system 15 based on pixel-specific ranges of values determined by the mathematical formulas given above.

Amplitude modulation-based manipulation within pixels or control of PPD charge distribution also allows range measurement and resolution to be controlled. The pixel level amplitude modulation of the PPD charge is a rolling shutter, such as in a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or a global shutter, such as in a CCD (Charge Coupled Device) image sensor. Although the description herein is provided primarily in the context of single-pulse TOF imaging systems, such as system 15 of FIGS. 1 and 2, the principles of the pixel-level internal amplitude modulation methods described herein can also be implemented in a continuous wave modulated TOF imaging system with pixels 43 (see FIG. 5) or in a non-TOF system through appropriate modifications (if necessary).

FIG. 9 illustrates the different signals in the system 15 of FIGS. 1 and 2 when the pixel 700 of FIG. 7 is used in a pixel array, such as the pixel array 42 of FIGS. 2 and 12, for measuring TOF values. FIG. 9 is a timing diagram 900 illustrating an example. The various signals shown and transmitted in the embodiments of FIGS. 2 and 7, such as optical pulses 28, VPIX signal 713, TXEN signal 708, etc., are identified in FIG. 9 using the same reference numerals. Prior to the discussion of FIG. 9, in the context of FIG. 9 (and also in the case of FIG. 14), the parameter T _dly is defined as the rising edge of the applied light pulse 28 and the VTX indicates the time delay between when the signal 710 begins to slope, and the parameter T _tof is the delay between the rising edges of the emitted light pulse 28 and the reflected light pulse 37, as indicated by the reference number "902". indicates the pixel-specific TOF value measured at , and the parameter T _sh is indicated by the reference numeral "903" and the activation (e.g., logic 1 or on) and deactivation (e.g., It should be noted that we indicate the time interval between the opening and closing of the electronic shutter as given by a logic 0 or off). Therefore, the electronic shutter signal 701 is considered to be activated during the interval _Tsh identified by the reference number "904". In some embodiments, the delay T _dly may be predetermined and fixed regardless of operating conditions. In other embodiments, the delay T _dly is adjustable in real time depending on, for example, external weather conditions. It should be noted here that the higher or lower signal level is related to the design of the pixel 700. The polarities and bias levels of the signals shown in FIG. 9 may differ for other types of pixel designs, based, for example, on the types of transistors and other circuit components used.

As mentioned above, the waveforms in FIG. 9 (also in FIG. 14) are simplified in nature and are for illustrative purposes only; the actual waveforms may vary depending on the implementation of the circuit. They can differ not only in form but also in timing. As shown in FIG. 9, the reflected light pulse 37 is a time-delayed version of the emitted light pulse 28. In some embodiments, the emitted light pulse 28 can be of very short duration, such as within a range of about 5 ns to about 10 ns. Reflected light pulses 37 are sensed using two or more SPADs of pixel 700. Electronic shutter signal 701 activates the SPAD to capture pixel specific photons of reflected light pulse 37. The electronic shutter signal 701 may have a gated delay (relative to the emitted light pulse 28 ) to avoid light scattering due to reaching the pixel array 42 . Light scattering of the emitted light pulses 28 may occur, for example, due to bad weather.

In addition to various external signals (e.g., VPIX signal 713, RST signal 709, etc.) and internal signals (e.g., TX signal 711, TXEN signal 708, voltage at floating diffusion node/junction 712), timing diagram 900 of FIG. , and also identify the next event or time interval. The following events or time intervals are: (i) PPD preset 905 when RST, VTX, TXEN, and X signals are high and VPIX signal 713 and electronic shutter signal 701 are low; (ii) first floating diffusion reset 906 from when the TX signal is low until the RST signal returns from high to low; (iii) delay time T _dly 901; (iv) It includes a time of flight T _tof 902, (v) an electronic shutter-on or activation period T _sh 903, and (vi) a second FD reset event 907 for the period when the RST signal 709 is logic 1 during the second time. FIG. 9 also shows when the electronic shutter is initially closing or off (indicated by reference numeral "908"), the charge initially transferred to the floating diffusion node/junction 712, and the data on PIXOUT 510. line (indicated by reference number "909"), when the voltage at the floating diffusion node/junction 712 resets the second time at the second FD reset event 907, and when the charge remaining on the PPD 508 is transferred to the floating diffusion Indicates whether it is transmitted to node/junction 712 and read at event 910 (eg, as an output to PIXOUT 510). In one embodiment, the shutter-on period T _sh is less than or equal to the ramp time of the VTX signal 710 .

Referring to FIG. 9, in the example pixel 700 of FIG. 7, the PPD 508 is filled to its full well capacity of charge during initialization (eg, PPD preset 905 event). During PPD preset 905 time, as shown, RST signal 709, VTX signal 710, TXEN signal 708, and TX signal 711 are high, and VPIX signal 713 and electronic shutter signal 701 are low. Thereafter, VTX signal 710 (and TX signal 711) goes low to shut off second transistor 704 and VPIX signal 713 goes high to begin charge transfer from fully charged PPD 508. In some embodiments, all pixels in a row of pixels of pixel array 42 are selected together at the same time, and the PPDs of all pixels in the selected row are reset together using RST signal 709. Each pixel of a selected row of pixels may be read out individually and the analog-based PIXOUT signal may be converted to a digital value by a corresponding column ADC unit (not shown). In one embodiment, the RST line of unselected rows of pixels is kept high or on to prevent blooming.

In the embodiment shown in FIG. 9, all signals except TXEN signal 708 start at a logic 0 or low level as shown. Initially, PPD 508 is preset when RST signal 709, VTX signal 710, TXEN signal 708, and TX signal 711 are at a logic one level and VPIX signal 713 is held low. Thereafter, when VTX signal 710 and TX signal 711 go to logic 0 and VPIX signal 713 goes high (or logic 1), floating diffusion node/junction 712 is reset while RST signal 709 is logic 1. For convenience, the same reference number "712" will be used to refer to the floating diffusion node/junction of FIG. 7 and the associated voltage waveforms of the timing diagram of FIG. After floating diffusion node/junction 712 is reset high (eg, 0C from the charge domain), VTX signal 710 ramps while TXEN signal 708 is a logic one. TOF interval 902 of time of flight T _tof is from when optical pulse 28 is transmitted until reflected optical pulse 37 is received and when charge is partially transferred from PPD 508 to floating diffusion node/junction 712. It is during the time when VTX signal 710 (and TX signal 711) ramps while electronic shutter signal 701 is on or open. This is a function of how much charge in PPD 508 is transferred to floating diffusion node/junction 712 and how long VTX ramps. Once the transmitted light pulse 28 is reflected off the object 26 and received by at least two SPADs in the SPAD core 501 of the pixel 700, the generated SPAD output 506 is processed by the logic unit 702, in other words, the TXEN Force signal 708 to be a fixed logic zero. Thus, in a time-correlated manner, detection of reflected light pulses 37 by at least two adjacent SPADs (ie, when the shutters are turned on or activated) is indicated by a logic zero level of TXEN signal 708. The logic low level of the TXEN signal 708 turns off the first transistor 703 and the second transistor 704, which stops the transfer of charge from the PPD 508 to the floating diffusion node/junction 712. When the electronic shutter signal 701 becomes a logic 0 and the SEL signal 714 (shown in FIG. 9) becomes a logic 1, the charge on the floating diffusion node/junction 712 is output to the data line at the voltage of PIXOUT 510 (PIXOUT1). Floating diffusion node/junction 712 is then reset again to a logic high RST signal 709 (as indicated by reference numeral "907"). Thereafter, when the TXEN signal 708 becomes a logic 1, the remaining charge on the PPD 508 is substantially completely transferred to the floating diffusion node/junction 712 and output from the PIXOUT 510 to the voltage (PIXOUT2) data line. As mentioned above, the PIXOUT1 and PIXOUT2 signals may be converted into corresponding digital values (P1, P2) by a suitable ADC unit (not shown). In certain embodiments, such values of P1 and P2 can be used in Equation 3 or Equation 4 above to determine a pixel-specific distance (or range) between pixel 700 and object 26.

In one embodiment, logic unit 702 includes logic circuitry (not shown) to generate an output based on a G() function (described with reference to FIG. 10), and then generates an output as shown in FIG. It can be logically ORed with an internally generated signal, such as a signal similar to TXRMD signal 1401, to ultimately obtain TXEN signal 708. These internally generated signals are held low while the electronic shutter is on, but when the TXEN signal 708 becomes a logic 1 and the transfer of the remaining charge on the PPD 508 (event 910 in FIG. 9) occurs. Activated high to enable. In some embodiments, the TXRMD signal or similar signal may be provided externally.

FIG. 10 shows how a logic unit, such as logic unit 702 (FIG. 7) or logic unit 1319 (FIG. 13), is implemented in a pixel, such as pixel 700 (FIG. 7) or pixel 1300 (FIG. 13). FIG. FIG. 10 shows a 2×2 configuration similar to that shown in FIG. 6a or 6b, with pixel 1000 (pixel 700 or 1300 (shown either). The availability of four SPADs allows the detection of four simultaneous photons that are temporally and spatially correlated. In some embodiments, the logic unit (not shown) of pixel 1000 includes a logic circuit (not shown) that implements the functions F(x,y) and G(a,b,c) shown in FIG. including. The F(x,y) blocks (1006 to 1009) in FIG. 10 indicate the inputs and outputs of the logic circuit that implements the function F(x,y). Thus, the F(x,y) blocks (1006-1009) represent such logic circuits and collectively form part of the logic unit of pixel 1000. For ease of explanation, such blocks are referred to as F(x,y) blocks. For convenience, the F(x,y) block (1006-1009) is illustrated as an external element of the PPD core 1001, but the logic circuit that implements the function of the F(x,y) block (1006-1009) is , may be part of a logical unit (not shown) within PPD core 1001.

As shown, each F(x,y) block (1006-1009) receives two inputs (x,y), one input from each of the two associated SPAD cores. In the context of FIGS. 5 and 7, such input is in the form of an output signal from SPAD core 501 (SPAD output 506). In the context of FIG. 13, the SPAD outputs (1310, 1318) indicate the required inputs (x, y) to the F(x, y) block within logic unit 1319. F(x,y) blocks of two similar inputs per pair of SPAD cores are used for pixels with more than four SPAD cores associated with the PPD cores, such as pixel array configuration 600c of FIG. 6c. provided for. In some embodiments, all of the F(x,y) blocks (1006-1009) are configured to operate on different pairs of SPAD outputs (as their own x and y inputs), and each can be integrated and implemented through a single F(x,y) unit within the PPD core 1001, which includes logic circuits implementing the functions of the F(x,y) blocks (1006-1009). As mentioned above, the TOF measurements described herein are based on the detection of spatially and temporally correlated photons by at least two SPADs within a pixel. Therefore, as mentioned in FIG. 10, each F(x,y) block (1006-1009) (more specifically, the logic circuit within the block of F(x,y)) may be configured to perform the operations of. The predefined operations are: (i) Negation NAND logical operation on corresponding respective inputs x and inputs y to detect two or four simultaneous photons (given by (x*y)); , and (ii) corresponding negated NOR logic operations on the respective inputs x and y (given by (x+y)) to detect three simultaneous photons. Therefore, if the signals from the SPAD cores (1002-1005) (SPAD outputs 506) (FIG. 5) indicate that two (or all four) SPADs have photons detected during the shutter-on interval, A logic circuit implementing the F(x,y) blocks (1006-1009) performs a NAND logic operation. Similarly, if the signals from the SPAD cores (1002-1005) (SPAD outputs 506) indicate that three SPADs have photons detected during the shutter-on interval, the NOR logic operation is selected. In the example shown in Figure 10, three pulses (1010-1012) are illustrated, each of the three SPAD cores (1003-1005) detecting an incident light beam such as reflected light pulse 37 (Figure 2). An example is shown in which sometimes three simultaneous photons are detected.

Referring again to FIG. 10, the output of each F(x,y) block (1006-1009) is represented using a corresponding reference character (a, b, c, d). A logic unit (not shown) within PPD core 1001 includes additional logic circuitry (not shown) to receive and process the outputs (a-d). The logic circuit receives all four such outputs as input and operates on them based on a predefined logic function G(a, b, c, d). For example, as shown in Figure 10, when detecting two simultaneous photons, the G() function performs a NAND logic operation ((a*b* c*d)). On the other hand, when detecting three or four simultaneous photons, the G() function performs a NOR logic operation (given by (a+b+c+d)) on all four of the inputs (a-d). In one embodiment, the TXEN signal, such as TXEN signal 708 of FIG. 7 or TXEN signal 1325 of FIG. 13, is the output of a logic circuit that implements the G( ) function. In other embodiments, the output of the logic circuit for the G() function is ORed with an internally generated signal, such as the TXRMD signal 1401 of FIG. 14, to produce the final TXEN signal. can be obtained.

FIG. 11 is a flowchart 1100 illustrating an example of how TOF values are determined in system 15 of FIGS. 1 and 2. The various steps illustrated in FIG. 11 may be performed by a single module or combination of modules or system components of system 15. In the description herein, certain tasks are described as being performed by particular modules or system components, by way of example only. Other modules or system components may also be suitably configured to perform such tasks. As noted in step 1101, system 15 (more specifically, projector module 22) applies a laser pulse, such as light pulse 28 of FIG. 2, to an object, such as object 26 of FIG. 2. At step 1102, processor 19 (or pixel processing unit 46 in a particular embodiment) applies an amplitude modulated signal, such as VTX signal 710 of FIG. 7, to a PPD within the pixel, such as PPD 508 within pixel 700 of FIG. do. Pixel 700 is any of pixels 43 of pixel array 42 of FIG. At step 1103, pixel processing unit 46 begins transferring a portion of the charge stored in PPD 508 based on the modulation received from the amplitude modulated signal (VTX signal 710). To initiate such charge transfer, pixel processing unit 46 receives various external signals, such as electronic shutter signal 701, VPIX signal 713, and RST signal 709, as shown in the exemplary timing diagram of FIG. Provided to pixel 700 at a logic level. At step 1104, reflected pulses, such as reflected light pulse 37, are detected using multiple SPADs within pixel 700. As mentioned above, reflected light pulse 37 is the illuminated light pulse 28 reflected from object 26, and each SPAD within pixel 700 (within SPAD core 501) receives the brightness from the reflected pulse. to a corresponding (SPAD specific) electrical signal.

For each SPAD receiving brightness, a first control circuit 504 within the SPAD core 501 within the pixel 700 processes the corresponding (SPAD-specific) electrical signal to produce a SPAD-specific digital output (step 1105). All such SPAD specific digital outputs are collectively represented by the arrows having the reference number "506" in FIGS. 5 and 7. As mentioned with reference to the description of FIG. 9, logic unit 702 processes SPAD outputs 506 and places TXEN signal 708 in a logic 0 (low) state as long as the outputs are temporally and spatially correlated. The logic 0 level of the TXEN signal 708 turns off the first transistor 703 and the second transistor 704 within the pixel 700, which stops the transfer of charge from the PPD 508 to the floating diffusion node/junction 712. Therefore, in step 1106, as the digital outputs of at least two SPADs are generated during a predetermined time interval, such as within the shutter of the interval (shutter-on interval 904) of FIG. 9, the second control circuit 507 ends the previously started transfer of a portion of the charge (at step 1103).

As discussed above with reference to FIG. 7, a portion of the charge transferred to the floating diffusion node/junction 712 is read out as the PIXOUT1 signal and converted to the appropriate digital value P1, which is subsequently generated. is used with the digital value P2 (for the PIXOUT2 signal) to obtain the TOF information from the ratio of "P1/(P1+P1)". Accordingly, at step 1107, upon termination (at step 1106), based on the portion of the transferred analog charge, the pixel processing unit 46 or processor 19 within the system 15 determines the TOF value of the reflected light pulse 37. be able to.

FIG. 12 is a layout diagram showing an example of a portion of the image sensor unit 1200. Image sensor unit 1200 corresponds to image sensor unit 24 shown in FIGS. 1 and 2. A portion of the image sensor unit 1200 shown in FIG. 12 captures the reflected light to generate values of P1 and P2 for subsequent calculation of the TOF value (from Equation 2) and, if necessary, 26 associated with providing the necessary signals for the generation of 3D images. As in FIG. 2, the pixel array 1201 in the image sensor unit 1200 of FIG. 12 is shown for convenience as having nine pixels arranged in a 3×3 arrangement. In practice, a pixel array may include hundreds of thousands or millions of pixels in numerous rows and columns. In some embodiments, each pixel of pixel array 1201 can have the same configuration, and thus each pixel is identified using the same reference number "1202" as shown in FIG. 12. In the example of FIG. 12, 2D pixel array 1201 may be a complementary metal oxide semiconductor (CMOS) array, where each pixel 1202 may be a pixel 1300 as shown in FIG. Although the exemplary layout of FIG. 12 is for the pixel configuration of FIG. 13, it is understood that the image sensor unit 1200 of FIG. 12 may be suitably modified if each pixel 1202 has the configuration shown in FIG. It should be. In some implementations, pixels 1202 can have a different configuration than shown in FIGS. 7 and 13, and the auxiliary processing units of FIG. 12, such as row decoder/driver 1203, column decoder 1204, etc. It can be modified appropriately to work with the required pixel configuration.

In addition to the pixel array 1201, in the embodiment shown in FIG. 12, the image sensor unit 1200 also includes a row decoder/driver 1203, a column decoder 1204, and a column-specific analog-to-digital conversion used during 2D and 3D imaging. It includes a pixel column unit 1205 that includes circuitry for correlated double sampling (CDS) as well as an ADC. In one embodiment, there is one ADC per column of pixels. In some examples, processing units (1203, 1204, 1205) may be part of pixel processing unit 46 shown in FIG. 2. In the embodiment of FIG. 12, the row decoder/driver 1203 controls the operation of the pixels in the pixel array 1201 by providing six different signals as inputs to each pixel 1202 in the row of pixels to control the operation of the pixels in the pixel array 1201. Shown as enabling the generation of signals (1206-1208). Each of the arrows (1209-1211) in FIG. 12 indicates a row-specific set of such signals applied to the input to each pixel 43 of the corresponding row. Such signals include a reset (RST) signal, a second transmission (TX2) signal, an electronic shutter (SH) signal, a transmission voltage (VTX) signal, a pixel voltage (VPIX) signal, and a row selection (SEL) signal. . Figure 13 shows how such a signal is applied to a pixel. FIG. 14 shows an exemplary timing diagram including a number of such signals.

In one embodiment, a row select (SEL) signal is activated to select the appropriate row of pixels. Row decoder/driver 1203 receives address or control information for the selected row via row address/control input 1212, for example from processor 19. Row decoder/driver 1203 decodes the received row address/control input 1212, causing row decoder/driver 1203 to select the appropriate row using the SEL signal, and the corresponding RST, VTX, and other Provide a signal to the selected/decoded row. A more detailed description of such signals when applied as pixel inputs is provided below with reference to the descriptions of FIGS. 13 and 14. In some embodiments, the row decoder/driver 1203 also receives control signals (not shown), such as from the processor 19, and controls the SEL, RST, VTX, SH, and various For other signals, row decoder/driver 1203 is configured to apply appropriate voltage levels.

Pixel column unit 1205 receives PIXOUT signals (1206-1208) from the selected row of pixels and processes them to generate pixel-specific signal values from which TOF measurements are obtained. The signal values are the values of P1 and P2 described above, as shown by arrow 1213 in FIG. Each column specific ADC unit processes the received input (PIXOUT signal) and produces a corresponding digital data output (P1 and P2 values). Additional details of the operation of the CDS and ADC provided by CDS and ADC circuitry (not shown) within pixel column unit 1205 are provided below with reference to FIG. In the embodiment shown in FIG. 12, column decoder 1204 is shown coupled to a column unit 1205 of pixels. Column decoder 1204 receives a column address/control input 1214, for example from processor 19, for a column to be selected in conjunction with a provided row select (SEL) signal. Column selection is sequential, thus allowing sequential reception of pixel output from each pixel in the row selected by the corresponding SEL signal. Processor 19 provides appropriate row address inputs to select a row of pixels and also provides appropriate column address inputs to column decoder 1204 to cause pixel column units 1205 to output from individual pixels of the selected row. (PIXOUT signal).

FIG. 13 is a circuit diagram showing another example of the pixel 1300. Pixel 1300 in FIG. 13 is another example of the more comprehensive pixel 43 shown in FIG. Pixel 1300 may include multiple SPAD cores (ie, SPAD core 1 to SPAD core N, where N is 2 or more) as part of the SPAD cores. Two such SPAD cores (1301A and 1301N) are shown in FIG. 13 along with some circuit details. It should be noted that in some embodiments, a similar circuit may be employed for the SPAD core within pixel 700 of FIG. SPAD core 1300A includes a SPAD 1302 that receives a SPAD operating voltage (VSPAD) 1303 via a resistive element 1304 (such as a resistor). However, the configuration of SPAD is not limited to that shown in FIG. 13. In one embodiment, the locations of register 1304 and SPAD 1302 may be swapped. In SPAD core 1301A, SPAD 1302 responds to light. When SPAD 1302 receives a photon, SPAD 1302 outputs a pulse that goes from the level of VSPAD to 0V and restores VSPAD. The output from SPAD 1302 is filtered through capacitor 1305 and applied to inverter 1306 (which functions as a combination buffer and latch). In one embodiment, capacitor 1305 may be omitted. SPAD core 1301A includes a PMOS transistor 1307 that receives an electronic shutter signal 1308 at its gate terminal, the drain terminal of transistor 1307 is coupled to a capacitor (and the input of inverter 1306), and the source terminal of transistor 1307 is connected to supply voltage 1309 (VDD ) (or the voltage of VPIX in some embodiments). When electronic shutter signal 1308 is turned off (e.g., logic 0 or low level), transistor 1307 conducts and output 1310 of inverter 1306 is fixed regardless of the state of any output received from SPAD core 1301A. voltage level (eg, a logic low or logic 0 state). When electronic shutter signal 1308 is turned on or active, the output from SPAD core 1301A is applied only to PPD core 1311. When the shutter is active (e.g., logic 1 level), transistor 1307 is turned off and the SPAD generated output is transferred to inverter 1306 (via coupling capacitor 1305) to generate a positive pulse (low to high) on output line 1310. ).

SPAD core 1301N may be identical to SPAD core 1301A in circuit details, so operational details of SPAD core 1301N are not provided. As shown, the SPAD core 1310N includes a core-specific SPAD 1312, a resistive element 1313 that provides a VSPAD signal 1303 to the SPAD 1312, and a PMOS transistor 1317 that controls the operation of an inverter 1316 via an input of an electronic shutter signal 1308. . The output of inverter 1316 is provided to PPD core 1311 for further processing. In some embodiments, the VSPAD signal 1303, the VDD signal 1309, and the electronic shutter signal 1308 are input to the row decoder/driver 1203 shown in FIG. 12 of the pixel processing unit 46 (or processor) of FIG. 2 or any other module. Each SPAD core (1301A, 1301N) may be supplied from an external unit such as (not shown). Both SPAD core specific outputs (1310, 1318) form a signal collectively identified with reference numeral "506" in FIG.

Therefore, electronic shutter signal 1308 is generated in addition to the fact that the outputs (1310, 1318) from SPAD cores (1301A, 1301N) are spatially correlated by the adjacent positions of SPAD cores (1301A, 1301N) within pixel 1300. , are temporally (or temporally) correlated. Additional pixel geometry information is shown in the exemplary embodiments of FIGS. 6a-6c.

Like the pixel 700 of FIG. 7, the pixel 1300 of FIG. , generates an internal input (TXEN signal 1325) and receives external inputs RST signal 1326, VTX signal 1327 (and TX signal 1328), VPIX signal 1329, and SEL signal 1330, and connects a floating diffusion (FD) node/junction. 1331 and outputs a PIXOUT signal 510. Unlike pixel 700 of FIG. 7, pixel 1300 of FIG. 13 also generates a TXEN signal (TXENB) 1333, which is the complement of TXEN signal 1325 and is provided to the gate terminal of a sixth NMOS transistor 1334. The sixth NMOS transistor 1334 has a drain terminal connected to the source terminal of the first NMOS transistor 1320 and a source terminal connected to a ground (GND) potential 1335. The TXENB signal 1333 is used to transfer the GND potential to the gate terminal of the second NMOS transistor 1321 (TX transistor). In the absence of the TXENB signal 1333, the gate of the second NMOS transistor 1321 (TX transistor) is floated when the TXEN signal 1325 goes low, and there is a possibility that the charge transfer from the PPD 508 is not completely completed. This situation can be improved using the TXENB signal 1333. Furthermore, pixel 1300 also includes a storage diffusion (SD) capacitor 1336 and a seventh NMOS transistor 1337. The SD capacitor 1336 is connected to a junction between the drain terminal of the second NMOS transistor 1321 and the source terminal of the seventh NMOS transistor 1337, forming an SD node 1338 at the junction. The seventh NMOS transistor 1337 receives as input a different second transmission (TX2) signal 1339 at its gate terminal. A drain of the seventh NMOS transistor 1337 is connected to the FD node 1331 as shown.

In some embodiments, the RST, VTX, VPIX, TX2, and SEL signals may be provided to pixel 1300 from an external unit, such as row decoder/driver 1203 shown in FIG. 12. In some embodiments, SD capacitor 1336 may simply be a junction capacitor at SD node 1338 rather than an extra capacitor. A comparison between FIG. 5 and FIG. 13 shows that in pixel 1300 all of the SPAD cores (1301A, 1301N) collectively form SPAD 503 of FIG. collectively form the first control circuit 504 of FIG. 5, and that all of the non-PPD circuit elements of the PPD core 502 can form the second control circuit 507 of FIG.

In pixel 1300, the charge transfer trigger portion includes SPAD cores (1301A, 1301N) (and other such cores) and logic unit 1319. The charge generation and transfer portion includes a PPD 508, NMOS transistors (1320-1322, 1334, 1337), and an SD capacitor 1336. The charge collection and output portion includes NMOS transistors (1322-1324). It should be noted that the division of various circuit components into respective parts is exemplary and for explanation purposes only. In some embodiments, such portions may include more or fewer other circuit components than those described herein.

As mentioned above, except for the CDS-based charge collection and output portions, the pixel configuration of FIG. 13 is substantially similar to that of FIG. 7. Therefore, for convenience, circuit parts and signals common between FIG. 7 and FIG. Not explained in the specification. CDS is a noise reduction technique for measuring electrical values such as pixel/sensor output voltages (PIXOUT) in a manner that allows removal of unnecessary offsets. In some embodiments, a column-specific CDS unit (not shown) can be employed in a pixel's column unit 1205 (FIG. 12) to perform correlated double sampling. With CDS, the output of a pixel, such as pixel 1300 in FIG. 13, can be measured twice, once under known conditions and once under unknown conditions. The value measured from known conditions is subtracted from the value measured from unknown conditions, representing a value that has a known relationship to the physical quantity being measured, i.e. represents a pixel-specific portion of the received light. PPD charges can be generated. Using CDS, noise may be reduced by removing the pixel's reference voltage (eg, the pixel's voltage after being reset) from the pixel's signal voltage at the end of each charge transfer. Thus, in the CDS, a reset/reference value is sampled before the pixel's charge is transferred to the output, and this is then subtracted from the value after the pixel's charge is transferred.

In the embodiment of FIG. 13, SD capacitor 1336 (or associated SD node 1338) stores PPD charge before the PPD charge is transferred to floating diffusion node 1331, so that the charge is transferred to floating diffusion node 1331. The appropriate reset value is established (and sampled) at the floating diffusion node 1331 before it is activated. As a result, each pixel specific output (PIXOUT1 and PIXOUT2) is processed in a column specific CDS unit (not shown) of the pixel column unit 1205 (FIG. 12) to obtain a pair of pixel specific CDS outputs. I can do it. Thereafter, the pixel-specific CDS outputs may be converted to digital values, such as P1 and P2, as indicated by arrows 1213 in FIG. 12, by each column-specific ADC unit (not shown) of pixel column unit 1205. Transistors (1334, 1337) and TXENB signal 1333 and TX2 signal 1339 of FIG. 13 provide necessary and auxiliary circuitry to accomplish CDS-based charge transfer. In one embodiment, the values of P1 and P2 may be generated in parallel utilizing the same pair of ADC circuits, eg, as part of a column-specific ADC unit. Therefore, the difference between the reset level of the PIXOUT1 and PIXOUT2 signals and the corresponding PPD charge level is converted to a digital value by the column-parallel ADC and output to the pixel-specific signal values (i.e., P1 and P2), and the above math Allows calculation of the pixel-specific TOF value of the reflected light pulse 37 for the pixel 1300 based on Equation 2. As discussed above, such calculations may be performed by pixel processing unit 46 or processor 19 of system 15. Consequently, the pixel-specific distance of object 26 (FIG. 2) may also be determined using, for example, Equation 3 and Equation 4. The pixel-by-pixel charge collection operation is repeated for all rows of pixels in pixel array 42. Based on the pixel-specific distance or range values of pixels 43 of pixel array 42, a 3D image of object 26 may be generated by processor 19 and displayed on a suitable display or user interface associated with system 15, for example. A 2D image of object 26 is generated by simply adding the values of P1 and P2, for example if a range of values is not calculated or if a range of values is possible but a 2D image is required. In some embodiments, such 2D images are simply grayscale images, for example when an IR laser is used.

It should be noted that the pixel configurations shown in FIGS. 7 and 13 are exemplary only. Other types of PPD-based pixels with multiple SPADs may also be used to implement the present invention. Such a pixel may, for example, be a pixel with a single output (such as the data line of PIXOUT 510 in the embodiments of Figures 7 and 13) or a dual output where the PIXOUT1 and PIXOUT signals are output through different outputs at the pixel. contains pixels with .

FIG. 14 illustrates the different signals of system 15 of FIGS. 1 and 2 when pixel 1300 of FIG. 13 is used to measure TOF values in a pixel array such as pixel array 42 of FIGS. 2 and 12. FIG. 14 is a timing diagram 1400 illustrating another example. The timing diagram 1400 of FIG. 14 shows, among other things, the waveforms of the VTX signal, the shutter signal, the VPIX signal, and the TX signal, and various time intervals, such as the PPD reset event, the shutter-on interval, and the delay time interval (T _dly ). or similar to timing diagram 900 of FIG. 9 for event identification. In contrast to the above extensive description of the timing diagram 900 of FIG. 9, a general description of the distinguishing features of the timing diagram 1400 of FIG. 14 is provided for convenience.

In FIG. 14, various externally provided signals such as VPIX signal 1329, RST signal 1326, electronic shutter signal 1308, amplitude modulation signal (VTX signal 1327), TX2 signal 1339, and internally generated TXEN Signal 1325 is identified using the same reference numerals as used in FIG. Similarly, for convenience, the same reference number "1331" is used to indicate the floating diffusion node 1331 of FIG. 13 and the associated voltages of the timing diagram of FIG. 14. The transmission mode (TXRMD signal 1401) is shown in FIG. 14, but not shown in the timing diagrams of FIG. 13 or FIG. 10. In some embodiments, TXRMD signal 1401 is generated internally by logic unit 1319 or externally provided to logic unit 1319 by a row decoder/driver (such as row decoder/driver 1203 of FIG. 12). can be supplied. In one embodiment, logic unit 1319 generates an output based on the G() function (FIG. 10) and then OR's the output with an internally generated signal, such as TXRMD signal 1401. and logic circuitry (not shown) to obtain the final TXEN signal 1325. As shown in FIG. 14, in one embodiment, such an internally generated TXRMD signal 1401 is held low while the electronic shutter is on, and then activated high to generate the TXEN signal 1325. At logic 1, the transfer of the remaining charge on the PPD (event 1410 in FIG. 14) is accomplished.

The PPD preset 1403 event, delay time (T _dly ) 1404, TOF interval (T _tof ) 1405, shutter-off interval 1406, shutter-on interval 1408 or valid interval (T _sh ) 1407, and FD reset signal 1409 in FIG. Similar to the corresponding event or time interval shown in 9. Therefore, no additional explanation of such parameters is provided. Initially, the FD Reset 1409 event causes the FD 1331 signal to go high, as shown. SD node 1338 is reset high after PPD 508 is preset low. More specifically, during the event of PPD preset 1403, TX signal 1328 is high, TX2 signal 1339 is high, RST signal 1326 is high, and VPIX signal 1329 is low, causing electrons to fill PPD 508. , and preset PPD 508 to 0V. The TX signal 1328 then goes low, and the TX2 signal 1339 and RST signal 1326 remain high briefly to reset the SD node 1338 high and remove electrons from the SD capacitor 1336, along with the VPIX signal 1329 high. At the same time, FD node 1331 is reset (following the FD Reset 1409 event). The voltage at SD node 1338 and the SD reset signal are not shown in FIG.

In contrast to the embodiments of FIGS. 7 and 9, when electronic shutter signal 1308 is activated and VTX signal 1327 ramps up, the PPD charge is amplitude modulated, as shown in the waveform of TX signal 1328. , and is initially transferred to SD node 1338 (via SD capacitor 1336). In response to detection of photons by at least two SPADs at pixel 1300 (FIG. 13) during shutter-on interval 1408, TXEN signal 1325 goes low and initial charge transfer from PPD 508 to SD node 1338 is stopped. The charge stored and transferred to SD node 1338 is read out from the data line of PIXOUT 510 (as the PIXOUT1 output) during a first read interval 1412. In the first read period 1412, the RST signal 1326 is briefly activated to high to reset the floating diffusion node 1331 after the electronic shutter signal 1308 is deactivated or turned off. The TX2 signal 1339 then pulses high, transferring the charge from the SD node 1338 to the floating diffusion node 1331 while the TX2 signal 1339 is high. The voltage waveform at floating diffusion node 1331 indicates a charge transfer operation. The transferred charge is then read out (as a PIXOUT1 voltage) via the data line of PIXOUT 510 during a first read interval 1412 using a SEL signal 1330 (not shown in FIG. 14).

During the first read interval 1412, after the initial charge is transferred from the SD node to the FD node and the TX2 signal 1339 returns to a logic low level, the TXRMD signal 1401 is activated (pulsed) high. Generate a high pulse at the input of the TXEN signal 1325. In turn, a high pulse is generated at the input of TX signal 1328 to transfer the remaining charge of PPD 508 to SD node 1338 (via SD capacitor 1336), as indicated by reference numeral "1402" in FIG. Thereafter, when RST signal 1326 is briefly reactivated high, FD node 1331 is reset again. The second RST high pulse defines a second readout interval 1413, and the TX2 signal 1339 pulses high again, transferring the remaining charge of the PPD 508 from the SD node 1338 to the floating diffusion node 1331 while the TX2 signal is high. (Event 1410). The voltage waveform at floating diffusion node 1331 indicates a second charge transfer operation. The remaining transferred charge is then read out (as the PIXOUT2 voltage) via the data line of PIXOUT 510 using the SEL signal 1330 (not shown in FIG. 14) during a second read interval 1413. As mentioned above, the PIXOUT1 and PIXOUT2 signals may be converted into corresponding digital values (P1, P2) by a suitable ADC unit (not shown). In certain embodiments, the values of P1 and P2 can be used in Equation 3 or Equation 4 above to determine a pixel-specific distance (or range) between pixel 1300 and object 26. The SD-based charge transfer shown in FIG. 14 allows for the generation of pixel-specific CDS output pairs, as described above with reference to the description of FIG. CDS-based signal processing provides additional noise reduction.

FIG. 15 is a block diagram showing an example of a time-resolved sensor 1500. Time-resolved sensor 1500 includes a SPAD circuit 1501, a logic circuit 1503, and a PPD circuit 1505.

The SPAD circuit 1501 includes a SPAD for detecting photons, a first input for receiving a VSPAD voltage, a second input for receiving a shutter signal for controlling opening and closing of an electronic shutter, and a third input for receiving a VDD voltage. It includes an input and an output for outputting a detection event (DE) signal. In response to receiving photons, the SPAD circuit 1501 outputs a pulse signal that rapidly increases from the VSPAD voltage to a voltage below the SPAD breakdown voltage, and then gradually returns from the VSPAD voltage.

Logic circuit 1503 has a first input coupled to the DE signal output from SPAD circuit 1501 and a first input for receiving a TXRMD signal for completely transferring the charge remaining in the PPD of PPD circuit 1505 to the FD node. It includes two inputs and an output for outputting the TXEN signal.

PPD circuit 1505 receives a first input coupled to the TXEN signal output from logic circuit 1503 and a VTX signal for partially or fully transferring charge from the PPD of PPD circuit 1505 to the FD node of PPD circuit 1505. a second input that receives the RST signal for resetting the charge of the FD node and presetting the charge of the PPD, a fourth input that receives the signal of VPIX for the PPD circuit 1505, and a fourth input that receives the PIXOUT1 signal ( a fifth input for receiving the SEL signal to enable reading of the FD node charge (representing the charge on the FD node) or the PIXOUT2 signal (indicating the charge remaining on the PPD) and outputting the PIXOUT1 and PIXOUT2 signals in response to the SEL signal; and a PIXOUT output for.

FIG. 16 is a circuit diagram showing an example of the SPAD circuit 1501 of the time-resolved sensor 1500 of FIG. 15. In one embodiment, SPAD circuit 1501 includes resistor 1601, SPAD 1603, capacitor 1605, p-type MOSFET 1607, and buffer 1609. Resistor 1601 includes a first terminal and a second terminal that receive the VSPAD voltage. SPAD 1603 includes a positive electrode connected to ground potential and a negative electrode connected to a second terminal of resistor 1601. In other embodiments, the positions of resistor 1601 and SPAD 1603 may be swapped. SPAD 1603 is responsive to light. In response to receiving photons, the SPAD 1603 outputs a pulse signal that changes rapidly from the VSPAD voltage to a voltage below the breakdown voltage and then returns more gradually to the VSPAD voltage. In one example, the breakdown voltage is a particular threshold voltage.

Capacitor 1605 includes a first terminal connected to the negative electrode of SPAD 1603 and the other second terminal. In alternative embodiments, capacitor 1605 may be omitted. P-type MOSFET 1607 has a first S/D terminal coupled to the second terminal of capacitor 1605, a second S/D terminal that receives the VPIX voltage (VDD), and a gate that receives the shutter signal. include. Buffer 1609 includes an input coupled to the second terminal of capacitor 1605 and an output that outputs the DE signal. The DE signal corresponds to the DE output of the SPAD circuit 1501. In an alternative embodiment, buffer 1609 may be an inverter.

FIG. 17 is a circuit diagram showing an example of the logic circuit 1503 of the time-resolved sensor 1500 of FIG. 15. Logic circuit 1503 includes a latch 1701 and a two-input logical sum (OR) gate 1703.

Latch 1701 includes an input coupled to the DE signal output from SPAD circuit 1501 and its output. In response to the DE signal, latch 1701 outputs a logic signal to change from, for example, a logic 1 to a logic 0 and maintain a logic 0. In other words, the latch 1701 converts a pulse type signal into a signal that changes from a logic 0 to a logic 0 and remains at a logic 0 without returning to a logic 1 until reset. The latch output is triggered by the leading edge of the DE signal, where the leading edge is in the positive or negative direction depending on the design of the SPAD circuit 1501.

Two-input OR gate 1703 includes a first input coupled to the output of latch 1701, a second input that receives the TXRMD signal, and an output that outputs the TXEN signal. Two-input OR gate 1703 performs a logical OR function and outputs the result on the TXEN signal. Specifically, when the shutter signal is a logic 1, a photon is received by the SPAD circuit 1501 or the remaining charge of the PPD of the PPD circuit 1505 is completely transferred to the FD node for being read out to the PIXOUT2 signal. When the TXRMD signal that occurs is a logic one, the output of the two-input OR gate 1703 will be a logic one.

FIG. 18 is a circuit diagram showing an example of the PPD circuit 1505 of the time-resolved sensor 1500 of FIG. 15. The PPD circuit 1505 includes a first transistor 1803, a second transistor 1805, a third transistor 1807, a fourth transistor 1809, a fifth transistor 1811, and a PPD 1801.

PPD 1801 includes a positive electrode and another negative electrode connected to ground potential. PPD 1801 stores charge in a manner similar to a capacitor. In one embodiment, PPD 1801 is covered and therefore does not respond to light and can be used as a TCC instead of a light sensing element.

The first transistor 1803 includes a gate terminal connected to the TXEN signal output of the logic circuit 1503, a first S/D terminal for receiving the VTX signal, and another second S/D terminal. The first transistor 1803 receives the VTX signal, passes the VTX signal through the first transistor 1803 under the control of the TXEN signal, and outputs the TX signal from the second S/D terminal of the first transistor 1803.

The second transistor 1805 has a gate terminal connected to the second S/D terminal of the first transistor 1803, a first S/D terminal connected to the negative electrode of the PPD 1801, and a second S/D terminal connected to the second S/D terminal of the first transistor 1803. D terminal. The second transistor 1805 receives the TX signal at its gate terminal and transfers the charge of the PPD 1801 at its source terminal to its drain terminal connected to the FD node. Although not shown in FIG. 18, stray capacitance (parasitic capacitance) may exist between the FD node and ground. In one embodiment, a parasitic capacitance may also be coupled between the FD node and ground.

The third transistor 1807 has a gate terminal for receiving the RST signal, a first S/D terminal for receiving the VPIX signal, and a second S/D terminal coupled to the second S/D terminal of the second transistor 1805. /D terminal.

The fourth transistor 1809 has a gate terminal connected to the second S/D terminal of the second transistor 1805 and a first S/D terminal connected to the first S/D terminal of the third transistor 1807. , and the other second S/D terminal.

The fifth transistor 1811 has a gate terminal that receives the SEL signal, a first S/D terminal coupled to the second S/D terminal of the fourth transistor 1809, and a second S/D terminal that is the PIXOUT output of the PPD circuit 1505. S/D terminal of. The fifth transistor 1811 receives the SEL signal for selecting a pixel and reads out the charge of the FD node (as PIXOUT1) or the remaining charge of the PPD (1801) (as PIXOUT2).

The charge transferred from PPD 1801 to the FD node is controlled by the TX signal. In one embodiment, the VTX signal is coupled through the first transistor to become the TX signal. The VTX signal slopes upward to gradually transfer more and more charge from PPD 1801 to the FD node. The amount of charge transferred from PPD 1801 to the FD node is a function of the level of the TX signal, and the slope of the TX signal is a function of time. Therefore, the charge transferred from PPD 1801 to the FD node is a function of time. During the transfer of charge from the PPD 1801 to the FD node, the second transistor 1805 is turned off in response to the SPAD circuit 1501 detecting the incoming photon, and the transfer of charge from the PPD 1801 to the FD node is stopped. The amount of charge transferred to the FD node and the amount of charge remaining on PPD 1801 are all related to the TOF of the incident photon. The transfer of charge from PPD 1801 to the FD node based on the TX signal and the detection of incident photons is considered to provide a single-ended differential conversion from charge to time.

The fourth transistor 1809 converts the charge stored in the FD node into a voltage at the second S/D terminal of the fourth transistor 1809. The SEL signal selects a pixel and reads the PIXOUT1 signal corresponding to the charge transferred to the FD node or the PIXOUT2 signal corresponding to the charge remaining in the PPD 1801 after the remaining charge of the subsequent PPD 1801 is transferred to the FD node. used for. In one embodiment, the ratio of the PIXOUT1 signal to the sum of the PIXOUT1 signal and the PIXOUT2 signal is proportional to the difference between the TOF and the delay time of the optical signal received by the pixel, as shown in the ratio of Equation 2 above. . In embodiments where the optical pulse is transmitted after the VTX signal begins to slope upward, the delay time is negative.

For the time-resolved sensor 1500, the ratio described in Equation 2 above is used to determine the depth or range of an object, and if the sum of the PIXOUT1 and PIXOUT2 signals does not change from measurement to measurement, Not so sensitive. In one embodiment, the VTX signal is ideally linear and ideally uniform across different pixels of the TOF pixel array. However, in reality, the VTX signals applied to different pixels of a TOF pixel array vary on a pixel-by-pixel basis, thus inducing errors in range measurements that depend on the variation of the VTX signal on a pixel-by-pixel basis, and also vary on a measurement unit. It is possible.

In one embodiment, the first transistor 1803, the second transistor 1805, the third transistor 1807, the fourth transistor 1809, and the fifth transistor 1811 are each an n-type MOSFET or a p-type MOSFET. However, the invention is not limited to the use of n-type or p-type MOSFETs, as any suitable transistor may be used.

FIG. 19 is a relative signal timing diagram illustrating an example for the time-resolved sensor 1500 of FIG. 15. In FIG. 19, during the shutter-off (initialization) period, each of the RST signal, VTX signal, and TX signal becomes high (logic 1), and later returns to 0 (logic 0) to reset the PPD circuit 1505. The TXEN signal is high. PPD 1801 is filled with charge to its full well capacity during the initialization period. The VTX and TX signals go low, turning off the second transistor 1805 of the PPD circuit 1505. The VPIX signal goes high, thus resetting the FD node. A light pulse is transmitted toward the object when the RST signal returns to zero, or some time thereafter. The VTX signal then begins to ramp upward and the shutter signal goes high, beginning the shutter-on interval.

As the VTX signal slopes upward, the TX signal also slopes upward and the charge on the FD node begins to decrease in response to the TX signal. The reflected light pulse causes the TXEN signal to go low (logic 0), thus stopping charge transfer between the FD node and PPD 1801.

The delay time (T _dly ) indicates the interval between the start of the transmitted light pulse and the time when the TX signal begins to slope upward. The time of flight (T _tof ) indicates the time between the beginning of the transmitted light pulse and the time the return signal is received. The shutter-on period (T _sh ) indicates the time (shutter-on period) from when the electronic shutter is opened to when the electronic shutter is closed. In one embodiment, the shutter interval (T _sh ) is equal to or less than the ramp time of the VTX signal.

The transferred charge is read out as a PIXOUT1 signal in a read period of the transferred charge. While the shutter signal is low, the RST signal goes a second high to reset the charge on the FD node, after which the TXRMD, TXEN, and TX signals go high to transfer the remaining charge in the PPD1801 to the PIXOUT2 signal. The data is transferred to the FD node for reading.

FIG. 20 is a block diagram showing another example of the time-resolved sensor 2000. Time-resolved sensor 2000 includes a SPAD circuit 2001, a logic circuit 2003, and a second PPD circuit 2005.

The SPAD circuit 2001 includes a SPAD for detecting photons, a first input for receiving a VSPAD voltage, a second input for receiving a shutter signal for controlling opening and closing of an electronic shutter, and a VDD voltage (VDD). and an output for outputting a detection event (DE) signal. In response to receiving photons, SPAD circuit 2001 outputs a pulse signal that rapidly goes from VSPAD to 0 and slowly returns to VSPAD. In one embodiment, SPAD circuit 2001 is the same as SPAD circuit 1501 shown in FIG.

The logic circuit 2003 has a first input coupled to the DE output of the SPAD circuit 2001 and a second input for receiving a TXRMD signal for completely transferring the charge remaining in the PPD of the second PPD circuit 2005, and a TXEN signal. and an output for outputting the . In one embodiment, logic circuit 2003 is the same as logic circuit 1503 shown in FIG.

A second PPD circuit 2005 has a first input coupled to the TXEN signal output from logic circuit 2003 and a second input coupled to a second input of logic circuit 2003 that receives the TXRMD signal; a third input receiving a VTX signal for partially or completely transferring charge from the PPD of the circuit 2005 to a first floating diffusion (FD1) node of the second PPD circuit 2005 and resetting the charge of the FD1 node; A fourth input receives the RST signal for presetting the charge of the PPD, a fifth input receives the VPIX signal for the second PPD circuit 2005, and reads the PIXOUT1 signal corresponding to the charge of the FD1 node from the PIXOUT1 output. and a sixth input for receiving a SEL signal for reading a PIXOUT2 signal corresponding to the charge remaining in the PPD of the second PPD circuit 2005 from the PIXOUT2 output.

FIG. 21 is a circuit diagram showing an example of the second PPD circuit 2005 of the time-resolved sensor 2000 in FIG. 20. The second PPD circuit 2005 includes a first transistor 2103, a second transistor 2105, a third transistor 2107, a fourth transistor 2109, a fifth transistor 2111, a sixth transistor 2113, a seventh transistor 2115, an eighth transistor 2117, a ninth transistor A transistor 2119 and a PPD 2101 are included.

PPD 2101 includes a positive electrode and another negative electrode connected to ground potential. PPD 2101 stores charge in a manner similar to a capacitor. In one embodiment, PPD 2101 is covered and therefore does not respond to light and can be used as a TCC instead of a light sensing element.

The first transistor 2103 has a gate terminal coupled to the output of the logic circuit 2003 that receives the TXEN signal output, and a first S/D terminal that receives the VTX voltage for controlling charge transfer from the PPD 2101. and the other second S/D terminal.

The second transistor 2105 has a gate terminal connected to the second S/D terminal of the first transistor 2103 to receive a TX signal for transferring charge from the PPD 2101, and a gate terminal connected to the negative terminal of the PPD 2101. 1 and a second S/D terminal connected to a first floating diffusion (FD1) node to which charge is transferred from the PPD 2101. The FD1 node may have a first capacitor. Stray capacitance not shown in FIG. 21 may exist between the FD1 node and ground. In one embodiment, a physical capacitor may also be coupled between the FD1 node and ground. The charge transferred from the PPD 2101 to the FD1 node via the second transistor 2105 is controlled by the TX signal.

The third transistor 2107 has a gate terminal connected to the FD1 node and the second S/D terminal of the second transistor 2105, a first S/D terminal that receives the voltage of VPIX, and another second S/D terminal. S/D terminal. The third transistor 2107 converts the charge stored in the FD1 node into a voltage at the second S/D terminal of the third transistor 2107.

The fourth transistor 2109 has a gate terminal that receives an RST signal for setting the charge level of the FD1 node, a first S/D terminal that receives the voltage of VPIX, and a second S/D terminal of the second transistor 2105. a second S/D terminal coupled to the D terminal.

The fifth transistor 2111 has a gate terminal that receives an SEL signal for reading the charge of the FD1 node, a first S/D terminal connected to the second S/D terminal of the third transistor 2107, and a first S/D terminal connected to the FD1 node. a second S/D terminal connected to the data line of the pixel output (PIXOUT1) for outputting a voltage corresponding to the charge of the pixel as a PIXOUT1 signal.

The sixth transistor 2113 has a gate terminal that receives a TXRMD signal for completely transferring the charge remaining in the PPD 2101 to a second floating diffusion (FD2) node, and a first S/D terminal connected to the negative electrode of the PPD 2101. and a second S/D terminal connected to the FD2 node. The FD2 node may have a second capacitance. Stray capacitance not shown in FIG. 21 may exist between the FD2 node and ground. In one embodiment, a physical capacitor may also be coupled between the FD2 node and ground. In one embodiment, the second capacitance of the FD2 node may be the same as the first capacitance of the FD1 node. Any remaining charge remaining in PPD 2101 is transferred to the FD2 node via the sixth transistor 2113.

The seventh transistor 2115 has a gate terminal connected to the second S/D node and the FD2 node of the sixth transistor 2113, a first S/D terminal that receives the VPIX signal, and the other second S/D terminal. /D terminal. The seventh transistor 2115 converts the charge stored in the FD2 node into a voltage at the second S/D terminal of the seventh transistor.

The eighth transistor 2117 has a gate terminal that receives the RST signal for setting the charge level of the FD2 node, a first S/D terminal that receives the VPIX signal, and a second S/D terminal of the sixth transistor 2113. a second S/D terminal coupled to the terminal.

The ninth transistor 2119 has a gate terminal for receiving an SEL signal for selecting a pixel and reading a voltage corresponding to the charge of the FD2 node, and a first transistor connected to the second S/D terminal of the seventh transistor 2115. and a second S/D terminal coupled to a data line of the pixel output (PIXOUT2) for outputting a voltage corresponding to the charge of the FD2 node as a PIXOUT2 signal.

In one embodiment, the VTX signal (and thus the TX signal) slopes upward to transfer charge from PPD 2101 to the FD1 node. The amount of charge transferred from PPD 2101 to the FD1 node is a function of the level of the TX signal, and the slope of the TX signal is a function of time. Therefore, the charge transferred from PPD 2101 to the FD1 node is a function of time. During the transfer of charge from the PPD 2101 to the FD1 node, if the second transistor 2105 is turned off in response to SPAD circuit 2001 detecting an incident photon, the transfer of charge from the PPD 2101 to the FD1 node is stopped. , the amount of charge transferred to the FD1 node and the amount of charge remaining on PPD 2101 are all related to the TOF of the incident photon. Transfer of charge from PPD 2101 to FD1 node based on the detection of the TX signal and incoming photons provides a single-ended-to-differential conversion of charge to time.

For the time-resolved sensor 2000, the ratio described in Equation 2 above is used to determine the depth or range of an object, and if the sum of the PIXOUT1 and PIXOUT2 signals does not change from measurement to measurement, it will be Not sensitive. In one example, the VTX signal may be ideally linear and may be ideally uniform across different pixels of the TOF pixel array. However, in reality, the VTX signals applied to different pixels of the TOF pixel array vary from pixel to pixel, and therefore induce errors in range measurement depending on the variation of the VTX signal from pixel to pixel, and also vary from measurement to measurement. obtain.

In one embodiment, the first transistor 2103, the second transistor 2105, the third transistor 2107, the fourth transistor 2109, the fifth transistor 2111, the sixth transistor 2113, the seventh transistor 2115, the eighth transistor 2117, and the ninth transistor 2119. may each be an n-type MOSFET or a p-type MOSFET, but any other suitable transistor may be used.

FIG. 22 is a relative signal timing diagram 2200 illustrating another example of the time-resolved sensor 2000 of FIG. The signal timing diagram of FIG. 22 is similar to the signal timing diagram of FIG. 19, and the similarities will be explained with reference to FIG. The signal timing diagram of FIG. 22 differs in that it includes the FD signal and the remaining charge of the PPD 2101 is transferred to the FD2 node by the operation of the TXRMD signal at the end of the shutter-on period. Further, the PIXOUT1 and PIXOUT2 signals are read out simultaneously.

Although the second PPD circuit 2005 relies on the unchanged full well capacity to determine the maximum range, the actual implementation of the time-resolved sensor 2000 depends on the thermal noise between the different second PPD circuits 2005. It should be noted that full-well capacity variations for PPD2101 will be experienced based on the PPD2101. Also, the VTX signal can have a different slope based on the location of the pixel in the pixel array. That is, at a pixel, the slope of the VTX signal varies depending on how close the pixel is from the source of the VTX signal.

FIG. 23 is a circuit diagram showing still another example of the time-resolved sensor 2300. Time-resolved sensor 2300 includes one or more SPAD circuits (2301a-2301n), a logic circuit 2303, and a third PPD circuit 2305.

In one embodiment, each of the one or more SPAD circuits 2301 includes a SPAD 2311, a resistor 2313, a capacitor 2315, a p-type MOSFET transistor 2317, and a buffer 2319. SPAD 2311 includes a positive electrode and another negative electrode connected to ground potential. Resistor 2313 includes a first terminal that receives the VSPAD voltage and a second terminal coupled to the negative terminal of SPAD 2311 . In other embodiments, the positions of SPAD 2311 and resistor 2313 may be swapped. SPAD 2311 is responsive to light. In response to receiving a photon, the SPAD 2311 outputs a pulse signal that abruptly drops from the VSPAD voltage to a voltage below the breakdown voltage and then slowly returns to the VSPAD voltage.

The capacitor 2315 includes a first terminal connected to the negative electrode of the SPAD 2311 and a second terminal. In alternative embodiments, capacitor 2315 may be omitted. P-type MOSFET transistor 2317 has a gate terminal that receives the shutter signal, a first S/D terminal coupled to the second terminal of capacitor 2315, and a second S/D terminal that receives the voltage (VDD) of VPIX. including a terminal. Buffer 2319 includes an input coupled to the second terminal of capacitor 2315 and an inverted output that outputs a DE signal corresponding to the output of SPAD circuit 2311. In alternative embodiments, buffer 2319 may not be inverted.

Logic circuit 2303 includes an input coupled to the DE signal output of each of one or more SPAD circuits (2301a-2301n), and an output that outputs a TXEN signal and a TXENB signal, which is the inverse of the TXEN signal.

The third PPD circuit 2305 includes a capacitive device (SC), a first transistor 2351, a second transistor 2353, a third transistor 2355, a fourth transistor 2357, a fifth transistor 2359, a sixth transistor 2361, a seventh transistor 2363, An eighth transistor 2365, a ninth transistor 2367, a tenth transistor 2369, an eleventh transistor 2371, a twelfth transistor 2373, and a thirteenth transistor 2375 are included.

A capacitive device (SC) includes a first terminal coupled to a ground potential and another second terminal. Capacitive devices (SCs) store charge in a manner similar to capacitors. In one example, the capacitive device (SC) may be a capacitor. In other embodiments, the capacitive device (SC) can be a PPD that is covered and non-responsive to light. In another example, a capacitive device (SC) may be used as part of the TCC.

The first transistor 2351 has a gate terminal 2351 connected to the RST signal, a first S/D terminal connected to ground potential, and a second S/D terminal connected to the second terminal of the capacitive device (SC). /D terminal.

The second transistor 2353 has a gate terminal connected to the TXA signal, a first S/D terminal connected to the first floating diffusion (FD1) node, a second S/D terminal of the first transistor 2351, and a first S/D terminal connected to the first floating diffusion (FD1) node. a second S/D terminal coupled to a second terminal of the capacitive device (SC). The first floating diffusion (FD1) node is represented by a capacitor symbol in FIG. Stray capacitance not shown in FIG. 23 may exist between the FD1 node and ground. In one embodiment, a physical capacitor may also be coupled between the FD1 node and ground.

The third transistor 2355 has a gate terminal connected to the FD1 node and the first S/D node of the second transistor 2353, a first S/D terminal connected to the voltage of VPIX, and the other second S/D terminal. S/D terminal of. The third transistor 2355 converts the charge at the FD1 node into a voltage at the second S/D terminal of the third transistor 2355.

The fourth transistor 2357 has a gate terminal connected to the RST signal, a first S/D terminal connected to the voltage of VPIX, and a first S/D terminal of the third transistor 2355 and FD1. A second S/D terminal.

The fifth transistor 2359 has a gate terminal connected to the TXEN signal, a first S/D terminal connected to the VTX signal, and a second S/D terminal connected to the gate terminal of the second transistor 2353. ,including.

The sixth transistor 2361 has a gate terminal connected to the TXENB signal, a first S/D terminal connected to the ground potential, a gate terminal of the second transistor 2353, and a second S/D terminal of the fifth transistor 2359. a second S/D terminal coupled to the terminal.

The seventh transistor 2363 has a gate terminal connected to the SEL signal, a first S/D terminal connected to the second S/D terminal of the third transistor 2355, and a data line (PIXA) of the pixel output. and a connected second S/D terminal.

The eighth transistor 2365 has a gate terminal connected to the TXB signal, a first S/D terminal connected to the second floating diffusion (FD2) node, a second S/D terminal of the first transistor 2351, a second S/D terminal coupled to a second terminal of the capacitive device (SC) and a second S/D terminal of the second transistor 2353. The second floating diffusion (FD2) node is indicated by the capacitor symbol in FIG. Stray capacitance not shown in FIG. 23 may exist between the FD2 node and ground. In one embodiment, a physical capacitor may also be coupled between the FD2 node and ground.

The ninth transistor 2367 has a gate terminal connected to the FD2 node and the first S/D terminal of the eighth transistor 2365, a first S/D terminal connected to the voltage of VPIX, and a second S/D terminal connected to the FD2 node and the first S/D terminal of the eighth transistor 2365. S/D terminal of. The ninth transistor 2367 converts the charge at the FD2 node into a voltage at the second S/D terminal of the ninth transistor 2367.

The tenth transistor 2369 has a gate terminal connected to the RST signal, a first S/D terminal connected to the voltage of VPIX, a gate terminal of the ninth transistor 2367, an FD2 node, and a first S/D terminal of the eighth transistor 2365. a second S/D terminal connected to the second S/D terminal.

The eleventh transistor 2371 has a gate terminal connected to the TXENB signal, a first S/D terminal connected to the VTX signal, and a second S/D terminal connected to the gate terminal of the eighth transistor 2365. ,including.

The twelfth transistor 2373 has a gate terminal connected to the TXEN signal, a first S/D terminal connected to the ground potential, a gate terminal of the eighth transistor 2365, and a second S/D terminal of the eleventh transistor 2371. a second S/D terminal coupled to the terminal.

The thirteenth transistor 2375 has a gate terminal connected to the SEL signal, a first S/D terminal connected to the second S/D terminal of the ninth transistor 2367, and a data line (PIXB) of the pixel output. and a connected second S/D terminal.

FIG. 24 is a relative signal timing diagram showing yet another example of the time-resolved sensor 2300 of FIG. 23. The signal timing diagram of FIG. 24 is similar to the signal timing diagrams of FIGS. 19 and 22, and the similarities will be described with reference to FIG. The signal timing diagram of FIG. 24 differs from the signal timing diagram of FIG. 22 in that it does not include the TXRMD signal and the TX signal, but instead includes the TXENB signal, TXA signal, and TXB signal.

In the signal timing diagram of FIG. 24, the TXENB signal is an inverted signal of the TXEN signal. When the shutter signal is active high, the TXEN signal is active, and the VTX signal passes through the fifth transistor 2359 so that the TXA signal is activated. The charge of the capacitive device (SC) is transferred to the FD1 node via the second transistor 2353. At the same time, the ground potential passes through the twelfth transistor 2373 to deactivate the TXB signal.

When a detection event (DE) occurs (receives a reflected light pulse), the TXEN signal is deactivated and the TXENB signal is activated. When the TXEN signal is deactivated, the TXA signal is also deactivated, stopping charge from being transferred from the capacitive device (SC) through the second transistor 2353 to the FD1 node. When the TXENB signal is activated, the TXB signal is activated and charge is transferred from the capacitive device (SC) to the FD2 node through the eighth transistor 2365.

When the shutter signal ends, the TXB signal is deactivated and charge is stopped from being transferred from the capacitive device (SC) through the eighth transistor 2365 to the FD2 node. The respective voltages associated with the charges on the FD1 and FD2 nodes are read from the PIXA and PIXB output lines.

The variation in the slope of the VTX signal on a pixel-by-pixel basis and the variation in the capacitance of the capacitive device (SC) is such that the second transistor 2353 (TXA) and the eighth transistor 2365 (TXB) are in linear mode during the active shutter signal. It must be noted that as long as there is no range measurement error.

FIG. 25 is a flowchart illustrating an example of a method 2500 for resolving time using the time-resolving sensor 2300 of FIG. The method begins at step 2501. At step 2502, an active shutter signal is generated. At step 2503, one or more photons incident on at least one SPAD circuit 2301 during an active shutter signal are detected. One or more of the detected photons are those reflected from the object. At step 2504, an output signal is generated based on the detected event (DE). At step 2505, a first active signal (eg, TXEN) and a second active signal (eg, TXENB) are generated based on the output signal for the detected event (DE). In one embodiment, the first active signal is activated in response to the initiation of the active shutter signal and deactivated in response to the output signal, and the second active signal is activated and activated in response to the output signal. Deactivated in response to termination of the shutter signal.

In step 2506, if the first active signal is active, the charge of the capacitive device (SC) is transferred to the first floating diffusion (FD1) node, and the first charge is transferred to the first floating diffusion (FD1) node. form. In step 2507, if the second active signal is active, the remaining charge of the capacitive device (SC) is transferred to the second floating diffusion (FD2) node, and the second charge is transferred to the second floating diffusion (FD2) node. Form. At step 2508, a first voltage based on the first charge and a second voltage based on the second charge are output. a first ratio of the first voltage to the sum of the first voltage and the second voltage is proportional to the time of flight of the one or more detected photons; The ratio is proportional to the time of flight of one or more detected photons. At step 2509, the method ends.

In one embodiment, transferring the first charge and the second charge includes modifying the VTX signal (or drive signal) based on a ramp function, the VTX signal (or drive signal) comprising: The one or more photons begin to change in response to the start time of the detected light pulse and continue to change until the end of the active shutter signal. The step of transferring the charge of the capacitive device to the first floating diffusion (FD1) node to form the first charge at the first floating diffusion (FD1) node further includes the step of transferring the charge of the capacitive device to the first floating diffusion (FD1) node to form a first charge at the first floating diffusion (FD1) node when the first active signal is active. (or the drive signal), the step of transferring the remaining charge of the capacitive device to the second floating diffusion (FD2) node to form a second charge at the second floating diffusion (FD2) node further comprises: This is performed based on the level of the VTX signal (or drive signal) when the second active signal is active.

In other embodiments, the first ratio of the first voltage to the sum of the first voltage and the second voltage is further proportional to a time of flight minus a delay time of the one or more detected photons. Similarly, the second ratio of the second voltage to the sum of the first voltage and the second voltage is further proportional to the time of flight of the one or more detected photons minus the delay time. The delay time includes the time between the start of the optical pulse's transmission time and when the VTX signal (or drive signal) begins to change.

LiDAR (light detection and ranging) systems that use SPADs generally do not provide light intensity-based imaging capabilities. Imaging intensity along with distance information can significantly improve object recognition performance in advanced driver assistance systems (ADAS) and autonomous driving applications. The present invention provides an imaging system that provides both distance measurement and intensity imaging information. Both the range and intensity images are generated from the same source, there are no image alignment problems, and there is no need for complex blending algorithms. The pixel embodiments described herein are configured to be TCC. Additionally, pixels configured to be time-to-digital converters (TDCs) may also be used, but may provide images with spatially lower resolution than pixels configured to be TCCs. can. That is, pixels configured to be TCC can provide a pixel array that is smaller and has a higher resolution than a pixel array using pixels configured to be TDC.

An example of an imaging processing system that provides distance and light intensity information is imaging system 15 shown in FIG. Pixel array 42 may include pixels such as pixel 43 shown in FIG. 5, pixel 700 shown in FIG. 7, pixel 1300 shown in FIG. 13, pixel 2100 shown in FIG. Includes an example of a TCC pixel as described herein. Light source 22 is controlled to provide a scan of points that is synchronized to pixel processing unit 46 as described in conjunction with FIGS. 3 and 4. The point scan can be repeated many times to provide a statistical average for both distance and light intensity information. Light intensity information may be used to determine the reflectance of objects within the field of view of image processing system 15. In an alternative embodiment, light source module 22 may be controlled to illuminate the entire scene. When multiple light pulses are emitted and captured, a histogram is formed for each pixel, and the bins around the peaks in the histogram are summed to greatly improve object recognition performance in ADAS and autonomous driving applications. A grayscale image can be generated that can be used to create an image. It should be noted that alternative embodiments may use pixels configured to provide a TDC output.

The reflectance of each point captured by image sensor unit 24 may be determined based on the distance and grayscale value for each point. Grayscale images can also be generated without laser pulses. By summing multiple frames together, image sensor unit 24 can operate like a Quanta image sensor that captures up to one photon per frame per pixel. High dynamic range imaging can be achieved when adding together multiple bitplanes (ie, frames). By utilizing both 3D and 2D images generated from the same image sensor unit for object recognition, complex image blending processing may be avoided and recognition performance may be improved.

In one example, the grayscale value of a pixel may be generated by directly utilizing the peak value of a histogram of arrival times (ie, detection times) of photons detected by the pixel. The window width may be the same as the full width at half-maximum (FWHM) of the applied laser light or pulse. A histogram of light detection times is formed with bins corresponding to the number of peaks of detected light that can be used to measure the surface reflectance of the object at the point where the light pulse was reflected. Alternatively, the histogram of pixels can be convolved with the trigger waveform output from the SPAD, and the maximum detected photon count can then be selected.

FIG. 26a is a diagram illustrating an example of a trigger waveform 2600 output from a SPAD. The horizontal axis of FIG. 26 is relative time (no units), and the vertical axis of FIG. 26 is relative amplitude (no units). FIG. 26b is a histogram 2601 showing an example of the light detection time of the formed pixels. The horizontal axis in Figure 26b is relative normalized time and the vertical axis shows counts of photon detection events.

FIG. 26c is a diagram for explaining that, as an example of the histogram 2602, the window width indicating the FWHM of the irradiated pulse (not shown) is represented by the window 2602a, and the maximum event count is determined. be. FIG. 26d shows an example of the histogram 2603, and is a diagram for explaining that the trigger waveform (FIG. 26a) output from the SPAD is convolved with the histogram to determine the maximum event count.

The surface reflectance (S) is measured starting from Equation 5 below.

where P is the pixel value, α is a system-dependent constant that converts lux to pixel value, S is the surface reflectance, and L is the light intensity. The light intensity (L) is expressed as the sum of L _amb and L _laser , which is the intensity of the ambient light and the intensity of the laser light reaching each sensor.

Therefore, the surface reflectance (S) is measured using Equation 6.

The intensity of ambient light (L _amb ) is expressed by Equation 7.

where N is the number of events detected from the light pulse, M is the number of events detected before the light pulse is detected (i.e. from the surroundings), and D is the measured distance. , β are variables that depend on the other systems being compensated.

FIG. 27 is a histogram 2700 showing an example for pixels. The horizontal axis of histogram 2700 is relative time (without units) and the vertical axis of histogram 2700 is count of photon detection events. The number of events (M) detected before the light pulse is shown at 2701. The number (N) of detected events among the light pulses is shown at 2702.

The relationship between the emitted laser power (denoted _Ilaser ) and the received laser power ( _Llaser ) is Equation 8.

Here, γ is a system constant.

Thereafter, the measurement of the reflectance (S) at the “<x, y>” position is expressed by Equation 9.

Here, f(D) is a distance-dependent function. That is, it is derived from mathematical formula 10.

FIG. 28 is a flowchart 2800 of an example method for generating scene depth or extent, a map, and a grayscale image. The method begins at step 2801. At step 2802, the scene is point-scanned by, for example, imaging system 15 shown in FIG. The pixel array (42) of the imaging system 15 includes exemplary pixels (43 (FIG. 5), 700 (FIG. 7), 1300 (FIG. 13), 2100 (FIG. 21), and/or 2300 (FIG. 23)). may be included. Although the point scan may be performed only once, it should be understood that better results may be obtained by repeating the point scan many times. At step 2803, light detection events are accumulated for pixels of pixel array 42. At step 2804, a depth or extent map is generated as described herein. Distance information is determined by pixel processing unit 46 and/or processor 19. At step 2805, a grayscale image of the scene is generated based on the reflectance measurements of the scene. The gray scale image is determined by pixel processing unit 46 and/or processor 19. The method ends at step 2806.

FIG. 29a is an image showing an example of a scene 2900. 29b and 29c are a depth map 2901 and a grayscale image 2902, respectively, showing an example of the scene shown in FIG. 29a. The scale on the right side of Figure 29b is in meters.

FIG. 30 is a block diagram showing an example of the overall layout of the imaging system 15 shown in FIGS. 1 and 2. As shown in FIG. Imaging module 17 includes the necessary hardware shown in the exemplary embodiments of FIGS. 2, 5, 7 (or 13) and provides 2D/3D imaging and TOF imaging based on advanced aspects of the present embodiments. measurements can be achieved. Processor 19 is configured to connect to a number of external devices. In one embodiment, imaging module 17 may function as an input device that provides data input for further processing to processor 19 in the form of processed pixel outputs such as P1 and P2 in FIG. Processor 19 may also receive input from other input devices (not shown) that are part of system 15. Some examples of such input devices may include a computer keyboard, touch pad, touch screen, joystick, physical or virtual "clickable buttons," and/or computer mouse/pointing device. In FIG. 30, processor 19 is coupled to system memory 20, peripheral storage unit 275, one or more output devices (display unit 277), and network interface 278. In some examples, system 15 may include one or more instances of the illustrated devices. Some examples of system 15 include a computer system (desktop or laptop), a tablet computer, a mobile device, a cell phone, a video game unit or console, a machine-to-machine (M2M) communication unit, a robot, an automobile, a virtual reality device, This includes stateless thin client systems, vehicle dash-cam or rearview camera systems, autonomous driving systems, and all other types of computing or data processing equipment. In various embodiments, all of the components shown in FIG. 30 may be implemented within a single housing. Accordingly, system 15 may be configured as a stand-alone system or any other suitable form factor. In some embodiments, system 15 may be configured as a client system rather than a server system.

In some embodiments, system 15 may include one or more processors (eg, in a distributed processing configuration). If system 15 is a multiprocessor system, there may be one or more instances of processor 19, or there may be multiple processors coupled to processor 19 via respective interfaces (not shown). Processor 19 is a system on a chip (SoC) and may include one or more central processing units (CPUs).

System memory 20 may be a semiconductor-based storage system such as, for example, DRAM, SRAM, PRAM, ReRAM, CBRAM, MRAM, STT-MRAM, etc. In some examples, memory unit 20 may include at least one 3DS memory module along with one or more non-3DS memory modules. Non-3DS memory modules include double data rate, double data rate 2, 3, or 4 synchronous dynamic random access memory (DDR/DDR2/DDR3/DDR4 SDRAM), Rambus DRAM, flash memory, various forms of read It may include dedicated memory (ROM) and the like. Also, in some embodiments, system memory 20 may include multiple different types of semiconductor memory rather than a single type of memory. In other embodiments, system memory 20 may be a non-transitory data storage medium.

Peripheral storage unit 275 may include, in various embodiments, a hard disk drive, an optical disk (such as a compact disc (CD) or a digital versatile disc (DVD)), a non-volatile random access memory (RAM) device, etc. , optical, magneto-optical, or solid state storage media. In some embodiments, peripheral storage unit 275 may include a disk array (which may be a suitable Redundant Array of Independent Disks (RAID) configuration) or a more complex storage device/system, such as a storage area network (SAN). The peripheral storage unit 275 includes a SCSI (Small Computer System Interface) interface, a fiber channel interface, a Firewire (registered trademark) (IEEE1394) interface, a PCIe Express (registered trademark) (Peripheral Component Interconnect). t Express) standards-based interface, It may be coupled to processor 19 via a standard peripheral interface, such as a Universal Serial Bus (USB) protocol-based interface, or other suitable interface. Such various storage devices may be non-transitory data storage media.

Display unit 277 may be an example of an output device. Other examples of output devices are graphics/display devices, computer screens, warning systems, CAD/CAM (Computer Aided Design/Computer Aided Machining) systems, video game stations, smartphone display screens, car dashboard mounted displays. including a screen or other suitable type of data output device. In some embodiments, input device(s) such as imaging module 17 and output device(s) such as display unit 277 may be coupled to processor 19 via an I/O or peripheral interface.

In one embodiment, network interface 278 communicates with processor 19 and couples system 15 to a network (not shown). In other embodiments, there may be no network interface 278 at all. Network interface 278 may include appropriate equipment, media, and/or protocol content for wirelessly or wiredly coupling the system to a network. In various embodiments, a network may include a local area network (LAN), a wide area network (WAN), a wired or wireless Ethernet, a wireless communication network, a satellite link, or other suitable type of network. .

System 15 includes an on-board power supply unit 280 that provides electrical power to the various system components shown in FIG. Power supply unit 280 includes a battery and may be coupled to an AC electrical power outlet or a vehicle-based power outlet. In one embodiment, power supply unit 280 can convert solar energy or other renewable energy into electrical power.

In one embodiment, the imaging module 17 may be integrated into a high-speed interface built into any personal computer (PC) or laptop computer, such as a USB (Universal Serial Bus) 2.0 or 3.0 interface. For example, a computer readable non-transitory storage medium, such as system memory 20 or a peripheral data storage unit such as a CD/DVD, may store program code or software. Processor 19 and/or pixel processing unit 46 (FIG. 2) of imaging module 17 is configured to execute program code, and system 15 performs 2D processing, such as the operations described above with reference to FIGS. Imaging (eg, grayscale images of 3D objects), TOF and distance measurements, and generation of 3D images of objects using pixel-specific distance (or range) values can be performed. For example, in certain embodiments, in response to execution of program code, processor 19 and/or pixel processing unit 46 may appropriately configure associated circuitry, such as row decoder/driver 1203 and column decoder 1204 of FIG. configuring (or activating) and applying an appropriate input signal, such as a shutter signal, RST signal, VTX signal, SEL signal, etc., to a pixel 43 of pixel array 42 to capture light from the reflected laser pulse; The pixel outputs are subsequently processed for pixel specific values (P1, P2) required for TOF and distance measurements. Program code or software is executed on a suitable processing entity, such as processor 19 and/or pixel processing unit 46, so that the processing entity processes various pixel-specific ADC outputs (values of P1 and P2). Registered or open source software that determines distance values based on TOF-based distance measurements and presents the results in various forms, including displaying 3D images of distant objects, for example based on TOF-based distance measurements. obtain. In certain embodiments, pixel processing unit 46 of imaging module 17 may perform a portion of the processing of the pixel output before the pixel output data is transmitted to processor 19 for additional processing and display. . In other embodiments, processor 19 may also perform some or all of the functions of pixel processing unit 46. In this case, pixel processing unit 46 may not be part of imaging module 17.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the present invention is not limited to the above-described embodiments, and can be variously modified without departing from the technical scope of the present invention. It is possible to implement it.

15 System 17 Imaging module 19 Processor (module)
20 Memory (module), system memory 22 Projector module (light source module)
24 Image sensor unit 26 (3D) object 28 Light pulse 33 Laser (light source)
34 laser controller 35 projection device lens 37 reflected light pulses 42, 1201 (2D) pixel array 43, 601-604, 621-624, 700, 1000, 1202, 1300 pixels 44 collection device lens;
46 Pixel processing unit 66 Scanning line (S _R )
68 scanning line ( _SR+1 )
70-73 Light spot 75 Row 77 RST electronic 275 Peripheral storage unit 277 Display unit 278 Network interface 280 Power supply unit 501, 606-609, 625, 642-650, 1002-1005, 1301A-1301N SPAD core (part)
502, 605, 1001, 1311 PPD core (part)
503 Many SPADs
504 First control circuit 505 Incident light 506, 1310, 1318 SPAD output 507 Second control circuit 508 PPD
510, 540, 1206 to 1208 Pixel (specific analog) output (PIXOUT) signal 600A, 600B Pixel array structure 641 PDR core 701, 1308 Electronic shutter signal 702, 1319 Logic unit 703 to 707 First to fifth (NMOS) transistors 708 , 1325 Transfer active (TXEN) signal 709, 1326 RST signal 710, 1327 Transmission voltage (VTX) signal (amplitude modulation signal)
711, 1328 TX signal 712, 1331 Floating diffusion (FD) node/junction 713, 1329 Pixel voltage (VPIX) signal 714, 1330 Selection (SEL) signal 801 Charge in PPD 802 Charge in FD 901, 1404 Delay time T _dly
902, 1405 TOF section (flight time) T _tof
903, 904, 1407, 1408 Shutter-on section T _sh
905 PPD preset 906, 907, 1409 Floating diffusion (FD) reset 908, 1406 Shutter off section 1006 to 1009 F (x, y) block 1200 Image sensor unit 1203 Row decoder/driver (unit)
1204 Column decoder (unit)
1205 pixel column units (column CDS and column ADC)
1209, 1210, 1211 RST, VTX, VPIX shutter, TX2, SEL signal 1212 Row address/control input 1213 P1/P2 value 1214 Column address/control input 1302, 1312, 1603, 2311 SPAD
1303 SPAD operating voltage (VSPAD) (signal)
1304, 1313, 1601, 2313 Resistance (sexual element)
1305, 1315, 1605, 2315 Capacitor 1306, 1316 Inverter 1307, 1317 (PMOS) Transistor 1309 Supply voltage (VDD) (signal)
1310, 1318 Output 1320, 1321, 1322, 1323, 1324, 1334, 1337 1st to 7th (NMOS) transistors 1337
1333 TXENB signal 1335 Ground (GND) potential 1336 Storage diffusion (SD) capacitor 1338 SD node 1339 Second transmission (TX2) signal 1401 TXRMD signal 1403 PPD preset 1500, 2000, 2300 Time-resolved sensor 1501, 2001 SPAD circuit 1503, 2003, 2303 Logic circuit 1505 PPD circuit 1607, 2317 P type MOSFET
1609, 2319 Buffer 1701 Latch 1703 2-input OR gate 1801, 2101 PPD
1803, 1805, 1807, 1809, 1811 First to fifth transistors 2005 Second PPD circuit 2103, 2105, 2107, 2111, 2113, 2115, 2117, 2119 First to ninth transistors 2301a to 2301n One or more SPADs Circuit 2305 Third PPD circuit 2351, 2353, 2355, 2357, 2359, 2361, 2363, 2365, 2367, 2369, 2371, 2373, 2375 1st to 13th transistors

Claims (16)

An image sensor,
a plurality of first signals corresponding to each light pulse in response to detecting by at least one pixel one or more photons reflected from the object in response to a light pulse directed toward the object; a time-resolved sensor including a plurality of pixels outputting a second pair of signals;
a processor that determines a surface reflectance of the object from which a plurality of light pulses are reflected based on the plurality of first and second signal pairs;
a first ratio of the amplitude of the first signal of the pair to the sum of the amplitude of the first signal of the pair and the amplitude of the second signal of the pair is proportional to the time of flight of the detected one or more photons; ,
a second ratio of the amplitude of the second signal of the pair to the sum of the amplitude of the first signal and the amplitude of the second signal of the pair is proportional to the time of flight of the detected one or more photons; ,
At least one pixel of the plurality of pixels is
at least one SPAD, each single photon avalanche diode (SPAD) outputting a signal based on detecting one or more photons reflected from the object and incident on the SPAD in response to an active shutter signal;
a first activation signal activated in response to the initiation of the active shutter signal and deactivated in response to the output signal of the at least one SPAD; and activated in response to the output signal of the at least one SPAD. a logic circuit coupled to an output signal of the at least one SPAD to generate a second activation signal that is activated and deactivated in response to termination of the active shutter signal;
An image sensor comprising: a differential time-to-charge conversion (DTCC) circuit coupled to the first activation signal and the second activation signal. The image sensor according to claim 1, wherein the processor further determines a distance to the object based on the plurality of pairs of first and second signals. The image sensor of claim 1 , wherein the processor generates a grayscale image of the object based on the plurality of first and second signal pairs. The image of claim 3 , wherein the processor generates at least one histogram of arrival times of photons detected by predetermined pixels of the plurality of pixels to generate the grayscale image. sensor. The DTCC circuit is
a capacitive device including a first terminal and a second terminal coupled to ground potential;
a first terminal coupled to the first terminal of the capacitive device, a second terminal coupled to the first floating diffusion node, and a third terminal coupled to the first activation signal; a first switching device that transfers a first charge of the capacitive device to the first floating diffusion node in response to a signal;
a first terminal coupled to a first terminal of the capacitive device, a second terminal coupled to a second floating diffusion node, and a third terminal coupled to the second activation signal; a second switching device responsive to a signal to transfer the remaining charge of the capacitive device to the second floating diffusion node;
outputting a pair of the first signal including a first voltage based on the first charge on the first floating diffusion node and the second signal including a second voltage based on the remaining charge on the second floating diffusion node; The image sensor according to claim 1 , further comprising an output circuit that performs the following steps. The image sensor further includes a drive signal that varies based on a slope function;
the drive signal begins changing in response to a start time of a light pulse in which the one or more photons are detected and changes until the end of the active shutter signal;
The drive signal is
when the first activation signal is active, the first activation signal is coupled to a third terminal of the first switching device;
The image sensor of claim 5 , wherein when the second activation signal is active, it is connected to a third terminal of the second switching device. a first ratio of the first voltage to the sum of the first voltage and the second voltage is further proportional to a time of flight of the one or more photons minus a delay time;
a second ratio of the second voltage to the sum of the first voltage and the second voltage is further proportional to a time of flight of the one or more photons minus a delay time;
The image sensor according to claim 6 , wherein the delay time includes the time from the start of the transmission time of the optical pulse to the time when the drive signal starts changing. The image sensor according to claim 6 , wherein the capacitive device is a capacitor or a buried diode. An imaging unit,
a light source illuminating the object with a series of light pulses directed toward the surface of the object;
a plurality of first signals synchronized with the light source and corresponding to each light pulse in response to detecting at least one pixel one or more photons corresponding to a light pulse reflected from a surface of the object; and a time-resolved sensor including a plurality of pixels outputting a second pair of signals;
determining a distance to the object based on the plurality of first and second signal pairs; a processor for determining surface reflectance of the object;
a first ratio of the amplitude of the first signal of the pair to the sum of the amplitude of the first signal of the pair and the amplitude of the second signal of the pair is proportional to the time of flight of the detected one or more photons; ,
a second ratio of the amplitude of the second signal of the pair to the sum of the amplitude of the first signal and the amplitude of the second signal of the pair is proportional to the time of flight of the detected one or more photons; ,
At least one pixel of the plurality of pixels is
at least one SPAD, each single photon avalanche diode (SPAD) outputting a signal based on detecting one or more photons reflected from the object and incident on the SPAD in response to an active shutter signal;
a first activation signal activated in response to the initiation of the active shutter signal and deactivated in response to the output signal of the at least one SPAD; and activated in response to the output signal of the at least one SPAD. a logic circuit coupled to an output signal of the at least one SPAD to generate a second activation signal that is activated and deactivated in response to termination of the active shutter signal;
An imaging unit comprising: a differential time-to-charge conversion (DTCC) circuit coupled to the first activation signal and the second activation signal . 10. The imaging unit of claim 9 , wherein the processor further generates a grayscale image of the object based on the plurality of surface reflectances. Imaging according to claim 10 , wherein the processor generates at least one histogram of arrival times of photons detected by predetermined pixels of the plurality of pixels to generate the grayscale image. unit. The DTCC circuit is
a capacitive device including a first terminal and a second terminal coupled to ground potential;
a first terminal coupled to the first terminal of the capacitive device, a second terminal coupled to the first floating diffusion node, and a third terminal coupled to the first activation signal; a first switching device that transfers a first charge of the capacitive device to the first floating diffusion node in response to a signal; a first terminal coupled to the first terminal of the capacitive device; a second terminal coupled to the second activation signal, and a third terminal coupled to the second activation signal to transfer the remaining charge of the capacitive device to the second floating diffusion node in response to the second activation signal. a second switching device,
the first signal includes a first voltage based on the first charge of the first floating diffusion node;
the second signal includes a second voltage based on the remaining charge of the second floating diffusion node;
The imaging unit further includes a drive signal that begins changing in response to a start time of a light pulse in which the one or more photons are detected and changes until the end of the active shutter signal;
The drive signal is
when the first activation signal is active, the first activation signal is coupled to a third terminal of the first switching device;
The imaging unit of claim 9 , wherein when the second activation signal is active, it is coupled to a third terminal of the second switching device. Imaging unit according to claim 12 , characterized in that the capacitive device is a capacitor or a buried diode. A method of generating a grayscale image of an object, the method comprising:
directing a series of light pulses from a light source toward the surface of the object;
detecting , at least one pixel, one or more photons corresponding to a light pulse reflected from a surface of the object;
responsive to detecting the one or more photons by a time-resolved sensor synchronized to the light source and including a plurality of pixels; generating a pair of first and second signals;
determining, by a processor, a distance to the object based on the plurality of pairs of first and second signals;
determining, by the processor, a surface reflectance of the object based on at least one of the plurality of pairs of first and second signals, the amplitude of the first signal of the pair; a first ratio of the amplitude of the first signal of the pair to the sum of the amplitude of the second signal is proportional to the time of flight of the detected one or more photons;
a second ratio of the amplitude of the second signal of the pair to the sum of the amplitude of the first signal and the amplitude of the second signal of the pair is proportional to the time of flight of the detected one or more photons; ,
At least one pixel of the plurality of pixels is
at least one SPAD, each single photon avalanche diode (SPAD) outputting a signal based on detecting one or more photons reflected from the object and incident on the SPAD in response to an active shutter signal;
a first activation signal activated in response to the initiation of the active shutter signal and deactivated in response to the output signal of the at least one SPAD; and activated in response to the output signal of the at least one SPAD. a logic circuit coupled to an output signal of the at least one SPAD to generate a second activation signal that is activated and deactivated in response to termination of the active shutter signal;
a differential time-to-charge conversion (DTCC) circuit coupled to the first activation signal and the second activation signal . 15. The method of claim 14 , comprising generating, by the processor, a grayscale image of the object based on the plurality of first and second signal pairs. 16. The method further comprising: generating, by the processor, at least one histogram of arrival times of photons detected by predetermined pixels of the plurality of pixels to generate the grayscale image. The method described in.

Images