KR20200011351A - A time-resolving image sensor for range measurement and 2d greyscale imaging - Google Patents

Abstract

An image sensor comprises a time-resolved sensor and a processor. The time-resolved sensor outputs a pair of a first signal and a second signal in response to detecting, by at least one pixel, one or more photons reflected from an object corresponding to a light pulse irradiated toward the object. A first ratio of the amplitude of the first signal to the sum of the amplitude of the first signal and the amplitude of the second signal is proportional to flight time of the one or more detected photons. A second ratio of the amplitude of the second signal to the amplitude of the first signal and the amplitude of the second signal is proportional to the flight time of the one or more detected photons. The processor determines a surface reflectance of the object to which the light pulse is reflected based on the pair of the first signal and the second signal.

Description

A TIME-RESOLVING IMAGE SENSOR FOR RANGE MEASUREMENT AND 2D GREYSCALE IMAGING}

The subject matter described herein relates generally to image sensors. More specifically, the subject matter described herein relates to a time-of-flight (TOF) image sensor that also produces a gray scale image from accumulated photon-detecting events.

Three-dimensional (3D) imaging systems are increasingly used in a wide variety of applications such as industrial manufacturing, video games, computer graphics, robotic surgeries, consumer displays, surveillance images, 3D modeling, real estate sales, etc. It is used a lot. Existing 3D imaging techniques include, for example, time-of-flight (TOF) based distance imaging, stereo vision systems, and structured light (SL) methods.

In the TOF method, the distance to the 3D object is determined by measuring the round trip time taken for the optical signal to travel between the camera and the 3D object for each point in the image, based on the known speed of light. TOF cameras can capture the entire scene with each laser or light pulse using a scannerless method. Some exemplary applications of the TOF method include improved vehicle applications such as active pedestrian protection or pre-collision detection based on street images in real time; Tracking people's movements, such as during interaction with games on a video game console; And identifying objects such as items on the conveyor belt in industrial machine vision and helping robots find the items.

In stereoscopic-imaging or stereo-vision systems, two cameras arranged horizontally with each other are used to obtain two different fields of view in the scene or 3D object in the scene. By comparing these two images, relative depth information for the 3D object can be obtained. Stereo vision is particularly important in fields such as robotics and extracts information about the relative position of 3D objects near autonomous systems / robots. Other applications for robotics include object recognition, where stereoscopic depth information separates the image components intercepted by the robotic system, in which case the robot is placed in front of one another Cannot distinguish two objects that partially or completely hide 3D stereo displays can also be used in entertainment and automated systems.

In the SL method, the 3D shape of the subject can be measured using the illuminated light pattern and the camera for imaging. In the SL method, a known pattern of light (often gratings or horizontal lines or patterns of parallel stripes) is irradiated to the scene or 3D object within the scene. The irradiated pattern can be deformed or replaced when impacting the surface of the 3D object. This modification can allow the SL vision system to calculate the depth and surface information of the object. Thus, irradiating narrowband light onto a 3D surface can produce lines of light that can appear distorted from the perspective of the projector and other perspectives, and can be used for geometric reconstruction of the illuminated surface shape. SL-based 3D imaging can be used for a variety of applications, such as the imaging of fingerprints in 3D scenes by police, inline inspection of components during the manufacturing process, and healthcare for vivid measurements of human forms and / or skin microstructures. Can be used in applications.

It is an object of the present invention to provide an apparatus and method for sensing light pulses reflected from a subject, thereby generating both 3D and 2D images of the subject.

Exemplary embodiments provide an image sensor that may include a time-resolved sensor and a processor. The time-resolved sensor may comprise at least one pixel, and the first signal in response to detecting by the at least one pixel one or more photons reflected from the object corresponding to the light pulse irradiated towards the object. And a pair of second signals, wherein the first ratio of the amplitude of the first signal of the pair to the sum of the amplitude of the pair of first signals and the amplitude of the second signal is one or more detected photons. And the second ratio of the amplitude of the first signal of the pair and the amplitude of the second signal of the pair to the amplitude of the second signal is proportional to the flight time of one or more detected photons. The processor may determine the surface reflectance of the object on which the light pulse is reflected, based on the pair of the first signal and the second signal. The processor may further determine a distance to the object based on the images of the first signal and the second signal. In one embodiment, in response to detecting one or more photons reflected from the object at a pixel for a plurality of light pulses directed toward the object, the time-resolved sensor is configured to determine the plurality of first and second signals. Pairs, wherein each pair of first and second signals corresponds to a respective light pulse, and the processor is based on the plurality of first and second signal pairs, The surface reflectance of the reflected object may be determined.

An example embodiment provides an imaging unit that may include a light source, a time-resolved sensor, and a processor. The light source can illuminate the object with a series of light pulses that are directed toward the representation of the object. The time-resolved sensor can include at least one pixel, can be synchronized with the light source, and in response to detecting one or more photons corresponding to the light pulses reflected from the surface of the object at at least one pixel. Outputs a pair of the first signal and the second signal, wherein a first ratio of the amplitude of the first signal of the pair to the sum of the amplitude of the first signal of the pair and the amplitude of the second signal is one or more detected; And the second ratio of the amplitude of the first signal of the pair and the amplitude of the second signal of the pair to the amplitude of the second signal may be proportional to the flight time of one or more detected photons. have. The processor may determine the distance to the object based on the images of the first signal and the second signal, and determine the surface reflectance of the object on which the light pulse is reflected based on the pair of the first signal and the second signal. In one embodiment, in response to detecting one or more photons reflected from an object in a pixel, the time-resolved sensor may output a plurality of pairs of first and second signals, where the first signal And each of the pair of second signals may correspond to each of the plurality of light pulses irradiated toward the object. The processor may further determine the plurality of surface reflectances of the object based on the pair of corresponding first and second signals. In one embodiment, the processor may further generate a gray scale image of the object based on the plurality of surface reflectances.

An example embodiment provides a method of generating a gray scale image of a subject, wherein the method includes: irradiating a series of light pulses from a light source towards a surface of the subject; Detecting one or more photons corresponding to the light pulses reflected from the surface of the object in the pixel; Detecting more photons;

Generating, by the time-resolved sensor, a pair of the first signal and the second signal in response to detecting one or more photons, the time-resolved sensor being synchronized with the light source, the amplitude of the first signal of the pair and The first ratio of the amplitude of the first signal of the pair to the sum of the amplitudes of the second signal is proportional to the flight time of one or more detected photons and to the amplitude of the first signal of the pair and the amplitude of the second signal. A second ratio of the amplitudes of the second signal of the pair for is proportional to the flight time of one or more detected photons; Determining, by the processor, a distance to the object based on the pair of the first signal and the second signal; And determining, by the processor, the surface reflectance of the object based on the pair of the first signal and the second signal. In one embodiment, the method includes detecting one or more photons reflected from the object in a pixel for a plurality of light pulses irradiated toward the object, each of the plurality of first and second pairs of signals Corresponding to each light pulse in the series of light pulses; And determining, by the processor, the surface reflectance of the object based on at least one of the plurality of first and second signal pairs. In one embodiment, the method may further include generating, by the processor, a gray scale image by generating at least one histogram of the arrival times of the photons detected by the predetermined pixel of the plurality of pixels.

According to the present invention, the amount of charge charge transferred is converted into the first signal from the irradiation of the light pulse toward the object until the light pulse reflected from the object is detected. The amount of residual charge in the charge charge is converted into a second signal. Based on the first and second signals, the flight time of the light pulse is calculated, and the distance from the flight time is calculated to produce a 3D image. Based on the first and second signals, the power of the reflected light pulse is calculated, and the reflectance is calculated from the calculated power and the calculated distance, producing a 2D gray scale image of the object.

In the following section, aspects of the subject matter described herein will be described with reference to exemplary embodiments shown in the figures.
1 shows a very simplified partial configuration of an image-sensor system according to the subject matter described herein.
2 shows an exemplary operational configuration of the image-sensor system of FIG. 1 in accordance with the subject matter described herein.
3 shows a flowchart of an example of how 3D depth measurements are performed, according to the subject matter described herein.
4 shows how an exemplary point scan can be performed for 3D-depth measurements, in accordance with the subject matter described herein.
5 shows a block diagram of an example pixel according to the subject matter described herein.
6A-6C each show three different examples of pixel array architectures in accordance with the subject matter described herein.
7 shows circuit details of an example embodiment of a pixel according to the subject matter described herein.
8 is an exemplary timing diagram providing an overview of the charge transfer mechanism modulated at the pixel of FIG. 7, according to the subject matter described herein.
9 is a timing diagram of exemplary timing of different signals in the image-sensor system of FIGS. 1 and 2 when the pixels of the embodiment of FIG. 7 are used in a pixel array for measuring TOF values, according to the subject matter described herein. to be.
10 shows how a logic unit is implemented in a pixel according to the subject matter described herein.
11 shows an exemplary flow chart showing how TOF values can be determined in the image-sensor system of FIGS. 1 and 2, according to the subject matter described herein.
12 is an exemplary layout of a portion of an image-sensor portion according to the subject matter described herein.
13 shows another exemplary embodiment of a pixel according to the subject matter described herein.
14 shows exemplary timings of different signals of the image-sensor system of FIGS. 1 and 2 when the pixels of the embodiment shown in FIG. 13 are used in a pixel array for measuring TOF values, according to the remedy described herein. Timing diagram.
15 shows a block diagram of an exemplary embodiment of a time-resolved sensor in accordance with the subject matter described herein.
FIG. 16 shows a conceptual diagram of an exemplary embodiment of the SPAD circuit of the time-resolved sensor of FIG. 15, in accordance with the subject matter described herein.
FIG. 17 shows a conceptual diagram of an example embodiment of the logic circuit of the time-decomposition sensor of FIG. 15, in accordance with the subject matter described herein.
18 shows a conceptual diagram of an exemplary embodiment of the pinned photodiode of the time-resolved sensor of FIG. 15, in accordance with the subject matter described herein.
19 shows an exemplary relative signal timing diagram for the time-resolved sensor of FIG. 15 in accordance with the subject matter described herein.
20 shows a block diagram of another exemplary embodiment of a time-resolved sensor in accordance with the subject matter described herein.
FIG. 21 shows a conceptual diagram of an exemplary embodiment of a second PPD circuit of the time-decomposition sensor of FIG. 20 in accordance with the subject matter described herein.
FIG. 22 shows an exemplary burn timing diagram for the time-resolved sensor of FIG. 20 in accordance with the subject matter described herein.
23 shows a block diagram of another exemplary embodiment of a time-resolved sensor in accordance with the subject matter described herein.
24 shows an exemplary relative signal timing diagram for the time-resolved sensor of FIG. 23 in accordance with the subject matter described herein.
FIG. 25 shows a flow chart of a method of decomposing time using the time-decomposition sensor of FIG. 23 in accordance with the subject matter described herein.
26A shows an exemplary trigger waveform output from SPAD.
26B shows an example histogram of light-detection times of an example pixel that may be formed according to the subject matter described herein.
FIG. 26C shows an exemplary histogram according to the subject matter described herein, where a window width representing the FWHM of the irradiated pulses (not shown) is displayed, and an event count maximum can be determined.
FIG. 26D shows an exemplary histogram according to the subject matter described herein, wherein the trigger waveform output from the SPAD (FIG. 26) is wound into a histogram to determine the event count maximum.
27 shows example histograms for example pixels according to the subject matter described herein.
28 shows a flowchart of an example method of generating a depth or distance map and a gray scale image of a scene according to the subject matter described herein.
29A shows an example scene.
29B and 29C show example depth maps and example gray scale images of the scene shown in FIG. 29A, respectively, according to the subject matter described herein.
30 shows an exemplary embodiment of the overall layout of the imaging system shown in FIGS. 1 and 2 according to the subject matter described herein.

In the following detailed description, numerous specific details are provided to provide a thorough understanding of the present disclosure. However, it will be understood by those skilled in the art that the described aspects may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail in order not to obscure the subject matter described herein. In addition, the described aspects may be implemented to perform low power, 3D depth measurements in any imaging device or system, including, but not limited to, a smartphone, a user device (UE), and / or a laptop computer. .

Reference herein to 'one embodiment' or 'an embodiment' means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment described herein. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” or “according to an embodiment” (or other phrases having similar meanings) in various places in the specification are all equivalent to the same embodiment. It should not be considered to be a reference. In addition, certain features, structures or features may be combined in any suitable manner in one or more embodiments. In this regard, as used herein, the word 'model' means 'provided as one example, example or illustration'. Any embodiment described herein as 'exemplary' should not be considered preferred or advantageous over other embodiments. In addition, certain features, structures or features may be combined in any suitable manner in one or more embodiments. Also, depending on the context discussed herein, the singular forms may include corresponding plural forms, and the plural forms may include corresponding singular forms. Similarly, terms that are hyphenated (e.g. '2-dimensional', 'pre-determined', 'pixel-specific', etc.) sometimes have corresponding hyphen free versions (e.g. 'two-dimensional', 'predetermined'). "," Pixel specialization ", etc.), and uppercase English letters (e.g.," Counter Clock "," Row Select "," PIXOUT ", etc.) It can be used in place of a missing version (e.g. 'counter clock', 'row select', 'pixout', etc.). Such occasional alternative uses should not be considered inconsistent with each other.

Also, depending on the context described herein, the singular forms may include corresponding plural forms, and the plural forms may include corresponding singular forms. The various drawings (including component diagrams) shown and discussed herein are for illustrative purposes only and are not limited to the same proportions. Likewise, various waveforms and timing diagrams are shown for purposes of explanation only. For example, some dimensions of the elements may be emphasized over others for clarity. Also, where considered appropriate, reference numerals are repeated in the figures to indicate corresponding and / or similar elements.

The terminology used herein is for the purpose of describing some example embodiments only and is not intended to be limiting of the claimed subject matter. As used herein, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the terms 'comprises' and / or 'comprising' specify the presence of the recited features, integers, steps, actions, elements and / or components, and one or more. It does not exclude the presence of many other features, integers, steps, actions, elements, components, and / or groups thereof. As used herein, the terms 'first', 'second' and the like are used as markers for preceding nouns and, unless expressly so defined, in any form of order (e.g., spatial, Temporal, logical, etc.). In addition, the same reference numerals may be used in two or more figures to refer to parts, components, blocks, circuits, units or modules having the same or similar function. However, this use is for simplicity of explanation and ease of discussion only, and the structural or structural details of these components or units may be the same or such commonly referenced parts / modules throughout all embodiments described herein. It does not imply that it is the only way to implement some of the embodiments.

When an element or layer is referred to as being above, connected or combined with another element or hierarchy, that element or hierarchy may be directly above or directly connected or directly joined to another element or hierarchy, or intervening elements or hierarchy It will be understood that these may exist. Conversely, when an element is referenced directly above, directly connected to, or directly coupled to another element or hierarchy, there are no intervening elements or layers. Like reference numerals refer to like elements throughout. As used herein, the term 'and / or' includes any and all combinations of one or more of the articles listed in association.

As used herein, the terms 'first', 'second' and the like are used as markers for preceding nouns and, unless so explicitly defined, in any form of order (e.g., spatial, Temporal, logical, etc.). Also, like reference numerals used throughout the two or more figures refer to parts, components, blocks, circuits, units or modules having the same or similar functionality. This use, however, is for simplicity of explanation and ease of discussion only, and the structure or structural details of these components or units may be the same or such commonly referenced parts / modules throughout all embodiments described herein. It is not implied that it is the only way to implement some of the illustrative embodiments.

As shown in the figures, spatially relative terms such as 'bottom', 'bottom', 'low', 'top', 'high', etc., are used herein to refer to one element or feature of another element (s) or feature (s). Are used for ease of explanation describing the relationship. It will be understood that the spatially relative terms are intended to include other directions of the device in use or operation in addition to the direction shown in the figures. For example, if the apparatus of the figures is turned upside down, elements described as 'below' or 'below' of other elements or features may be in the 'up' direction of other elements or features. Thus, the term 'below' encompasses both up and down directions. If the device is oriented differently (rotated 90 degrees or in other directions), the spatially relative descriptions used herein are interpreted accordingly.

Unless defined otherwise, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this subject belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the associated field, and unless expressly so defined herein, will not be idealized or overly formal.

As used herein, the term 'module' refers to any combination of software, firmware and / or hardware that is configured to associate with a module and provide the functionality described herein. Software is embodied in software packages, code and / or instruction sets or instructions, and the term 'hardware' used in any implementation described herein refers to programmable circuitry, for example, alone or in any combination. Hardware hardwired circuitry, programmable circuitry, state machine circuitry, and / or firmware that stores instructions executed by the processor. Modules may be mounted collectively or separately, for example, but not limited to, circuitry that forms part of a larger system such as an integrated circuit (IC), a system-on-chip (SoC), or the like.

The aforementioned 3D techniques have many problems. For example, a TOF based 3D imaging system may require high power to operate optical or electrical shutters. Such systems typically operate over a range of several meters to tens of meters, but the resolution of such systems decreases in short distance measurements, thus making it impossible to produce 3D images within a distance of about 1 meter. Thus, the TOF system is undesirable for cell phone based camera applications where pictures are mostly taken at close ranges. TOF sensors also require special pixels with pixel sizes usually greater than 7 um. These pixels are also vulnerable to ambient light.

Stereoscopic imaging methods generally operate only on textured surfaces. This has a high computational complexity due to the need to match features and find correspondences between the images of the stereo pair of the object. This requires high system power and is not a suitable feature for applications that require power savings, such as smartphones. In addition, stereo imaging requires two permanent high bit resolution sensors along with two lenses, making the entire assembly inadequate for applications in portable devices such as mobile phones or tablets that are out of device space.

The SL method exhibits ambiguity in distance and also requires high system power. For 3D depth measurements, the SL method may require multiple images with multiple patterns. All of these increase computational complexity and power consumption. In addition, SL imaging may require permanent image sensors with high bit resolution. Thus, a structured light based system may not be suitable for low cost, low power, compact image sensors of smartphones.

In contrast to the 3D technologies mentioned above, some embodiments described herein provide for implementing a low power 3D imaging system in portable electronic devices such as smart phones, tablets, user devices (UEs). A 2D imaging sensor in accordance with some embodiments described herein captures 3D depth measurements with 2D RGB (red, green, blue) images and visible light laser scan and can block ambient light during 3D depth measurements. Although the following technique frequently mentions a visible light laser as a light source for point scan, and a 2D RGB sensor as an image / light capture device, this reference is for the purpose of consistency of explanation and discussion only. Visible light laser and RGB sensor based examples described below may find applications in low power, consumer-grade mobile electronic devices with a camera such as smartphones, tablets or user devices (UEs). have. However, the subject matter described herein is not limited to the visible light-RGB sensor-based examples mentioned below. Instead, according to some embodiments described herein, the point-scan based 3D depth measurements and the ambient light blocking method are not limited thereto, but (i) the laser source is red (R), green (G), or blue. (B) a 2D color (RGB) sensor with a visible light laser source wherein the light laser or laser source produces a combination of these lights; (ii) a visible light laser with a 2D RGB color sensor with an infrared (IR) blocking filter; (iii) a near infrared (NIR) laser with a 2D IR sensor; (iv) a NIR laser with a 2D NIR sensor; (v) a NIR laser with a 2D RGB sensor (without IR cutoff filter); (vi) a NIR laser with a 2D RGB sensor (without NIR blocking filter); (vii) a 2D RGB-IR sensor with a visible or NIR laser; (viii) can be performed using a number of different combinations of 2D sensors such as 2D RGBW (red, green, blue, white) sensor with visible or NIR laser and laser light sources (for point scans).

During 3D depth measurements, all sensors act as laser scans and rough binary sensors to reconstruct 3D content. In some embodiments, the pixel size of the sensor may be as small as 1 um. In addition, due to the low bit resolution, analog-to-digital converter (ADC) units of the image sensor according to some embodiments described herein typically require much less processing power than is required for high bit resolution sensors in a 3D imaging system. You can do Due to the need for less processing power, the 3D imaging module according to the subject matter described here requires low system power and is therefore well suited for inclusion in low power devices such as smartphones.

In some embodiments, the subject matter described herein uses triangulation and point scans with a laser light source for 3D depth measurements with a group of line sensors. The laser scan plane and imaging plane are directed using epipolar geometry. An image sensor according to one embodiment described herein uses timestamps to eliminate ambiguity in the triangulation method, thus reducing the amount of depth operations and system power. The same image sensor (i.e. each pixel of the image sensor) can be used in the 2D (RGB color or non-RGB) imaging mode as well as the 3D laser scan mode. However, in laser scan mode, the resolution of the ADCs of the image sensor is reduced to binary output (1 bit resolution), thus improving read speed and reducing power consumption (eg, ADC unit) in the chip comprising the image sensor and associated processing units. Due to switching in the field). The point scan method allows the system to make all measurements at once, thus reducing latency in depth measurements and reducing motion blur.

As mentioned above, in some embodiments, the entire image sensor may be used for 3D depth imaging using visible laser scan, as well as for conventional 2D RGB color imaging with ambient light, for example. Dual use of this same camera unit can save space and cost of mobile devices. In certain applications, visible lasers for 3D applications may be better for user eye safety compared to near infrared (NIR) lasers. The sensor has a higher quantum efficiency in the visible spectrum than in the NIR spectrum and can lead to less power consumption of the light source. In one embodiment, the dual use image sensor may operate in a linear mode of operation for 2D imaging as a conventional 2D sensor. However, for 3D imaging, the sensor operates in linear mode under normal light conditions, and in logarithmic mode under strong ambient light, enabling the continuous use of visible laser sources through the blocking of strong ambient light. Ambient light blocking may also be required for NIR lasers, for example, when the bandwidth of the pass band of the IR blocking filter employed in the RGB sensor is not narrow enough.

In summary, the present disclosure discloses that a time-to-charge converter (TCC) with amplitude-modulated charge-transfer operation is obtained from multiple adjacent SPADs in a pixel. The pinned photodiode (PPD) is used in the pixel when determining the TOF, controlled by the outputs of the < RTI ID = 0.0 > When the ambient light is high, there is a high probability that the SPAD will be triggered by the ambient photons instead of the reflected photons (eg, within the reflected pulse 37). Dependence on such a trigger can cause a ranging error. Thus, in this disclosure, PPD charge transfer is stopped to write TOF only when two or more SPADs are triggered within a very short, predefined time interval, such as when the electronic shutter is on. As a result, an all-weather autonomous driving system in accordance with the teachings of the present disclosure may provide drivers with improved visibility in difficult driving conditions, such as, for example, low light, fog, bad weather, strong ambient light, and the like. In some embodiments, the travel system according to the teachings of the present disclosure can have high ambient light shielding up to 100 kLux. In some embodiments, a higher spatial resolution pixel structure with a smaller pixel size may be provided at a SPAD to PPD ratio of 1: 1. In some embodiments, the SPADs are biased below the breakdown voltage, and can be used in avalanche photodiode (APD) mode.

1 shows a very simplified partial configuration of an imaging system 150 according to the subject matter described herein. System 15 may include processor module 19 or imaging module 17 in communication with and in communication with a host. System 15 may also include a memory module 20 that is coupled with processor module 19 and stores content such as image data received from imaging module 17. In some embodiments, the entire system 150 may be encapsulated in a single integrated circuit (IC) or chip. Alternatively, each of the modules 17, 19, 20 may be implemented as a separate chip. Memory module 20 may include one or more memory chips, and processor module 19 may also include multiple processing chips. The details of the packaging of the modules of FIG. 1 and the details of how the modules are manufactured or implemented (either on a single chip or on a plurality of separate chips) are not relevant to the present discussion and thus these details are provided herein. It doesn't work.

System 15 may be any low power electronic device configured for 2D and 3D camera applications in accordance with the subject matter described herein. System 15 may or may not be portable. Some examples of portable versions of system 15 include, but are not limited to, mobile devices, mobile phones, smart phones, user devices (UEs), tablets, digital cameras, laptops or desktop computers, electronic smart watches, machine-to- Popular consumer electronic tools such as machine (M2M) communication units, virtual reality (VR) devices or modules, robots, and the like. Conversely, some examples of non-portable versions of system 15 are game consoles in interactive video arcades, interactive video terminals, automobiles, machine vision systems, industrial robots, VR devices, and driver-side mounting cameras (e.g. Dozing and observing whether or not) and the like. The 3D imaging functions described herein include, but are not limited to, automotive applications such as all-weather autonomous driving and driver assistance in low light or inclement weather conditions, human-machine interface and gaming applications, machine vision and robotics applications. It can be used in many applications such as

In some embodiments described herein, the imaging module 17 may include a projector module (or light source module) 22 and an image sensor unit 24. The light source of the projector module 22 may be, for example, an infrared (IR) laser, such as a near infrared (NIR) or short wave infrared (SWIR) laser, so that illumination is not noticeable. You may not. In other embodiments, the light source can be a visible light laser. The image sensor unit 24 may include a pixel array and auxiliary processing circuits as shown in FIG. 2.

In one embodiment, processor module 19 may be a central processing unit (CPU) that is a general purpose microprocessor. As used herein, the terms 'processor' and 'CPU' may be used interchangeably. However, instead of or in addition to the CPU, the processor module 19 may include, but is not limited to, other such as microcontrollers, digital signal processors (DSPs), graphics processing units (GPUs), dedicated application specific integrated circuits (ASIC) processors, and the like. It may include any type of processor. In one embodiment, processor module / host 19 may include more than one CPU operating in a distributed processing environment. The processor module 19 executes instructions, and is not limited to these, but x86 instruction set architecture (32-bit or 64-bit versions), Power PC®ISA, or MIPS that relies on the Reduced Instruction Set Computer (RISC) ISA. (Microprocessor without Interlocked Pipeline Stages) may be configured to process data according to a particular instruction set architecture (ISA), such as an instruction set structure. In one embodiment, the processor module 19 may be a system on chip (SoC) with additional functionality in addition to the CPU functionality.

In some embodiments, memory module 20 may be, but is not limited to, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), or a high bandwidth memory (HBM) module or hybrid memory. DRAM-based three-dimensional stack (3DS) memory modules, such as cube (HMC) memory modules. In other embodiments, memory module 20 may be a solid state drive (SSD), a non-3DS DRAM module, or not limited to, but not limited to, static random access memory (SRAM), phase change random access memory (PRAM or PCRAM), Or any other semiconductor based storage system such as resistive random access memory (RRAM or ReRAM), conductive-bridge RAM (CBRAM), magnetic RAM (MRAM), or spin transfer torque MRAM (STT-MRAM).

2 shows an exemplary operating configuration of the imaging system 15 of FIG. 1 in accordance with the subject matter described herein. System 15 may be used to obtain distance or depth information (along the Z-axis) for an object, such as object 26, which may be an individual object or an object within a scene (not shown). System 15 may be a direct TOF imager in which a single pulse per image frame (of an array of pixels) may be used. In some embodiments, multiple short pulses may be sent to the object 26. In one embodiment, the distance / depth information may be determined by the processor module 19 based on scan data received from the image sensor unit 24. In other embodiments, distance / depth information may be determined by the image sensor unit 24. In some embodiments, depth information is used by processor module 19 as part of a 3D user interface such that a user of system 15 communicates with or executes a 3D image of an object in system 15. It can be used as part of other applications such as games or autonomous driving applications. 3D imaging according to the subject matter described herein can also be used for other purposes or applications, and can be applied to virtually any scene or 3D object.

In FIG. 2, the X axis is in the horizontal direction along the front of the system 15, the Y axis is in the vertical direction (out of the page in the field of view), and the Z axis extends in the general direction of the object 26 to be imaged from the system 15. do. For depth measurements, the optical axes of the modules 22, 24 can be parallel to the Z axis. Other optical arrangements may be used to implement the principles described herein, and such alternative arrangements are considered to be within the scope of the subject matter described herein.

Associated with corresponding dotted lines 30, 31 representing an illumination path of an optical beam or optical radiation that can be used to point scan the object 26 within an optical field of view (FOV). As indicated by arrows 28 and 29, projector (or light source) module 22 may illuminate object 26. The point-by-line point scan of the object surface may be performed using an optical radiation source, which in one embodiment may be a laser light source 33 operated and controlled by the laser controller 34. The light beam from the laser source 33 is point scanned in the X-Y direction across the surface of the object 26 via the projection device 35 under the control of the laser controller 34. Point scan projects light points on the surface of the object along the scan line, as discussed in more detail with reference to FIG. 4. Projection device 35 may be a focusing lens, glass / plastic surface, or other cylindrical optical element that concentrates a laser beam from laser 33 point or point onto surface of object 26. In the embodiment shown in FIG. 2, a convex structure is shown as the focusing lens 35. However, any other suitable lens design can be selected as the projection device 35. The object 26 may be located at a concentrated position where the illumination light from the light source 33 is concentrated by the projection device 35 at the light point. Thus, in a point scan, the point or narrow area / point of the surface of the object 26 can be sequentially illuminated by the concentrated light beam from the projection device 35.

In some embodiments, the light source 33 (or illumination source) is a diode laser or light emitting diode (LED) that emits visible light, an NIR laser, a point light source, a monochromatic illumination source in the visible light spectrum (white lamp and monochromator ( such as a combination of monochromators), or any other form of laser light source. The laser 33 may be fixed at one location within the housing of the system 15, but may be rotatable in the X-Y directions. The laser 33 is X-Y addressable (eg, by the laser controller 34) to perform a point scan of the 3D object 26. In one embodiment, the visible light may be substantially green light. Visible light illumination from the laser source 33 can be projected onto the surface of the 3D object 26 using a mirror (not shown), or the point scan can be performed completely without the mirror. In some embodiments, the light source module 22 may include more or fewer components than those shown in the exemplary embodiment shown in FIG. 2.

In the embodiment of FIG. 2, light reflected from the point scan of the object 26 may travel along the collection path indicated by arrows 36 and 37 and dotted lines 38 and 39. The light collection path may carry photons reflected or scattered from the surface of the object 26 as illumination from the laser source 33 is received. Here, the display of the various transmission paths using solid arrows and dashed lines in FIG. 2 (and also applicable to FIG. 4) is for illustrative purposes only, and the illustration should not be considered to show any actual optical signal transmission paths. . In practice, the illumination and acquisition signal paths may differ from those shown in FIG. 2 and may not be clearly defined as shown in FIG. 2.

Light received from the illuminated object 26 may be focused on one or more pixels of the 2D pixel array 42 via the collecting device 44 of the image sensor unit 24. Like the projection device 35, the collection device 44 may be of a focused lens, glass / plastic surface, or other cylindrical shape that focuses the reflected light received from the object 26 to one or more pixels in the array 42. It may be an optical element. In the embodiment shown in FIG. 2, a convex structure is shown as the focusing lens 44. However, any other suitable lens design can be selected as the collecting device 44. Although pixel array 42 is shown in FIG. 2 as only a 3 × 3 pixel array, it should be understood that modern pixel arrays may include thousands or millions of pixels. The pixel array 42 may be an RGB pixel array in which different pixels may collect light signals of different colors. In some embodiments, pixel array 42 may be any 2D such as, but not limited to, 2D RGB sensor, 2D IR sensor, 2D NIR sensor, 2D RGBW sensor, 2D RGB-IR sensor with IR blocking filter. It may be a sensor. System 15 may use the same pixel array 42 for 3D imaging (including depth measurements) of object 26 as well as for 2D RGB color imaging of object 26.

The pixel array 42 may convert the received photons into corresponding electrical signals, which may then be processed by an associated pixel processing unit 46 to determine a 3D depth image of the object 26. . In one embodiment, pixel processing unit 46 may use triangulation for depth measurements. The triangulation method is described later with reference to FIG. 4. Pixel processing unit 46 may also include circuits for controlling the operation of pixel array 42.

The processor 19 may control the operations of the light source module 22 and the image sensor unit 24. For example, system 15 may have a mode switch (not shown) that may be controllable by a user to switch the 2D imaging mode to the 3D imaging mode. When the user selects the 2D imaging mode by using the mode switch, the processor 19 may activate the image sensor unit 24, but the 2D imaging mode uses ambient light so that the light source module 22 may not be activated. have. On the other hand, if the user selects the 3D imaging mode using the mode switch, the processor 19 can activate both modules 22 and 24, and also the reset (RST) signal in the pixel processing unit 46. By triggering a change in level, for example, if the ambient light is so strong that it is rejected by the linear mode (as described further below), it can change from linear mode to algebraic mode. The processed image data received from pixel processing unit 46 may be stored in memory 20 by processor 19. The processor 19 may also display a 2D or 3D image selected by the user on a display screen (not shown) of the system 15. The processor 19 may be programmed with software or firmware to perform the various processing tasks described herein. Instead or in addition, the processor 19 may include programmable hardware logic circuits to perform some or all of the functionality of the processor 19. In some embodiments, memory 20 may store program code, lookup tables, and / or intermediate computational results that cause processor 19 to provide the functions of processor 19.

3 shows a flowchart 50 of an exemplary embodiment of how 3D depth measurements may be performed in accordance with the subject matter described herein. The various operations shown in FIG. 3 may be performed by a single module or a combination of modules or system components in system 15. Certain tasks are described as being performed by specific modules or system components only by way of example. Other modules or system components may be appropriately configured to perform these tasks.

In FIG. 3, in operation 52, the system 15 (more specifically, the processor 19) uses a light source, such as the light source module 22, along a scan line to display a 3D, such as object 26 of FIG. 2. One-dimensional (1D) point scan of the object may be performed. As part of the point scan, the light source module 22 may be configured to project a series of light points on the surface of the 3D object 26 in a line-by-line manner, for example by the processor 19. In operation 54, pixel processing unit 46 of system 15 may select a row of pixels in an image sensor, such as 2D pixel array 42. Image sensor 42 may include a plurality of pixels arranged in a 2D array forming an image plane, wherein the selected row of pixels forms an epipolar line of the scanning line (in operation 52) on the image plane. . A brief description of the isopolar geometry is provided with reference to FIG. 4. In operation 56, pixel processing unit 46 may be cooperatively configured by processor 19 to detect each light point using a corresponding pixel in a row of pixels. Light reflected from the illumination light point may be detected by a single pixel or more than one pixel, such that light reflected from the illumination point is concentrated by two or more adjacent pixels by the collecting device 44. It should be noted that Light reflected from two or more light points may also be collected at a single pixel in the 2D pixel array 42. The timestamp based method may be used to remove ambiguity associated with depth calculations resulting from two different points being imaged by the same pixel or a single point being imaged by two different pixels. In operation 58, pixel processing unit 46 (as properly configured by processor 19) performs pixel-specific detection of the corresponding light point of the series of light points (of the point scan of operation 52). Responsive to), it may generate pixel-specific output. As a result, in operation 60, the pixel processing unit 46 is based on the scan angle used by the light source to project at least the pixel-specific output (in operation 58) and the corresponding light point (in operation 52), 3D. The 3D distance (or depth) for the corresponding light point on the surface of the object can be determined. Depth specification is described in more detail with reference to FIG. 4.

4 shows how an example point scan is performed for 3D depth measurements according to the subject matter described herein. In FIG. 4, the XY rotational capability of the laser source 33 is indicated by arrows 62, 64 showing the angular movement of the laser in the X-direction (with angle β) and Y-direction (with angle α). do. In one embodiment, the laser controller 34 may control the XY rotation of the laser source 33 based on scan commands / inputs received from the processor 19. For example, if the user selects a 3D imaging mode, the processor 19 may configure and control the laser controller 34 to initiate a 3D depth measurement of the object surface facing the projection device 35. In response, the laser controller 34 may initiate a 1D XY point scan of the object surface via the XY movement of the laser light source 33. As shown in FIG. 4, the laser 33 can point scan the surface of the object 26 by projecting light points along the 1D scanning lines, with two scanning lines S _R 66 and S _{R +. 1} (68) is indicated by dashed lines in FIG. Due to the bending of the surface of the object 26, the light points 70-73 form a scan line S _R 66 in FIG. 4. Light points forming scan line S _{R + 1} 68 are not indicated by reference numbers. The laser 33 can scan the object 26 one point at a time, for example, from left to right along the rows R, R + 1. The values of rows R, R + 1 refer to the rows of pixels of 2D pixel array 42 and are known. For example, in the 2D pixel array 42 of FIG. 4, the pixel row R is indicated using the reference numeral '75' and the row R + 1 is indicated using the reference numeral '76'. It will be appreciated that the rows R, R + 1 are selected from a plurality of rows of pixels for illustrative purposes only.

The plane containing the rows of pixels of the 2D pixel array 42 may be called an image plane, and the plane containing scan lines, such as lines S _R and S _{R + 1} , may be called a scan plane. In the embodiment shown in FIG. 4, the image plane and the scan plane are the isolines of the scan line S _R , S _{R + 1} to which each row R, R + 1 of the pixels in the 2D pixel array 42 corresponds. Oriented using isopolar geometry to form If the projection of the illumination point on the image plane (in the scan line) can form a separate point along the line that is the row R itself, then the row R of pixels can be considered as an isopolar line for the corresponding scan line. have. For example, in FIG. 4, the arrow 78 indicates the illumination of the light spot 71 by the laser 33, and the arrow 80 indicates that the light spot 71 is in a row R by the focusing lens 44. To be imaged or projected. Although not shown in FIG. 4, all of the light points 70-73 will be imaged at the corresponding pixels in row R. FIG. Thus, in one embodiment, the physical arrangement of laser 33 and pixel array 42, such as position and direction, is such that the illuminated light points of the scan line of the surface of object 26 correspond to that of pixel array 42. It can be captured or detected by the pixels in a row, the corresponding row of pixels forming the isolines of the scan line.

The pixels of the 2D pixel array 42 may be arranged in rows and columns. Illuminated light points can be referenced by corresponding rows and columns of pixel array 42. For example, in FIG. 4, the light point 71 of the scan line S _R is designated as X _{R, i such} that the point 71 has a row R and a column i (C _i ) of the pixel array 42. Can be imaged by Column C _i is indicated by dashed line 82. Other illuminated points can be similarly identified. As mentioned above, light reflected from two or more light points may be received by a single pixel in a row, or light reflected from a single light point may be received by one or more pixels in a row of pixels. Can be received. Timestamp based methods can be used to remove ambiguity in depth calculations resulting from such multiple or overlapping projections.

In FIG. 4, an arrow with reference numeral '84' indicates the depth or distance Z of the light point 71 from the X-axis, such as the X-axis shown in FIG. 2, along the front of the system 15. Along the axis). In FIG. 4, the dotted line with the reference numeral '86' represents an axis that can be visualized as being included in a vertical plane that also includes the projection device 35 and the collection device 44. However, for ease of triangulation based description, in FIG. 4 the laser source 33 is shown as being on the X-axis 86 instead of the projection device 35. In a triangulation based method, the value of Z can be determined using Equation 1 below.

In Equation 1, 'h' is the distance along the Z-axis between the collecting device 44 and the image sensor 42 and is considered to be in the vertical plane behind the collecting device 44. 'd' is the offset distance between the light source 33 and the collecting device 44 associated with the image sensor unit 24. 'q' is the pixel that detected the collecting device 44 and the corresponding light point (in the example of FIG. 4, the detection / imaging pixel i is a column C _i associated with the light point X _{R, i} 71). (Indicated by)). 'θ is the scan angle or beam angle of the light source with respect to the light point under consideration (in the example of FIG. 4, light point 71). Alternatively, 'q' may also be considered as the offset of the light point within the field of view of pixel array 42. The parameters of equation 1 are also represented in FIG. 4.

From Equation 1 it is shown that only parameters θq are variables for a given point scan, and h and d are in principle predetermined or fixed based on the physical geometry of the system 15. Since row R 75 is an isoline of scan line S _R , the depth difference or depth profile of object 26 is imaged in the horizontal direction as indicated by the values of q for the different light points being imaged. Can be reflected in the movement. The timestamp based method can be used to find correspondence between the pixel position of the captured light point and the corresponding scan angle of the laser source 33. That is, the time stamp may indicate a relationship between q and θ values. Thus, from the known value of the scan angle (θ) and the corresponding position of the imaged light point (as indicated by q), the distance Z to the light point can be determined using triangulation in equation (1). Triangulation for measurements is also described in related documents, including, for example, US Patent Application Publication No. 2011 / 0102763A1, filed by Brown et al. The disclosure of Brown's publications relating to the measurements is hereby incorporated by reference in its entirety.

5 shows a block diagram of an exemplary embodiment of a pixel, such as pixel 43 of pixel array 42 of FIG. 2 in accordance with the subject matter described herein. For TOF measurements, pixel 43 can act as a time-resolved sensor. As shown in FIG. 5, the pixel 43 may include a SPAD core portion 501 electrically connected to the PPD core portion 502. As described herein, other exemplary configurations of SPAD and PPD core arrangements in a pixel are shown in FIGS. 6A-6C. The SPAD core portion 501 may include two or more SPADs 503 operatively connected with the first control circuit 504. One or more SPADs 503 receive the incident light 505 and generate a corresponding SPAD-specific electrical signal, which is processed by the first control circuit 504 to be SPAD-specific digital. You can generate output. All of these SPAD-specific digital outputs are represented integrally and symbolically by arrow 506 in FIG. 5. The PPD core 502 may include a second control circuit 507 coupled with the PPD 508. The second control circuit 507 can receive the SPAD outputs 506 and in response to control the charge transfer from the PPD 508 to produce a pixel-specific analog output (PIXOUT) 510. . More specifically, as discussed in greater detail below, when two or more adjacent SPADs 503 in pixel 43 detect (reflected) photons within a predetermined time interval in incident light 505 However, charge transfer from the PPD 508 is stopped by the second control circuit 507 to record the TOF value and the corresponding distance to the 3D object 26. In other words, the spatial-temporal correlation between the outputs of at least two adjacent SPADs 503 is used to control the operation of the PPD 508. For pixel 43, the light sensing function is performed by SPADs 503, and PPD 508 is used as a TCC instead of a light sensing element. Reflected photons (of returned light pulses 37) are correlated with the transmitted pulse 28 (compared to unrelated ambient photons), and control of the transfer of charge from the PPD 508 is predetermined. Based on the trigger of two or more adjacent SPADs within a time interval, rejecting ambient photons provides improved performance of the image sensor unit 24 in strong ambient light conditions, thus substantially preventing distance measurement errors. .

6A-6C each show three different examples of pixel array structures in accordance with the subject matter described herein. Any of the pixel array structures shown in FIGS. 6A-6C can be used to implement the pixel array 42 of FIG. 2. In FIG. 6A, an exemplary 2x2 pixel array structure 600A is shown, wherein each pixel 601-604 (which may represent pixel 43 of FIG. 5 in some embodiments) is shown in 2x2 pixel array structure 600A. It includes one pixel-specific PPD core and four pixel-specific SPAD cores. For simplicity, PPD and SPAD cores are identified only for pixel 601, PPD cores are indicated by reference numeral '605', and SPAD cores are indicated by reference numerals '606-609'.

The structure 600A shown in FIG. 6A may be considered a low (spatial) resolution structure due to the physical space occupied by each pixel of a semiconductor die of a given size. As a result, a relatively small number of pixels can be formed in the pixel array of the die, compared to the example structure 600B of FIG. 6B which provides a higher resolution 3x3 pixel array structure. In the higher resolution structure 600B of FIG. 6B, one SPAD core is shared by four (2 × 2) adjacent PPD cores. For example, in FIG. 6B, the SPAD core 625 is shared by PPD cores of adjacent pixels 621-624 (in some embodiments, each of which may represent pixel 43 of FIG. 5). It is shown to be. For simplicity, other components of the pixel array structure 600B of FIG. 6B are not identified by reference numerals. The configuration of pixel array structure 600B of FIG. 6B in which one SPAD is shared between four adjacent pixels provides an efficient ratio of 1: 1 between the PPDs in the pixel and the SPADs associated with the pixel.

This sharing can be extended to 3x3 shares or more, as shown by pixel array structure 600C in FIG. 6C. As each SPAD is shared by neighboring pixels of the die, more pixels can be formed in the pixel array, so the SPAD sharing configuration shown in FIG. 6B provides a high (spatial) resolution structure for the pixel array, thus More space can be made to accommodate more pixels. In addition, since the pixels of the pixel array structure 600B of FIG. 6B have a single PPD core associated with four SPAD cores in a 2 × 2 configuration, up to four matching photons are detected by each pixel (ie, one per SPAD). Photons).

6A and 6B show exemplary pixel array structures in which PPDs and SPADs can be realized in a single die. That is, the SPADs and PPDs may be at the same level in the die. In contrast, FIG. 6C shows an exemplary 4x4 pixel array structure 600c that may be realized in dies in which pixels are stacked. For example, SPAD cores can be realized on the upper die, and PPD cores (and read circuits) can be realized on the lower die. Thus, the PPDs and SPADs can be in two different dies that can be stacked and the circuit elements of these dies (PPDs, SPADs, transistors, etc.) are electrically connected by wires or metal bumps. Can be connected. Like the structure 600B of FIG. 6B, the pixel array structure 600C of FIG. 6C may also provide a high resolution structure in which a single SPAD core is shared by nine (3 × 3) adjacent PPD cores. Likewise, as shown in FIG. 6C, a single PPD core, such as PPD core 641, may be associated with nine SPAD cores, such as SPAD cores 642-650, to form a single pixel. SPAD cores 642-650 can also be shared by other pixels. For simplicity, other pixels, their PPD cores, and associated SPAD cores are not identified by reference numerals in FIG. 6C. Additionally, since the pixels of the pixel array structure 600C of FIG. 6C have a single PPD core associated with nine SPAD cores in a 3x3 configuration, up to nine matching photons are detected by each pixel (ie, one per SPAD). Photons).

7 shows circuit details of an example embodiment of a pixel 700 according to the subject matter described herein. The pixel 700 shown in FIG. 7 may be an example of the more comprehensive pixel 43 shown in FIGS. 2 and 5. An electronic shutter signal 701 is provided to each pixel (as discussed in more detail below with reference to the timing diagrams of FIGS. 8, 9 and 14), so that pixel 700 is returned to the returned light pulse 37. Enable pixel-specific photoelectrons to be captured in a time-related manner. More generally, pixel 700 may be considered to have a charge transfer trigger portion, a charge generation and transfer portion, and a charge collection and output portion. The charge transfer trigger portion may include SPAD cores 501 and a logic unit 702. The charge generation and transfer portion includes a PPD 508, a first N-channel metal oxide semiconductor field effect transistor (NMOSFET) or an NMOS transistor 703, a second NMOS transistor 704, and a third NMOS transistor 705. can do. The charge collection and output portion may include a third NMOS transistor 705, a fourth NMOS transistor 706, and a fifth NMOS transistor 707. In some embodiments, the PPD core of pixel 700 of FIG. 7 and the PPD core of pixel 900 of FIG. 13 may be P-channel metal oxide semiconductor field effect transistors (PMOSFETs) or PMOS transistors, or other types of transistors, or It can be formed as a charge transfer device. In addition, each portion of pixel 700 described herein is for purposes of illustration and discussion only. In some embodiments, portions can include more, fewer, and / or other circuit elements than those described herein.

PPD 508 may store charge similar to a capacitor. In one embodiment, the PPD 508 is covered and thus may not respond to light. Thus, PPD 508 can be used as a TCC instead of a light sensing element. However, as mentioned above, the light sensing function may be accomplished by the SPADs in the SPAD cores 501. In some embodiments, a photogate or other semiconductor device (with appropriate modifications) may be used instead of the PPD of the pixel configurations of FIGS. 7 and 13.

The charge transfer trigger portion may generate a transfer active (TXEN) signal 708 under the control of the electronic shutter signal 701 to trigger the transfer of charge stored in the PPD 508. The SPAD detects photons of light pulses reflected and transmitted from an object such as the object 26 of FIG. It may be latched under the control of the operation of the shutter signal 701 for subsequent processing by 702. Logic unit 702 processes all digital SPAD outputs 506 when, for example, outputs 506 are received from at least two adjacent SPADs within a predetermined time interval while shutter signal 701 is active. Logic circuits to generate the TXEN signal 708.

In the charge generation and transfer portion, the PPD 508 may initially be set to its full well capacity using the reset signal RST together with the third transistor 705. The first transistor 703 can receive the transfer voltage (VTX) signal 710 at the drain terminal of the first transistor 703 and the TXEN signal 708 at the gate terminal. The TX signal 711 is available at the source terminal of the first transistor 703 and can be applied to the gate terminal of the second transistor 704. As shown, the source terminal of the first transistor 703 may be connected to the gate terminal of the second transistor 704. The VTX signal 710 (or, equivalently, the TX signal 711) is used as an amplitude modulated signal to control the charge to be transferred from the PPD 508, which may be connected to the source terminal of the second transistor 704. Can be. The second transistor 704 transfers the charge of the PPD 508 from the source terminal of the transistor 704 to the drain terminal of the transistor 704, which terminal may be connected to the gate terminal of the fourth transistor 706 and A charge “collection site” referred to herein as a floating diffusion node / junction 712 may be formed. In some embodiments, the charge transmitted from PPD 508 may depend on the modulation provided by amplitude modulated signal 710 (or, equivalently, TX signal 711). In the embodiments of FIGS. 7 and 13, the transmitted charges are electrons. However, the subject matter described herein is not limited to this disclosure, and PPDs having different designs may be used as the charges to be transmitted may be holes.

In the charge collection and output portion, the third transistor 705 receives the RST signal 709 at the gate terminal of the third transistor 705, and the pixel voltage VPIX signal at the drain terminal of the third transistor 705. 713 may be received. The source terminal of the third transistor 705 may be connected to the floating diffusion node / junction 712. In one embodiment, the voltage level of the VPIX signal 713 may be equal to the voltage level of the collective power supply voltage VDD, and may be in the range of 2.5V to 3.0V. The drain terminal of the fourth transistor 706 may also receive the VPIX signal 713. In some embodiments, the fourth transistor 706 acts as an NMOS source follower and can function as a buffer amplifier. The source terminal of the fourth transistor 706 may be connected to the drain terminal of the fifth transistor 707, and the corresponding drain terminal may be a cascode with the source follower 706 and the cas terminal of the fifth transistor 707. A select (SEL) signal 714 may be received at the gate terminal. The charge transmitted from the PPD 508 and collected at the floating diffusion node / junction 712 may appear as a pixel-specific output (PIXOUT) data line at the source terminal of the fifth transistor 707.

The charge transmitted from the PPD 508 to the FD node / junction 712 is controlled by the VTX signal 710 (and the TX signal 711). The amount of charge that reaches the floating diffusion node / junction 712 is modulated by the TX signal 711. In one embodiment, the transfer voltage (VTX) signal 710 (and TX 711) may be ramped to gradually transfer charge from the PPD 508 to the floating diffusion node / junction 712. Thus, the amount of charge transferred is a function of the amplitude modulated signal 711 (TX), and the ramping of the TX signal 711 is a function of time. Thus, the amount of charge transferred from the PPD 508 to the floating diffusion node / junction 712 is also a function of time. During the transfer of charge from the PPD to the floating diffusion node / junction 712, the TXEN signal by the SPAD cores 501 as photo-detection events occur for at least two adjacent SPADs in the SPAD cores 501. When the second transistor 704 is turned off due to the generation of 708, the transfer of charge from the PPD 508 to the floating diffusion node / junction 712 stops. As a result, both the amount of charge transferred to floating diffusion node / junction 712 and the amount of charge remaining in PPD 508 are a function of the TOF of the incident photon (s). The result is a time-to-charge transformation, and a single end-differential signal transformation. Accordingly, PPD 508 operates as a time-to-charge converter (TCC). The more charge is transferred to the floating diffusion node / junction 712, the more the voltage at the floating diffusion node / junction 712 decreases and the more the voltage at the PPD 508 increases.

The voltage of the floating diffusion node / junction 712 is then transmitted as a PIXOUT signal through the fifth transistor 707 to an analog-to-digital converter (ADC) unit (not shown) and with an appropriate digital signal / value for further processing. Can be converted. More details of the timing and operation of the various signals of FIG. 7 are provided with reference to the discussion of FIG. 9. In the embodiment of FIG. 7, after the charge is fully transferred to the floating diffusion node / junction 712 that converts the charge of the PPD 508 into a voltage, the fifth transistor 707 selects the SEL for selecting the pixel 700. The signal 714 can be received to read the charge of the floating diffusion node / junction 714 into the PIXOUT1 (or pixel output 1) voltage and the charge remaining in the PPD 508 to the PIXOUT2 (or pixel output 2) voltage. The pixel output data line 510 sequentially outputs the PIXOUT1 and PIXOUT2 signals sequentially as will be discussed later with reference to FIG. 10. In another embodiment, not one of the PIXOUT1 signal or the PIXOUT2 signal may be read.

FIG. 8 is an exemplary timing diagram 800 that provides an overview of the modulated charge transfer mechanism in pixel 700 of FIG. 7 in accordance with the subject matter described herein. The waveforms shown in FIG. 8 (as well as FIGS. 9 and 14) are in fact simplified and for illustrative purposes only. Actual waveforms may vary in shape as well as timing depending on circuit implementations. Common signals in FIGS. 7 and 8 are identified using the same reference numerals and include a VPIX signal 713, an RST signal 709, an electronic shutter signal 701, and an amplitude modulated signal 710. In FIG. 8 two additional waveforms 801, 802 are also shown, showing the state of charge in the PPD 508 and the floating diffusion node / junction 712 when the amplitude modulation signal 710 is applied during charge transfer. Show the state of charge, respectively. In the embodiment of FIG. 8, the VPIX signal 713 starts the pixel 700 starting with a low logic voltage (eg, logic 0 or 0V) during operation of the pixel 700 and then starts a high logic voltage (eg, For example, to logic 1 or 3V). The reset (RST) signal 709 starts with a high logic voltage pulse (eg, a pulse from logic 0 to logic 1 and back to logic 0) during the initialization of the pixel 700, thereby recovering the charge in the PPD 508. Set to total capacity and set charge of floating diffusion node / junction 712 to zero Coulombs. The reset voltage level for floating diffusion node / junction 712 may be a logic one level. During a distance (TOF) measurement operation, the more electrons the floating diffusion node / junction 712 receives from the PPD 508, the lower the voltage of the floating diffusion node / junction 712. The electronic shutter signal 701 starts with a low logic voltage (e.g., logic 0 or 0V) at the beginning and during the pixel 700 and at a logic 1 level at a time corresponding to the minimum measurement distance during operation of the pixel 700 Change to (e.g., 3V), causing the SPADs 503 of the SPAD cores 501 to detect the photon (s) of the returned light pulse 37, and then the time corresponding to the maximum measurement distance To a logic zero level (e.g. 0V). Thus, the logical one level interval of the shutter signal 701 provides a predefined time interval / window such that the outputs received from adjacent SPADs during that time have spatial and temporal correlation. The charge in PPD 508 starts fully charged during initialization, and decreases as the VTX signal 710 slopes (preferably in a linear fashion) from 0V to a higher voltage. The PPD charge level under control of the amplitude modulated signal (VTX) 710 is represented by a waveform with reference numeral 801 in FIG. 8. The PPD charge reduction may be a function of how long the VTX signal slopes, which causes the transfer of a certain amount of charge from the PPD 508 to the floating diffusion node / junction 712. Thus, as indicated by the waveform with reference numeral 801 in FIG. 8, the charge of the floating diffusion node / junction 712 starts at a low charge (eg, 0C) and the VTX signal 710 is further from 0V. Incrementally ramps to a high level and partially transfers a certain amount of charge from the PPD 508 to the floating diffusion node / junction 712. Charge transfer is a function of how long the VTX signal 710 slopes.

As mentioned above, the pixel-specific output of data line 510 (PIXOUT) is derived from the PPD charge transferred to floating diffusion node / junction 712. Accordingly, PIXOUT signal 510 may be considered to be amplitude modulated over time by amplitude modulated signal 710 (VTX) (or, equivalently, TX signal 711). In this way, TOF information is provided via amplitude modulation of pixel specific output 715 using amplitude modulation signal 710 (VTX) (or, equivalently, TX signal 711). In some embodiments, the modulation function for generating the VTX signal 710 can be monotonic. In the exemplary embodiments shown in FIGS. 8, 9 and 14, amplitude modulated signals are generated using a ramp function, so they are shown to have gradient-type waveforms. However, in other embodiments, other types of analog waveforms / functions may be used as the modulation signals.

In one embodiment, the ratio of one pixel output (eg, PIXOUT1) to the sum of two pixel outputs (here, PIXOUT1 + PIXOUT2) is shown, for example, in FIG. 9 and described in more detail below. As can be seen, it can be proportional to the time difference between T _tof and T _dly . For example, in the example of pixel 700, parameter T _tof may be a pixel-specific TOF value of an optical signal received by two or more SPADs in SPAD cores 501, and may include a delay time parameter ( T _dly ) may be the time from when the optical signal 28 is initially transmitted until the VTX signal 710 starts to slope. Typically, if the optical pulse 28 is transmitted after the VTX signal 710, which occurs when the electronic shutter 701 is opened, begins to slope, the delay time T _dly may be negative. The proportional relationship may be expressed by Equation 2.

However, the subject matter described herein is not limited to the relationship in equation (2). As discussed below, the ratio of Equation 2 can be used to calculate the depth or distance of the object and is less susceptible to pixel-to-pixel variation unless the sum of PIXOUT1 and PIXOUT2 is always the same.

For convenience, as used herein, the term 'P1' may be used to refer to 'PIXOUT1' and the term 'P2' may be used to refer to 'PIXOUT2'. It may be the relationship of Equation 2 that the pixel-specific TOF value may be determined as the ratio of pixel specific output values P1 and P2. In some embodiments, if the pixel-specific TOF value is so determined, the pixel-specific distance (D) or range (R) or over the object (such as object 26 in FIG. 2). The specific position of may be given by Equation 3.

Where c is the speed of light. Alternatively, in some embodiments in which a modulated signal, such as the VTX signal 710 (or TX signal 711) of FIG. 7 is linear, for example within a shutter window, the range / distance may be calculated by equation (4).

Thus, a 3D image of an object, such as object 26, may be generated by TOF system 15, based on pixel-specific range values determined as given above.

Amplitude modulation based manipulation or control of the PPD charge dispersion within the pixels also enables control of distance measurement and resolution. The pixel level amplitude modulation of the PPD charge may be a rolling shutter, for example in a complementary metal oxide semiconductor (CMOS) image sensor, or an electronic shutter, which may be a global shutter, for example in a charge coupled device (CCD) image sensor. Can work with The description herein is in the context of a single pulse TOF imaging system such as system 15 of FIGS. 1 and 2. Although primarily provided, the principles of the pixel-level internal amplitude modulation method discussed herein are, through appropriate modifications (if necessary), a continuous wave modulation TOF imaging system or non-TOF with pixels 43 (FIG. 5). It can also be implemented in the system.

9 shows FIGS. 1 and 2 when pixel 700 of the embodiment of FIG. 7 is used in a pixel array such as pixel array 42 of FIGS. 2 and 12 to measure TOF values according to the subject matter described herein. A timing diagram 900 of exemplary timing of different signals in system 15 of FIG. Various signals, such as transmitted pulse 28, VPIX signal 713, TXEN signal 708, and the like, shown in the embodiments of FIGS. 2 and 7, are identified in FIG. 9 using the same reference numerals. Prior to the discussion of FIG. 9, in the context of FIG. 9 (and also in the case of FIG. 14), the parameter T _dly is denoted by reference numeral 901, the rising edge of the irradiated pulse 28 and the VTX signal 710. ) Indicates a time delay between the times when it starts to slope; The parameter T _tof indicates a pixel-specific TOF value measured with a delay between the rising edges of the irradiated pulse 28 and the received pulse 37, as indicated by reference numeral 902; And the parameter T _sh is indicated by reference numeral 903 and given by the activation (e.g., logic 1 or on) and inactivation (e.g., logic 0 or off) of the shutter signal 701, Note the time interval between opening and closing of the electronic shutter. Accordingly, the electronic shutter 701 is considered to be active during the period T _sh , which is also identified by reference numeral 904. In some embodiments, the delay T _dly may be predetermined and fixed regardless of operating conditions. In other embodiments, the delay T _dly may be adjustable at runtime, for example, depending on external weather conditions. It should be noted here that high or low signal levels are associated with the design of pixel 700. The signal polarities or bias levels shown in FIG. 9 may differ in other types of pixel designs, for example based on the types of transistors or other circuit components used.

As mentioned above, the waveforms of FIG. 9 (also of FIG. 14) are in fact simplified and for illustrative purposes only, and the actual waveforms may vary in timing as well as shape depending on the circuit implementation. As shown in FIG. 9, the returned pulse 37 may be a time delayed version of the irradiated pulse 28. In some embodiments, the irradiated pulse 28 may be a very short interval, such as in the range of about 5 ns to about 10 ns, for example. The returned pulse 37 can be sensed using two or more SPADs of the pixel 700. The electronic shutter signal 701 may cause the SPADs to capture pixel-specific photon (s) in the received light 37. The electronic shutter signal 701 may have a gate delay to avoid light scattering reaching the pixel array 42. Light scattering of the irradiated pulses 28 may occur due to bad weather, for example.

Various external signals (eg, VPIX signal 713, RST signal 709, etc.) and internal signals (eg, TX signal 711, TXEN signal 708, and floating diffusion node / junction ( In addition to the voltage of 712, the timing diagram 900 of FIG. 9 also identifies the following events or time intervals. The following events or time intervals include (i) a PPD preset event 905 when the RST, VTX, TXEN, and TX signals are high and the VPIX and shutter signals 713, 701 are low; (ii) a first floating diffusion reset event 906 from when the TX signal is low until the RST signal returns from high to low; (iii) delay time T _dly 901; (iv) flight time T _tof 902; (v) electronic shutter on or activation interval T _sh 903; And (vi) a second FD reset event 907 for the interval when the RST signal 709 is logic 1 for a second time. 9 also shows when the electronic shutter is initially closed or off (indicated by reference number 908), when the electronic shutter is initially open or on (indicated by reference number 904), and initially a floating diffusion node. When the charge transferred to junction 712 is read out via PIXOUT data line 510 (indicated by reference number 909), and when the floating diffusion node / junction 712 voltage is reset second (again) at 907. And when the charge remaining in PPD 508 is transferred to floating diffusion node / junction 712 and read back in event 910 (eg, as output to PIXOUT 510). In an embodiment, the shutter-on period T _sh may be less than or equal to the inclination time of the VTX signal 710.

Referring to FIG. 9, in the example of pixel 700 of FIG. 7, PPD 508 may be filled up to its total capacity with charge in an initialization step (eg, PPD preset event 905). During the PPD preset time 905, as shown, the RST signal 709, the VTX signal 710, the TXEN signal 708, and the TX signal 711 are high, and the VPIX signal 713 and shutter signal ( 710 may be low. Thereafter, the VTX signal 710 (and the TX signal 711) goes low to shut off the second transistor 704 and the VPIX signal 713 goes high to charge from the fully charged PPD 508. You can start the transfer. In some embodiments, all the pixels of a row of pixels of pixel array 42 may be selected together at the same time, and the PPDs of all the pixels of the selected row may be reset together using RST signal 709. Each pixel of the selected row of pixels is read separately and the analog based PIXOUT signal can be converted to a digital value by a corresponding column ADC unit (not shown). In one embodiment, the RST lines may remain high or on for unselected rows to prevent blooming.

In the embodiment shown in FIG. 9, all signals except the TXEN signal 708 start at a logic zero or low level as shown. Initially, the PPD 508 when the RST signal 709, the VTX signal 710, the TXEN signal 708, and the TX signal 711 are at a logic one level and the VPIX signal 713 remains low. Is preset. Thereafter, when the VTX signal 710 and the TX signal 711 become logic 0 and the VPIX signal 713 goes high (or logic 1), the floating diffusion node / degree while the RST signal 709 is logic 1 In the embodiment shown in 9, all signals except the TXEN signal 708 start at a logic zero or low level as shown. Initially, the PPD 508 when the RST signal 709, the VTX signal 710, the TXEN signal 708, and the TX signal 711 are at a logic one level and the VPIX signal 713 remains low. Is preset. Thereafter, when the VTX signal 710 and the TX signal 711 become logic 0 and the VPIX signal 713 goes high (or logic 1), the floating diffusion node / junction while the RST signal 709 is logic 1 712 is reset. For convenience, the same reference numeral 712 is used to indicate the floating diffusion node / junction of FIG. 7 and the associated voltage waveform of the timing diagram of FIG. 9. After the floating diffusion node / junction 712 is reset to high (eg, 0C in the charge domain), the VTX signal 710 is tilted while the TXEN signal 708 is logic one. The flight time T _tof interval 901 is from when the pulsed light 28 is transmitted until the returned light 37 is received and also charges from the PPD 508 to the floating diffusion node / junction 712. Partly during the time of transmission. The VTX signal 710 (and the TX signal 711 is tilted while the shutter 701 is on or open. This causes a significant amount of charge in the PPD 508 to be transferred to the floating diffusion node / junction 712, where the VTX is It can be a function of how long it slopes in. The generated SPAD output when the transmitted pulse 28 is reflected at the object 26 and received by at least two SPADs in the SPAD cores 501 of the pixel 700. 506 are processed by logic unit 702, resulting in TXEN signal 708 being a fixed logic 0. Thus, the returned light by at least two adjacent SPADs in a temporally correlated manner, The detection of 37 (ie, when the shutter is on or activated) is indicated by a logical zero level for the TXEN signal 708. The logic low level of the TXEN signal 708 is represented by the first transistor 703 and the second. Turn off transistor 704, which is floating diffusion node / junction 712 from PPD 508. Stops the transfer of charge to the electric charge .. When the electronic shutter signal 701 becomes logic 0 and the SEL signal 714 (not shown in Figure 9) becomes logic 1, the charge of the floating diffusion node / junction 712 is Output is a voltage PIXOUT1 at PIXOUT line 510. Thereafter, floating diffusion node / junction 712 may be reset (as indicated by reference numeral 907) back to RST pulse 709, which is logic high. Thereafter, when the TXEN signal 708 becomes logic 1, the charge remaining in the PPD 508 is substantially completely transferred to the floating diffusion node / junction 712 and output from the PIXOUT line 510 to the voltage PIXOUT2. As mentioned above, the PIXOUT1 and PIXOUT2 signals may be converted into corresponding digital values P1, P2 by an appropriate ADC unit (not shown) In certain embodiments, these P1 and P2 values are Used in equations 3 or 4, to pick between pixel 700 and object 26 You can determine a specific distance / range.

In one embodiment, logic unit 702 generates logic based on the G () function (shown and discussed with reference to FIG. 10), including logic circuits (not shown), and then plots the output. The final TXEN signal 708 can be obtained by logically ORing the internally generated signal, such as a signal similar to the TXRMD signal 1401 shown in FIG. 14. This internally generated signal remains low while the electronic shutter is on, but the TXEN signal 708 becomes logic 1 to facilitate the transfer of charge remaining in the PPD 508 (event 910 in FIG. 9). Can be activated high. In some embodiments, the TXRMD signal or similar signal can be supplied externally.

FIG. 10 illustrates that a logic unit, such as logic unit 702 (FIG. 7) or logic unit 1319 (FIG. 13), may be pixels 700 (FIG. 7) or pixels 1300 (FIG. 13), according to the subject matter described herein. Shows how it is implemented in such a pixel. FIG. 10 shows a pixel 1000 (pixels 700 or 1300) with a PPD core 1001 associated with four SPAD cores 1002-1005, in a 2 × 2 structural configuration similar to that shown in FIG. 6A or 6B. A very simplified diagram of which) is represented. The availability of four SPADs enables detection of up to four simultaneous photon (s) correlated temporally and spatially. In some embodiments, the logic unit (not shown) of the pixel 1000 is configured with logic circuits (not shown) that implement the functions F (x, y) and G (a, b, c) shown in FIG. 10. ) May be included. Blocks 1006-1009 of FIG. 10 show the inputs and outputs of the logic circuits that implement the function F (x, y). Thus, blocks 1006-1009 represent such logic circuits and may be considered to collectively form part of the logic unit of pixel 1000. For ease of discussion, these blocks may be referred to as F (x, y) blocks. Although blocks 1006-1009 are shown outside of the PPD core 1001 for ease of use, logic circuits that implement the functions of blocks 1006-1009 may be implemented in a logic unit (not shown) within the PPD core 1001. It should be understood that it may be part of it.

As shown, each F (x, y) block 1006-1009 receives one input from each of two inputs (x, y), namely two associated SPAD cores. In the context of FIGS. 5 and 7, these inputs may be in the form of output signals 506 from SPAD cores 501. In the context of FIG. 13, the SPAD outputs 1310, 1318 may represent the inputs (x, y) needed for the F (x, y) blocks in the logic unit 1319. Two similar input F (x, y) blocks per pair of SPAD cores may be provided for pixels with more than four SPAD cores associated with the PPD core, such as, for example, the pixel array configuration 600c of FIG. 6C. . In some embodiments, all of the F (x, y) blocks 1006-1009 are configured to operate on different pairs of SPAD outputs (as their x and y inputs) to separate F (x, y) It may be integrated and implemented through a single F (x, y) unit in the PPD core 1001 that includes logic circuits that implement the functionality of blocks 1006-1009. As mentioned above, the TOF measurement described herein may be based on the detection of spatial and temporally correlated photons by at least two SPADs in a pixel. Thus, as mentioned in FIG. 10, each F (x, y) block 1006-1009 (more specifically, a logic circuit within the F (x, y) block) is adapted to perform the following predefined operations. Can be configured. Predefined operations include (i) a logical negative AND operation (given by (x * y)) for respective inputs (x, y) for detecting two or four simultaneous photons; (ii) a NOR operation (given by (x + y)) for each of the inputs (x, y) to detect three simultaneous photons. Thus, when signals 506 (FIG. 5) from SPAD cores 1002-1005 indicate that two (or four) SPADs have detected photons during the shutter on period, F (x, y) The logic circuit implementing the blocks 1006 to 1009 may perform a logical NAND operation. Similarly, a logical NOR operation may be selected when signals 506 from SPAD cores 1002-1005 indicate that three SPADs detected photons during the shutter on period. In the example diagram of FIG. 10, the three pulses 1010-1012 are three when each of the three SPAD cores 1003-1005 detects incident light, such as the returned pulse 37 (FIG. 2). It is shown to represent the detection of simultaneous photons.

Referring back to FIG. 10, the output of each F (x, y) block 1006-1009 is indicated using the corresponding reference characters a, b, c, d. The logic unit (not shown) in the PPD core 1001 may also include additional logic circuitry (not shown) that receives and processes the outputs a through d. The logic circuitry receives all four of these outputs as inputs and can operate on them according to a predefined logic function G (a, b, c, d). For example, as shown in FIG. 10, in the case of the detection of two simultaneous photons, the G () function is a logical NAND operation ((a * b * c) for all four inputs a to d. given by * d). On the other hand, in the case of detection of three or four simultaneous photons, the G () function performs a logical NOR operation (given by (a + b + c + d)) on all four inputs (a to d). Can be done. In one embodiment, the TXEN signal, such as the TXEN signal 708 of FIG. 7 or the TXEN signal 1325 of FIG. 13, may be an output of a logic circuit that implements a G () function. In another embodiment, the output of the logic circuit for the G () function may be ORed with an internally generated signal, such as the TXRMD signal 1401 of FIG. 14, to obtain a final TXEN signal.

FIG. 11 shows an exemplary flow chart 1100 showing how the TOF value is determined in the system 15 of FIGS. 1 and 2 in accordance with the subject matter described herein. The various steps shown in FIG. 11 may be performed by a single module or combination of modules or system components of system 15. In the discussion herein, by means of example only, certain tasks are described as being performed by specific modules or system components. Other modules or system components may also be appropriately configured to perform these tasks. As mentioned in operation 1101, the system 15 (more specifically, the projector module 22) may irradiate a laser pulse, such as the pulse 28 of FIG. 2, to an object, such as the object 26 of FIG. 2. Can be. In operation 1102, the processor 19 (or the pixel processing unit 46 in certain embodiments) transmits an amplitude modulated signal, such as the VTX signal 710 of FIG. 7, such as a PPD 508 within the pixel 700 of FIG. 7. The PPD in the pixel can be examined. Pixel 700 may be any pixel 43 of pixel array 42 of FIG. 2. In operation 1103, pixel processing unit 46 may initiate the transfer of a portion of the charge stored in PPD 508 based on the modulation received from amplitude modulated signal 710. To initiate this charge transfer, the pixel processing circuit 46 may add various external signals such as the electronic shutter signal 701, the VPIX signal 713, and the RST signal 709 to the logic shown in the exemplary timing diagram of FIG. 9. May be provided to pixel 700 at levels. In operation 1104, a returned pulse, such as the returned pulse 37, may be detected using a plurality of SPADs within the pixel 700. As mentioned above, the returned pulse 37 may be an irradiated pulse 28 reflected from the object 26, with each SPAD in the pixel 700 (in SPAD cores 501) returned a pulse. The brightness received from can be converted into a corresponding (SPAD-specific) electrical signal.

For each SPAD receiving the brightness, the first control circuit 504 in the SPAD cores 501 in the pixel 700 can process the corresponding (SPAD-specific) electrical signal to produce a SPAD specific digital output. (Operation 1105). All of these SPAD-specific digital outputs are collectively represented by arrows with reference numeral 506 of FIGS. 5 and 7. As mentioned with reference to the discussion of FIG. 9, the logic unit 702 can process the outputs 506, and as long as the outputs are correlated temporally and spatially, the TXEN signal 708 is logical 0 (low). ) State. The logic zero level of the TXEN signal 708 turns off the first transistor 703 and the second transistor 704 in the pixel 700, which charges from the PPD 508 to the floating diffusion node / junction 712. Stops transmission. Thus, in operation 1106, as at least two SPAD-specific digital outputs are generated during a predetermined time interval, such as in the shutter of section 904 of FIG. The transmission may be terminated (in operation 1103).

As discussed above with reference to FIG. 7, a portion of the charge transferred to the floating diffusion node / junction 712 is read into the PIXOUT1 signal, and converted into an appropriate digital value P1, which is then followed. It can be used in conjunction with the digital value P2 (for PIXOUT2 signal) generated to obtain TOF information from the ratio of P1 / (P1 + P1). Thus, at operation 1107, upon termination (at operation 1106), based on the portion of the analog charge transmitted, pixel processing unit 46 or processor 19 in system 15 may return the TOF of the returned pulse 37. The value can be determined.

12 is an exemplary layout of a portion of an image sensor unit 1200 in accordance with the subject matter described herein. The image sensor unit 1200 may correspond to the image sensor unit 24 illustrated in FIGS. 1 and 2. A portion of the image sensor unit 1200 shown in FIG. 12 captures the returned light, P1 and P2 for subsequent calculations of TOF values (from Equation 2) and for the generation of a 3D image of the object 26 if desired. Associated with providing the signals needed to generate the values. As in the case of FIG. 2, the pixel array 1201 in the image sensor unit 1200 of FIG. 12 is shown as having nine pixels arranged in 3 × 3 for convenience. Indeed, a pixel array may contain hundreds of thousands or millions of pixels in multiple rows and columns. In some embodiments, each pixel of pixel array 1201 may have the same configuration, so each pixel is identified using the same reference numeral 1202 as shown in FIG. 12. In the embodiment of FIG. 12, the 2D pixel array 1201 may be a complementary metal oxide semiconductor (CMOS) array in which each pixel 1202 may be the pixel 1300 illustrated in FIG. 13. Although the example layout of FIG. 12 relates to the pixel configuration of FIG. 13, it should be understood that the image sensor unit 1200 of FIG. 12 may be modified as appropriate when each pixel 1202 has the configuration shown in FIG. 7. . In some embodiments, pixels 1202 may have other configurations than those shown in FIGS. 7 and 13, and the auxiliary processing units of FIG. 12, such as row decoder / driver 1203, column decoder 1204, and the like, may be used. It may be modified as appropriate to operate at the desired pixel configuration.

In addition to the pixel array 1201, in the embodiment shown in FIG. 12, the image sensor unit 1200 also includes a row decoder / driver 1203, a column decoder 1204, and column-specific analogs used during 2D and 3D imaging. A pixel column unit 1205 including circuits for correlated double sampling (CDS) as well as digital converters (ADCs). In one embodiment, there may be one ADC per column of pixels. In some embodiments, the processing units 1203, 1204, 1205 may be part of the pixel processing unit 46 shown in FIG. 2. In the embodiment of FIG. 12, the row decoder / driver 1203 provides six different signals as inputs to each pixel 1202 of a row of pixels to control the operation of the pixels of the pixel array 1201. It is shown to enable the generation of certain PIXOUT signals 1206-1208. Each of the arrows 1209-1211 of FIG. 12 shows a row-specific set of these signals applied as inputs to each pixel 43 of the corresponding row. These signals include a reset (RST) signal, a second transfer (TX2) signal, an electronic shutter (SH) signal, a transfer voltage (VTX) signal, a pixel voltage (VPIX) signal, and a row select (SEL) signal. Figure 13 shows how these signals are applied to the pixel. 14 shows an example timing diagram that includes many of these signals.

In one embodiment, the row select (SEL) signal may be activated to select the appropriate row of pixels. The row decoder / driver 1203 may receive the address or control information for the row to be selected, for example, from the processor 19 via the row address / control inputs 1212. The row decoder / driver 1203 decodes the received inputs 1212 to cause the row decoder / driver 1203 to select the appropriate row using the SEL signal, and also to select the corresponding RST, VTX and other signals. Provide to the decoded row. When applied as pixel inputs, a more detailed description of these signals is provided later with reference to the discussion of FIGS. 13 and 14. In some embodiments, the row decoder / driver 1203 also receives control signals (not shown), for example, from the processor 19, such that the SEL, RST, VTX, SH indicated by arrows 1209-1211. And row decoder / driver 1203 to apply appropriate voltage levels for various other signals.

The pixel column unit 1205 can receive PIXOUT signals 1206-1208 from the pixels in the selected row and process them to generate pixel-specific signal values from which TOF measurements can be obtained. The signal values may be the P1 and P2 values described above, as indicated by arrow 1213 in FIG. 12. Each column-specific ADC unit may process the received inputs (PIXOUT signals) to generate corresponding digital data outputs (P1 / P2 values). Further details of the CDS and ADC operations provided by the CDS and ADC circuits (not shown) in the pixel column unit 1205 are provided below with reference to FIG. 14. In the embodiment shown in FIG. 12, column decoder 1204 is shown coupled with pixel column unit 1205. The column decoder 1204 may receive, for example, from the processor 19 a column address / control input 1214 for the columns to be selected in association with a given row select (SEL) signal. Column selection is sequential, thus enabling sequential reception of pixel output from each pixel of the row selected by the corresponding SEL signal. The processor 19 may select the row of pixels by providing the appropriate row address inputs, and also provide the appropriate column address inputs to the column decoder 1204 so that the pixel column unit 1205 outputs from the individual pixels of the selected row. (PIXOUTs) can be received.

13 shows another exemplary embodiment of a pixel 1300 according to the subject matter described herein. Pixel 1300 of FIG. 13 is another example of the more comprehensive pixel 43 shown in FIG. The pixel 1300 may include a number of SPAD cores (ie, SPAD cores 1 to SPAD core N) (N is two or more) as part of the SPAD cores. In FIG. 13, two such SPAD cores 1301A and 1301N are shown with some circuit details. It should be noted that in some embodiments, similar circuits may be employed for the SPAD cores in pixel 700 of FIG. 7. The SPAD core 1300A may include a SPAD 1302 that receives the SPAD operating voltage (VSPAD) 1303 through the resistive element 1304 (such as a resistor). However, the configuration of the SPAD is not limited to that shown in FIG. In one embodiment, the locations of register 1304 and SPAD 1302 may be exchanged. In pixel 1301A, SPAD 1302 responds to light. When the SPAD 1302 receives a photon, the SPAD 1302 may output a pulse that goes from the level of VSPAD to 0V and restores to VSPAD. The output from the SPAD 1302 can be filtered through the capacitor 1305 and applied to the inverter 1306 (which can function as a combination of buffer and latch). In one embodiment, the capacitor 1305 may be omitted. The SPAD core 1301A may include a PMOS transistor 1307 that receives an electronic shutter signal 1308 at the gate terminal, where the drain terminal of the transistor 1307 is connected to a capacitor (and an input of the inverter 1306) and In addition, the source terminal of the transistor 1307 may receive the supply voltage 1309 (VDD) (or the VPIX voltage in some embodiments). When the electronic shutter signal 1308 is turned off (eg, logic zero or low level), the transistor 1307 is conducting and the output 1310 of the inverter 1306 is any received from the SPAD 1301A. Regardless of the state of the outputs, it may remain at a fixed voltage level (eg, a logic low or logic 0 state). Only when the electronic shutter signal 1308 is turned on or active, the output from the SPAD 1301A can be applied to the PPD core 1311. When the shutter is active (eg, logic one level), transistor 1307 is turned off, the SPAD-generating output is sent to inverter 1306 (via coupling capacitor 1305), and the output line It may appear as a positive pulse (low to high) at 1310.

The SPAD core 1301N may be the same as the SPAD core 1301A in circuit details, so no operational details of the SPAD core 1301N are provided. As shown, the SPAD core 1310N includes a resistive element 1313, a coupling capacitor 1315, and a SPAD 1312 through which the core-specific SPAD 1312, the VSPAD voltage 1303 is supplied to the SPAD 1312. Inverter 1316 for latching and outputting the output generated by < RTI ID = 0.0 >), < / RTI > and a PMOS transistor 1317 for controlling the operation of inverter 1316 via shutter input 1308. The output of inverter 1316 may be provided to PPD core 1311 for further processing. In some embodiments, the VSPAD signal 1303, the VDD signal 1309, and the electronic shutter signal 1308 may be the row decoder / driver 1203 shown in FIG. 12 or the pixel processing unit 46 (or processor) of FIG. 2. May be supplied to each SPAD core 1301A, 1301N from an external unit, such as any other module of FIG. All of the SPAD core-specific outputs 1310, 1318 can collectively form the signals identified using reference numeral 506 in FIG. 5.

Thus, the electronic shutter signal 1308 is spatially correlated due to the adjacent positions of the SPAD cores 1301A, 1301N within the pixel 1300 with the outputs 1310, 1318 from the SPAD cores 1301A, 1301N. In addition to being correlated, it is correlated in time (or in a time manner). Additional pixel geometric information is shown in the example embodiments of FIGS. 6A-6C.

Like the pixel 700 of FIG. 7, the pixel 1300 of FIG. 13 also includes a PPD 508, a logic unit 1319, a first NMOS transistor 1320, a second NMOS transistor 1321, and a third NMOS transistor ( 1322, fourth NMOS transistor 1323, and fifth NMOS transistor 1324; Generate an internal input (TXEN signal) 1325; Receive external inputs RST signal 1326, VTX signal 1327 (and TX signal 1328), VPIX signal 1333, and SEL signal 1330; It has a floating diffusion (FD) node / junction 1332 and outputs a PIXOUT signal 510. Unlike pixel 700 of FIG. 7, pixel 1300 of FIG. 13 is also a complementary signal of TXEN signal 1325 and a second TXEN signal TXENB that can be supplied to the gate terminal of sixth NMOS transistor 1334. (1333). The sixth NMOS transistor 1334 may have a drain terminal connected to the source terminal of the first transistor 1320, and a source terminal connected to the ground (GND) potential 1335. The TXENB signal 1333 can be used to transfer the GND potential to the gate terminal of the second NMOS transistor 1321 (TX transistor). Without the TXENB signal 1333, the gate of the second transistor 1321 (TX transistor) is floated when the TXEN signal 1325 goes low, and charge transfer from the PPD 508 may not be terminated completely. This situation can be improved using the TXENB signal 1325. Additionally, the pixel 1300 may also include a storage diffusion (SD) capacitor 1336 and a seventh NMOS transistor 1335. The SD capacitor 1336 may be connected to the junction of the drain terminal of the second transistor 1321 and the source terminal of the seventh transistor 1335, and may form an SD node 1338 at the junction. The seventh NMOS transistor 1335 may receive a different second transmit (TX2) signal 1335 as an input at the gate terminal. The drain of the seventh transistor 1335 may be connected to the FD node 1331 as shown.

In some embodiments, the RST signal, VTX signal, VPIX signal, TX2 signal, and SEL signal may be supplied to the pixel 1300 from an external unit such as the row decoder / driver 1203 shown in FIG. 12. In some embodiments, SD capacitor 1336 is not a spare capacitor, but may just be a junction capacitor of SD node 1338. Comparison of FIGS. 5 and 13 shows that all of the SPADs 1301A, 1301N, etc. in the pixel 1300 collectively form the SPADs 503 of FIG. 5; All of the non-SPAD circuit elements from each SPAD core 1301A, 1301N, etc. collectively form the first control circuit 504 of FIG. 5; And all of the non-PPD circuit elements of the PPD core 502 can form the second control circuit 507 of FIG. 5.

In pixel 1300, the charge transfer trigger portion may include SPAD cores 1301A, 1301N (and other such cores) and logic unit 1319. The charge generation and transfer portion may include a PPD 508, NMOS transistors 1320-1322, 1334, and 1337, and an SD capacitor 1336. The charge collection and output portion may include NMOS transistors 1322-1324. It should be noted that the division into the respective portions of the various circuit components is exemplary and for illustrative purposes only. In some embodiments, these portions may include more or fewer other circuit components than those listed herein.

As mentioned above, except for the CDS based charge collection and output portion, the pixel configuration of FIG. 13 is substantially similar to that of FIG. Thus, for convenience, circuit parts and signals common between FIGS. 7 and 13 embodiments, such as transistors 1320-1234 and associated inputs (RST signal, SEL signal, VPIX signal, etc.) are discussed herein. It doesn't work. CDS is a noise reduction technique for measuring electrical values such as pixel / sensor output voltage (PIXOUT) in a way that allows for the elimination of unnecessary offsets. In some embodiments, a column-specific CDS unit (not shown) may be employed in pixel column unit 1205 (FIG. 12) to perform correlated double sampling. In the CDS, the output (s) of a pixel, such as pixel 1300 of FIG. 13, can be measured twice, once in known conditions and once in unknown conditions. The value measured from the known condition can be subtracted from the value measured from the unknown condition to produce a value having a known relationship to the physical quantity to be measured, ie a PPD charge representing the pixel-specific portion of the received light. Using CDS, noise can be reduced by removing the pixel's reference voltage (eg, the voltage of the pixel after reset) from the signal voltage of the pixel at the end of each charge transfer. Thus, in the CDS, the reset / reference value is sampled before the charge of the pixel is transferred to the output, which is then subtracted from the value after the charge of the pixel is transferred.

In the embodiment of FIG. 13, the SD capacitor 1336 (or associated SD node 1338) stores the PPD charge before the PPD charge is transferred to the floating diffusion node 1331, so that the charge is floating diffusion node 1331. Appropriate reset values are established (and sampled) at floating diffusion node 1331 before being sent to. As a result, each pixel-specific output PIXOUT1 and PIXOUT2 may be processed in a column-specific CDS unit (not shown) of the pixel column unit 1205 (FIG. 12) to obtain a pair of pixel-specific CDS outputs. . The pixel-specific CDS outputs are then converted by the respective column-specific ADC unit (not shown) of the pixel column unit 1205 into digital values such as P1 and P2 indicated by arrows 1213 in FIG. 12. Can be. The transistors 1334, 1337, TXENB signal 1333, and TX2 signal 1339 of FIG. 13 provide auxiliary circuit components needed to facilitate CDS based charge transfer. In one embodiment, P1 and P2 values may be generated in parallel using the same pair of ADC circuits, for example, as part of a column-specific ADC unit. Thus, the difference between the reset levels of the PIXOUT1 and PIXOUT2 signals and the corresponding PPD charge levels can be converted into digital values by a column parallel ADC and output as pixel-specific signal values (ie, P1 and P2). Based on Equation 2 enables the calculation of the pixel-specific TOF value of the returned pulse 37 for the pixel 1300. As mentioned above, this calculation may be performed by the pixel processing unit 46 or the processor 19 of the system 15. As a result, the pixel-specific distance to the object 26 (FIG. 2) can also be determined using, for example, equations (3) and (4). The pixel-by-pixel charge collection operation can be repeated for all of the rows of pixels of pixel array 42. Based on all pixel-specific distance or range values for the pixels 43 of the pixel array 42, the 3D image of the object 26 is for example generated by the processor 19 and the system 15 May be displayed on an appropriate display or user interface associated with the. A 2D image of the object 26 can be generated by simply adding the P1 and P2 values, for example when no range values are calculated or when a 2D image is needed even though range values are possible. In some embodiments, such a 2D image may be simply a gray scale image, for example when an IR laser is used.

It should be noted that the pixel configurations shown in FIGS. 7 and 13 are merely exemplary. Other types of PPD based pixels with multiple SPADs can also be used to implement the subject matter described herein. Such pixels can be, for example, pixels having a single output (such as the PIXOUT line 510 of the embodiments of FIGS. 7 and 13) or a double in which the PIXOUT1 and PIXOUT @ signals can be output through different outputs at the pixel. It can include pixels with outputs.

FIG. 14 shows when the pixels 1300 of the embodiment shown in FIG. 13 are used in a pixel array such as the pixel array 42 of FIGS. 2 and 12 for measuring TOF values according to the subject matter described herein. 2 is a timing diagram 1400 showing exemplary timing of different signals of the system 15 of FIG. The timing diagram 1300 of FIG. 14 shows, in particular, the waveforms of the VTX signal, the shutter signal, the VPIX signal, and the TX signal, and various time intervals such as, for example, a PPD reset event, a shutter on interval, a delay time interval (T _dly ), and the like. Or similar to timing diagram 900 of FIG. 9 for identification of events. The foregoing extensive discussion of the timing diagram 900 of FIG. 9 provides a brief discussion of the features distinguished from the timing diagram 1400 of FIG. 14 for convenience.

In FIG. 14, various externally supplied signals, such as VPIX signal 1333, RST signal 1326, electronic shutter signal 1308, amplitude modulated signal (VTX) 1327, TX2 signal 1333, and An internally generated TXEN signal 1325 is identified using the same reference numbers as those used in FIG. 13. Likewise, for convenience, the same reference numeral 1331 is used to indicate the floating diffusion node 1332 of FIG. 13 and the associated voltage waveform of the timing diagram of FIG. 14. Although the transmission mode (TXRMD) signal 1401 is shown in FIG. 14, it is not shown in FIG. 13 or in the earlier timing diagram of FIG. 10. In some embodiments, TXRMD signal 1401 is generated internally by logic unit 1319 or external to logic unit 1319 by a row decoder / driver (such as row decoder / driver 1203 in FIG. 12). Can be supplied as In one embodiment, the logic unit 1319 generates an output based on the G () function (FIG. 10) and then logically ORs the output with an internally generated signal, such as the TXRMD signal 1401. Logic circuits (not shown) to obtain the final TXEN signal 1325. As shown in FIG. 14, in one embodiment, this internally generated TXRMD signal 1401 remains low while the electronic shutter is on, but is subsequently activated high to cause the TXEN signal 1325 to be logic one. It may facilitate the transfer of residual charge of the PPD (event 1402 of FIG. 14).

PPD reset event 1403, delay time T _dly 1404, TOF interval T _tof 1405, shutter off interval 1406, shutter on or activation interval T _sh (1407 or 1408). And the FD reset event 1409 is similar to the corresponding events or time intervals shown in FIG. 9. Thus, no further discussion of these parameters is provided. Initially, the FD reset event 1409 causes the FD signal 1331 to go high as shown. The SD node 1338 is reset high after the PPD 508 is preset low. More specifically, during the PPD preset event 1403, the TX signal 1328 can be high, the TX2 signal 1333 can be high, the RST signal 1326 can be high, and the VPIX signal ( 1329 may be low, filling electrons PPD 508 and preset PPD 508 to 0V. Thereafter, the TX signal 1328 goes low, and the TX2 signal 1333 and the RST signal 1326 briefly remain high, resetting the SD node 1338 high with the high VPIX signal 1333 and SD. Electrons may be removed from the capacitor 1336. At the same time, FD node 1331 is reset (following FD reset event 1409). The voltage or SD reset event of the SD node 1338 is not shown in FIG. 14.

In contrast to the embodiment of FIGS. 7 and 9, when the shutter 1308 is activated and the VTX signal 1327 is tilted up, the PPD charge is amplitude modulated, and initially SD, as shown in the TX waveform 1328. Sent to node 1338 (via SD capacitor 1336). Upon detection of photons by at least two SPADs in pixel 1300 (FIG. 13) during shutter on interval 1408, TXEN signal 1325 goes low and from PPD 508 to SD node 1338. Initial charge transfer stops. The transmitted charge stored at the SD node 1338 may be read at the PIXOUT line 510 (as the PIXOUT1 output) during the first readout period 1412. In the first readout period 1412, the RST signal 1326 may be briefly activated high after the electronic shutter 1308 is inactivated or turned off to reset the floating diffusion node 1331. Thereafter, the TX2 signal 1409 becomes a high pulse to transfer charge from the SD node 1338 to the floating diffusion node 1331 while the TX2 signal 1335 is high. Floating diffusion voltage waveform 1331 shows charge transfer operation. The transmitted charge can then be read (as a PIXOUT1 voltage) using the SEL signal 1330 (not shown in FIG. 14) via the PIXOUT line 510 during the first readout period 1412.

During the first readout period 1412, after the initial charge is transferred from the SD node to the FD node and the TX2 signal 1339 returns to the logic low level, the TXRMD signal 1401 is activated (pulsed) high to TXEN Generate a high pulse at input 1325, and as indicated by reference number 1402 in FIG. 14, this in turn generates a high pulse at TX input 1328 to convert residual charge of PPD 508 to SD node 1338. (Via SD capacitor 1336). Thereafter, when the RST signal 1326 is briefly reactivated high, the FD node 1331 may be reset again. The second RST high pulse may define a second readout period 1413, where the TX2 signal 1339 is pulsed high again to draw the remaining charge of the PPD 508 while the TX2 signal is high to the SD node ( From 1338 to floating diffusion node 1331 (event 1402). Floating diffusion voltage waveform 1331 shows a second charge transfer operation. The transmitted residual charge can then be read (as the PIXOUT2 voltage) using the SEL signal 1332 (not shown in FIG. 14) via the PIXOUT line 510 during the second readout period 1418. As mentioned above, the PIXOUT1 and PIXOUT2 signals can be converted into corresponding digital values P1, P2 by a suitable ADC unit (not shown). In certain embodiments, P1 and P2 values may be used in Equation 3 or 4 to determine the pixel-specific distance / range between pixel 1300 and object 26. The SD based charge transfer shown in FIG. 14 enables the generation of a pair of pixel-specific CDS outputs, as discussed above with reference to the discussion of FIG. 13. CDS-based signal processing provides additional noise reduction.

15 shows a block diagram of an example embodiment of a time-decomposition sensor 1500 in accordance with the subject matter described herein. The time-decomposition sensor 1500 may include a SPAD circuit 1501, a logic circuit 1503, and a PPD circuit 1505.

The SPAD circuit 1501 includes a SPAD for detecting photons, a first input for receiving a VSPAD voltage, a second input for receiving a shutter signal for controlling the opening and closing of the electronic shutter, and a V _DD voltage for receiving the VAD signal. And a third input, and an output for outputting a detection event (DE) signal. In response to receiving the photons, the SPAD circuit 1501 may output a pulse signal that rises quickly from the VSPAD voltage to a voltage lower than the SPAD breakdown voltage, and then returns more slowly to the VSPAD voltage.

The logic circuit 1503 is a first input coupled to the DE signal output from the SPAD circuit 1501, a second to receive a TXRMD signal for completely transferring charge remaining in the PPD of the PPD circuit 1505 to the FD node. It may include an input and an output for outputting a TXEN signal.

The PPD circuit 1505 is a VTX for partially or completely transferring the first input, charge, connected to the TXEN signal output from the logic circuit 1503 from the PPD of the PPD circuit 1505 to the FD node of the PPD circuit 1505. A second input to receive a signal, a third input to reset the charge at the FD node and a RST signal to preset the charge in the PPD, and a fourth input to receive a VPIX voltage for the PPD circuit 1505 A fifth input for receiving a SEL signal to enable reading of the PIXOUT1 signal (which represents the charge of the FD node) or the PIXOUT2 signal (which represents the charge remaining in the PPD), and a PIXOUT1 signal and a PIXOUT2 in response to the SEL signal. It may include a PIXOUT output for outputting a signal.

16 shows a conceptual diagram of an example embodiment of a SPAD circuit 1501 of a time-decomposition sensor 1500 according to the subject matter described herein. In one embodiment, the SPAD circuit 1501 may include a resistor 1601, a SPAD 1603, a capacitor 1605, a p-type MOSFET transistor 1607, and a buffer 1609. The resistor 1601 may include a first terminal and a second terminal for receiving a VSPAD voltage. The SPAD 1603 may include a positive electrode connected to a ground potential and a negative electrode connected to a second terminal of the resistor 1601. In other embodiments, the locations of resistor 1601 and SPAD 1603 may be exchanged. The SPAD 1603 may respond to light. In response to receiving the photons, the SPAD 1603 may output a pulse signal that rises sharply from the VSPAD voltage to a voltage lower than the breakdown voltage and then returns more slowly to the VSPAD voltage. In one example, the breakdown voltage may be a specific threshold voltage.

The capacitor 1605 may include a first terminal and a second terminal connected to the cathode of the SPAD 1603. In alternative embodiments, the capacitor 1604 may be omitted. P-type MOSFET 1607 includes a first S / D terminal connected to a second terminal of capacitor 1605, a gate for receiving a shutter signal, and a second S / D for receiving a VPIX voltage (V _DD ). It may include a terminal. The buffer 1609 may include an input connected to the second terminal of the capacitor 1605 and an output for outputting a DE signal. The DE signal may correspond to the DE output of the SPAD circuit 1501. In alternative embodiments, buffer 1609 may be an inverter.

17 shows a conceptual diagram of an example embodiment of a logic circuit 1503 of a time-decomposition sensor 1500 in accordance with the subject matter described herein. The logic circuit 1503 may include a latch 1701 and a two-input OR gate 1703.

The latch 1701 may include an input and an output coupled to the DE signal output from the SPAD circuit 1501. In response to the DE signal, latch 1701 may, for example, go from logic 1 to logic 0 and output a logic signal that holds logic 0. In other words, latch 1701 may convert the pulse type signal from logic 1 to logic 0 and into a signal that maintains logic 0 without returning to logic 1 until reset. The latch output is triggered by the leading edge of the DE signal, where the leading edge can be positive or negative depending on the design of the SPAD circuit 1501.

The two input OR gate 1703 may include a first input coupled to the output of the latch 1701, a second input for receiving a TXRMD signal, and an output for outputting a TXEN signal. The two input OR gate 1703 may perform a logical OR function and output the result as a TXEN signal. In particular, a TXRMD signal that occurs when a photon is received by the SPAD circuit 1501 when the shutter signal is logic 1, or when the remaining charge of the PPD of the PPD circuit 1505 is completely transferred to the FD node for reading into the PIXOUT2 signal. Is a logic one, the output of the two input OR gate 1703 is a logic one.

18 shows a conceptual diagram of an exemplary embodiment of a PPD circuit 1505 of a time-decomposition sensor 1500 according to the subject matter described herein. The PPD circuit 1505 may include a PPD 1801, a first transistor 1803, a second transistor 1805, a third transistor 1807, a fourth transistor 1809, and a fifth transistor 1811. have.

The PPD 1801 may include an anode and a cathode connected to ground potential. PPD 1801 may store charge in a manner similar to a capacitor. In one embodiment, the PPD 1801 is covered and thus not responding to light, and can be used as a TCC instead of a light-sensitive element.

The first transistor 1803 may include a gate terminal connected to the TXEN signal output of the logic circuit 1503, a first S / D terminal for receiving a VTX signal, and a second S / D terminal. The first transistor 1803 receives the VTX signal and causes the VTX signal to pass through the first transistor 1803 under the control of the TXEN signal to output the TX signal at the second S / D terminal of the first transistor 1803. can do.

The second transistor 1805 includes a gate terminal connected to the second S / D terminal of the first transistor 1803, a first S / D terminal connected to the cathode of the PPD 1801, and a second S / D terminal. It may include. The second transistor 1805 may receive the TX signal at the gate terminal and transfer the charge of the PPD 1801 of the source terminal to the drain terminal connected to the FD node. Although not shown in FIG. 18, parasitic capacitance may be present between the FD node and ground. In one embodiment, physical capacitance may also be coupled between the FD node and ground.

The third transistor 1807 is connected to a gate terminal for receiving an RST signal, a first S / D terminal for receiving a VPIX voltage, and a second S / D terminal connected to the second S / D terminal of the second transistor 1805. It may include a D terminal.

The fourth transistor 1809 is a gate terminal connected to the second S / D terminal of the second transistor 1805, a first S / D terminal connected to the first S / D terminal of the third transistor 1807, and It may include a second S / D terminal.

The fifth transistor 1811 is a gate terminal for receiving a SEL signal, a first S / D terminal connected to the second S / D terminal of the fourth transistor 1809, and a PIXOUT output of the PPD circuit 1505. It may include 2 S / D terminals. The fifth transistor 1811 may receive a SEL signal for selecting a pixel to read the charge (as PIXOUT1) of the FD node or the residual charge (as PIXOUT2) of the PPD 1801.

The charge transmitted from the PPD 1801 to the FD node is controlled by the TX signal. In one embodiment, the VTX signal is coupled through the first transistor to become a TX signal. The VTX signal slopes upwards, transferring more and more charge from the PPD 1801 to the FD node. The amount of charge transferred from the PPD 1801 to the FD node may be a function of the level of the TX signal, and the slope of the TX signal may be a function of time. Thus, the charge transmitted from the PPD 1801 to the FD node may be a function of time. During transfer of charge from the PPD 1801 to the FD node, when the second transistor 1805 is turned off in response to the SPAD circuit 1501 detecting incident photons, the charge of the charge from the PPD 1801 to the FD node is lost. The transmission is stopped. The amount of charge transferred to the FD node and the amount of charge remaining in the PPD 1801 can both be associated with the TOF of the incident photons. The transfer of charge from the PPD 1801 to the FD node based on the TX signal and the detection of incident photons can be considered to provide a single-ended differential conversion of the charge's time.

The fourth transistor 1809 may convert charge stored in the FD node into a voltage at the second S / D terminal of the fourth transistor 1809. The SEL signal selects a pixel so that the PIXOUT1 signal corresponds to the charge transferred to the FD node, or subsequently the PIXOUT2 signal corresponding to the charge remaining in the PPD 1801 after the residual charge of the PPD 1801 is transferred to the FD node. Can be used to read. In one embodiment, the ratio of the PIXOUT1 signal to the sum of the PIXOUT1 signal and the PIXOUT2 signal may be proportional to the difference between the TOF and the delay time of the optical signal received by the pixel, as shown in the ratio of equation (2). . In an embodiment in which an optical pulse is transmitted after the VTX signal begins to slope upward, the delay time may be negative.

For the time-decomposition sensor 1500, the ratios described in Equation 2 can be used to determine the depth or range of the subject, and fluctuations between measurements if the sum of the PIXOUT1 and PIXOUT2 signals does not vary from measurement to measurement. Less sensitive to In one embodiment, the VTX signal may be ideally linear and ideally uniform across different pixels of the TOF pixel array. In practice, however, the VTX signal that may be applied to the different pixels of the TOF pixel array may vary from pixel to pixel, thus causing errors in range measurements that depend on changes in the VTX signal between pixels and may also vary from measurement to measurement.

In one embodiment, the first transistor 1803, the second transistor 1805, the third transistor 1807, the fourth transistor 1809, and the fifth transistor 1811 are each an n-type MOSFET or a p-type, respectively. It may be a MOSFET. However, any suitable transistor can be used, so the subject matter described herein is not limited to using n-type MOSFETs or p-type MOSFETs.

FIG. 19 shows an exemplary relative signal timing diagram for the time-resolved sensor 1500 of FIG. 15 in accordance with the subject matter described herein. In Fig. 19, during the shutter off (initialization) period, each of the RST signal, the VTX signal, and the TX signal becomes high (logical 1), and then returns to 0 (logical 0) to reset the PPD circuit 1505. TXEN signal is high. The PPD 1801 may be filled with charge up to its total capacity in the initialization period. The VTX signal and the TX signal go low to turn off the second transistor 1805 of the PPD circuit 1505. The VPIX voltage goes high, thus causing the FD node to reset. The light pulse is transmitted towards the object when the RST signal returns to zero or after a while. The VTX signal then begins to slope up, and the shutter signal goes high to start the shutter on period.

As the VTX signal slopes up, the TX signal also slopes up and the charge of the FD node begins to decrease in response to the TX signal. The returned light pulse causes the TXEN signal to go low (logical 0), thus stopping the transfer of charges between the FD node and the PPD 1801.

The delay time T _dly represents the time between the start of the transmitted light pulse and the time when the TX signal starts to slope upward. The flight time T _tof represents the time between the start of the transmitted light pulse and the time when the return signal is received. The electronic shutter time T _sh represents the time (shutter on section) from when the electronic shutter is opened to when the electronic shutter is closed. In one embodiment, the electronic shutter time T _sh may be less than or equal to the inclination time of the VTX signal.

The transferred charge is read as the PIXOUT1 signal during the transferred charge read interval. While the shutter signal is low, the RST signal goes high second to reset the charge at the FD node, after which the TXRMD signal, TXEN signal, and TX signal go high to transfer the charge remaining in the PPD 1801 to the PIXOUT2 signal. Send it to the FD node for reading.

20 shows a block diagram of another exemplary embodiment of a time-decomposition sensor 2000 according to the subject matter described herein. The time-decomposition sensor 2000 may include a SPAD circuit 2001, a logic circuit 2003, and a second PPD circuit 2005.

The SPAD circuit 2001 receives a SPAD for detecting photons, a first input for receiving a VSPAD voltage, a second input for receiving a shutter signal for controlling the opening and closing of the electronic shutter, and a VDD voltage VDD. A third input for outputting and an output for outputting a detection event (DE) signal. In response to receiving the photons, the SPAD circuit 2001 may output a pulse signal that quickly goes from VSPAD to zero and slowly returns to VSPAD. In one embodiment, the SPAD circuit 2001 may be the same as the SPAD circuit 1501 shown in FIG. 15.

The logic circuit 2003 has a first input coupled to the DE output of the SPAD circuit 2001, a second input for receiving a TXRMD signal for fully transferring charge remaining in the PPD of the second PPD circuit 2005, and It may include an output for outputting the TXEN signal. In one embodiment, the logic circuit 2003 may be the same as the logic circuit 1503 shown in FIG. 15.

The second PPD circuit 2005 has a first input coupled to the TXEN signal output from the logic circuit 2003, a second input coupled to the second input of the logic circuit 2003 for receiving the TXRMD signal, and a second PPD. Reset charge at the third input, FD1 node to receive a VTX signal for partially or fully transferring charge from the PPF of circuit 2005 to the first floating diffusion FD1 node of second PPD circuit 2005. And a fourth input for receiving an RST signal for presetting the charge of the PPD, a fifth input for receiving a VPIX signal for the second PPD circuit 2005, and a PIXOUT1 signal corresponding to the charge of the FD1 node. And a sixth input for receiving a SEL signal to read from the PIXOUT2 output a PIXOUT2 signal corresponding to the charge remaining in the PPD of the second PPD circuit 2005.

21 shows a conceptual diagram of an exemplary embodiment of a second PPD circuit 2005 of a time-decomposition sensor 2000 according to the subject matter described herein. The second PPD circuit 2005 includes a PPD 2101, a first transistor 2103, a second transistor 2105, a third transistor 2107, a fourth transistor 2109, a fifth transistor 2111, and a sixth transistor. The transistor 2113, the seventh transistor 2115, the eighth transistor 2117, and the ninth transistor 2119 may be included.

The PPD 2101 may include an anode connected to ground potential, and a cathode. PPD 2101 can store charge in a manner similar to a capacitor. In one embodiment, the PPD 2101 is covered and thus not responding to light, and may be used as a TCC instead of a light-sensitive element.

The first transistor 2103 is a gate terminal connected to the output of a logic circuit 2003 for receiving a TXEN signal output, a first S / for receiving a VTX voltage for controlling the transfer of charge from the PPD 2101. And a second S / D terminal.

The second transistor 2105 is connected to a gate terminal connected to a second S / D terminal of the first transistor 2103 for receiving a TX signal for transferring charge from the PPD 2101, to a cathode of the PPD 2101. And a second S / D terminal connected to the first floating diffusion (FD1) node through which charge is transferred from the PPD 2101. The FD1 node may have a first capacitor. Between the FD1 node and ground, there may be parasitic capacitance not shown in FIG. In one embodiment, physical capacitance may also be coupled between the FD1 node and ground. The charge transmitted from the PPD 2101 to the FD1 node through the second transistor 2105 may be controlled by the TX signal.

The third transistor 2107 includes a gate terminal connected to the FD1 node and the second S / D terminal of the second transistor 2105, a first S / D terminal for receiving a VPIX voltage, and a second S / D terminal. It may include. The third transistor 2107 may convert charge stored in the FD1 node into a voltage at the second S / D terminal of the third transistor 2107.

The fourth transistor 2109 is a gate terminal for receiving the RST signal for setting the charge level of the FD1 node, a first S / D terminal for receiving the VPIX voltage, and a second S / of the second transistor 2105. It may include a second S / D terminal connected to the D terminal.

The fifth transistor 2111 is a gate terminal for receiving a SEL signal for reading the charge of the FD1 node, a first S / D terminal connected to the second S / D terminal of the third transistor 2107, and an FD1 node. And a second S / D terminal connected to the pixel output PIXOUT1 data line for outputting a voltage corresponding to the charge of the PIXOUT1 signal.

The sixth transistor 2113 is a gate terminal for receiving a TXRMD signal for completely transferring the charge remaining in the PPD 2101 to the second floating diffusion (FD2) node, and a first S connected to the cathode of the PPD 2101. / D terminal, and a second S / D terminal connected to the FD2 node. The FD2 node may have a second capacitance. Between the FD2 node and ground, there may be parasitic capacitance that is not shown in FIG. In one embodiment, physical capacitance may also be coupled between the FD2 node and ground. In one embodiment, the second capacitance of the FD2 node may be equal to the first capacitance of the FD1 node. Any residual charge remaining in PPD 2101 may be transferred to node FD2 through sixth transistor 2113.

The seventh transistor 2115 includes a gate terminal connected to the second S / D node and the FD2 node of the sixth transistor 2113, a first S / D terminal for receiving a VPIX voltage, and a second S / D terminal. It may include. The seventh transistor 2115 may convert charge stored in the FD2 node into a voltage at the second S / D terminal of the seventh transistor.

The eighth transistor 2117 is a gate terminal for receiving the RST signal for setting the charge level of the FD2 node, a first S / D terminal for receiving the VPIX signal, and a second S / of the sixth transistor 2113. It may include a second S / D terminal connected to the D terminal.

The ninth transistor 2119 is a gate terminal for receiving a SEL signal for selecting a pixel and reading a voltage corresponding to the charge of the FD2 node, and a first S / D terminal connected to the second S / D terminal of the seventh transistor 2115. An S / D terminal and a second S / D terminal connected to a pixel output (PIXOUT2) data line for outputting a voltage corresponding to the charge of the FD2 node as a PIXOUT2 signal.

In one embodiment, the VTX signal (and the TX signal) can be sloped upward to transfer charge from the PPD 2101 to the FD1 node. The amount of charge transferred from the PPD 2101 to the FD1 node may be a function of the level of the TX signal, and the slope of the TX signal may be a function of time. Thus, the charge transmitted from the PPD 2101 to the FD1 node may be a function of time. During transfer of charge from the PPD 2101 to the FD1 node, the charge from the PPD 2101 to the FD1 node is turned off when the second transistor 2105 is turned off in response to detecting the photon incident by the SPAD circuit 2001. Is stopped, and both the amount of charge transferred to the FD1 node and the amount of charge remaining in the PPD 2101 is associated with the TOF of the incident photons. The transfer of charge from the PPD 2101 to the FD1 node based on the TX signal and the detection of incident photons provides a single end-to-differential conversion over the time of charge.

For the time-decomposition sensor 2000, the ratio described in Equation 2 can be used to determine the depth or range of the object, and if the sum of the PIXOUT1 and PIXOUT2 signals does not change in units of measurement, Less sensitive. In one embodiment, the VTX signal may be ideally linear and ideally uniform across different pixels of the TOF pixel array. In practice, however, the VTX signal applied to the different pixels of the TOF pixel array fluctuates on a pixel basis, thus causing errors in range measurement depending on the variation of the VTX signal on a pixel basis and may also fluctuate on a measurement basis. .

In one embodiment, the first transistor 2103, the second transistor 2105, the third transistor 2107, the fourth transistor 2109, the fifth transistor 2111, the sixth transistor 2113, and the seventh transistor. 2115, eighth transistor 2117, and ninth transistor 2119 may each be an n-type MOSFET or a p-type MOSFET, but any other suitable transistor may be used.

22 shows an exemplary relative signal timing diagram 2200 for a time-resolved sensor 2000 according to the subject matter described herein. The signal timing diagram of FIG. 22 is similar to the signal timing diagram of FIG. 19, and similarities are described with reference to FIG. 19. The signal timing diagram of FIG. 22 differs in that it includes the FD signal and at the end of the shutter-on period, the remaining charge of the PPD 2101 is transferred to the FD2 node by the operation of the TXRMD signal. In addition, the PIXOUT1 and PIXOUT2 signals can be read simultaneously.

Although the second PPD circuit 2005 determines the maximum range depending on the total capacity which does not change, the actual implementation of the time-decomposition sensor 2000 is based on the thermal noise between the different second PPD circuits 2005. It should be noted that the total dose variation for (2101) may be experienced. In addition, the VTX signal may have different inclinations (tilts) based on the position of the pixels in the pixel array. That is, the slope (tilt) of the VTX signal at the pixel may vary depending on how close the pixel is from the source of the VTX signal.

FIG. 23 shows a block diagram of another exemplary embodiment of a time-decomposition sensor 2300 in accordance with the subject matter described herein. The time-decomposition sensor 2300 may include one or more SPAD circuits 2301a-2301n, a logic circuit 2303, and a third PPD circuit 2305.

In one embodiment, each of the one or more SPAD circuits 2301 may include a SPAD 3211, a resistor 2313, a capacitor 2315, a p-type MOSFET transistor 2317, and a buffer 2319. Can be. The SPAD 2311 may include a positive electrode and a negative electrode connected to a ground potential. The resistor 2313 may include a first terminal for receiving a VSPAD voltage and a second terminal connected to a cathode of the SPAD 2311. In other embodiments, the locations of SPAD 2311 and resistor 2313 may be exchanged. SPAD 2311 may respond to light. In response to receiving the photons, the SPAD 2311 may output a pulse signal that suddenly goes from the VSPAD voltage to a voltage lower than the breakdown voltage and then returns more slowly to the VSPAD voltage.

The capacitor 2315 may include a first terminal and a second terminal connected to the cathode of the SPAD 2311. In alternative embodiments, capacitor 2315 may be omitted. P-type MOSFET transistor 2317 is a first S / D terminal connected to a second terminal of capacitor 2315, a gate terminal for receiving a shutter signal, and a second S for receiving VPIX voltage (V _DD ). May include the / D terminal. The buffer 2319 may include an input connected to the second terminal of the capacitor 2315, and an inverted output capable of outputting a DE signal corresponding to the output of the SPAD circuit 2311. In alternative embodiments, buffer 2319 may not be inverted.

Logic circuit 2303 may include inputs coupled to each DE signal output of one or more SPAD circuits 2301a-2301n and outputs that output a TXENB signal, which may be the inversion of the TXEN signal and the TXEN signal. Can be.

The third PPD circuit 2305 includes the capacitive device SC, the first transistor 2351, the second transistor 2353, the third transistor 2355, the fourth transistor 2357, the fifth transistor 2359, The sixth transistor 2361, the seventh transistor 2363, the eighth transistor 2365, the ninth transistor 2367, the tenth transistor 2369, the eleventh transistor 2237, the twelfth transistor 2373, and The thirteenth transistor 2375 may be included.

The capacitive device SC may include a first terminal connected to a ground potential, and a second terminal. The capacitive device SC can store charge in a manner similar to a capacitor. In one embodiment, the capacitive device SC may be a capacitor. In another embodiment, the capacitive device SC may be a PPD that is covered and does not respond to light. In another embodiment, the capacitive device SC can be used as part of the TCC.

The first transistor 2351 has a gate terminal 2351 connected to the RST signal, a first S / D terminal connected to the ground potential, and a second S / D terminal connected to the second terminal of the capacitive device SC. It may include.

The second transistor 2353 is a gate terminal connected to a TXA signal, a first S / D terminal connected to a first floating diffusion (FD1) node, and a second S / D terminal and a capacitive property of the first transistor 2351. It may include a second S / D terminal connected to the second terminal of the device (SC). The first floating diffusion FD1 node is represented by a capacitor sign in FIG. 23. Between the FD1 node and ground, there may be parasitic capacitances not shown in FIG. In one embodiment, the physical capacitance may also be coupled between the FD1 node and ground.

The third transistor 2355 includes a gate terminal connected to the FD1 node and the first S / D node of the second transistor 2353, a first S / D terminal connected to the VPIX voltage, and a second S / D terminal. can do. The third transistor 2355 may convert the charge of the FD1 node into a voltage at the second S / D terminal of the third transistor 2355.

The fourth transistor 2357 includes a gate terminal connected to the RST signal, a first S / D terminal connected to the VPIX voltage, and a second S / D terminal of the first transistor 2351 and the first of the capacitive device SC. It may include a second S / D terminal connected to the two terminals.

The fifth transistor 2359 may include a gate terminal connected to the TXEN signal, a first S / D terminal connected to the VTX signal, and a second S / D terminal connected to the gate terminal of the second transistor 2353. have.

The sixth transistor 2361 includes a gate terminal connected to the TXENB signal, a first S / D terminal connected to a ground potential, and a gate terminal of the second transistor 2353 and a second S / D of the fifth transistor 2359. It may include a second S / D terminal connected to the terminal.

The seventh transistor 2363 has a gate terminal connected to the SEL signal, a first S / D terminal connected to the second S / D terminal of the third transistor 2355, and a second connected to the pixel output line PIXA. It may include an S / D terminal.

The eighth transistor 2365 includes a gate terminal connected to the TXB signal, a first S / D terminal connected to a second floating diffusion (FD2) node, and a second S / D terminal of the first transistor 2351, and a capacitive property. And a second S / D terminal connected to the second terminal of the device SC and the second S / D terminal of the second transistor 2353. The second floating diffusion FD2 node is indicated by the capacitor symbol in FIG. Between the FD2 node and ground, there may be parasitic capacitances not shown in FIG. In one embodiment, physical capacitance may also be coupled between the FD2 node and ground.

The ninth transistor 2367 includes a gate terminal connected to the FD2 node and the first S / D terminal of the eighth transistor 2365, a first S / D terminal connected to the VPIX voltage, and a second S / D terminal. can do. The ninth transistor 2367 may convert the charge of the FD2 node into a voltage at the second S / D terminal of the ninth transistor 2367.

The tenth transistor 2369 includes a gate terminal connected to the RST signal, a first S / D terminal connected to a VPIX voltage, a second S / D terminal of the first transistor 2351, and a first of the capacitive device SC. A second S / D terminal connected to the second terminal and the second S / D terminal of the eighth transistor 2365 may be included.

The eleventh transistor 2861 may include a gate terminal connected to the TXENB signal, a first S / D terminal connected to the VTX signal, and a second S / D terminal connected to the gate terminal of the eighth transistor 2365. have.

The twelfth transistor 2373 includes a gate terminal connected to a TXEN signal, a first S / D terminal connected to a ground potential, a gate terminal of an eighth transistor 2365, and a second S / D of an eleventh transistor 2237. It may include a second S / D terminal connected to the terminal.

The thirteenth transistor 2375 may include a gate terminal connected to a SEL signal, a first S / D terminal connected to a second S / D terminal of a ninth transistor 2367, and a second connected to a pixel output line PIXB. It may include an S / D terminal.

24 shows an exemplary relative signal timing diagram for a time-resolved sensor 2300 in accordance with the subject matter described herein. The signal timing diagram of FIG. 24 is similar to the signal timing diagrams of FIGS. 19 and 22, with similarities being described with reference to FIG. 19. The signal timing diagram of FIG. 24 differs from the signal timing diagram of FIG. 22 by not including the TXRMD signal and the TX signal but instead including the TXENB signal, the TXA signal, and the TXB signal.

In the signal timing diagram of FIG. 24, the TXENB signal is the inversion of the TXEN signal. When the shutter signal is active high, the TXEN signal is active and the VTX signal passes through the fifth transistor 2359 to cause the TXA signal to become active. Charge of the capacitive device SC is transmitted to the FD1 node through the second transistor 2353. At the same time, the ground potential passes through the twelfth transistor 2373 so that the TXB signal is deactivated.

When a detection event DE occurs, the TXEN signal is deactivated and the TXENB signal is activated. When the TXEN signal is deactivated, the TXA signal is also deactivated and the transfer of charge from the capacitive device SC through the second transistor 2353 is stopped. When the TXENB signal is activated, the TXB signal is activated and charge is transmitted from the capacitive device SC through the eighth transistor 2365 to the FD2 node.

When the shutter signal ends, the TXB signal is deactivated and the charge is stopped from the capacitive device SC through the eighth transistor 2365 to the FD2 node. Respective voltages associated with the charges of the FD1 node and the FD2 node are read from the PIXA and PIXB output lines.

Variations in the slope of the VTX signal between pixels and variations in the capacitance of the capacitive device SC are provided as long as the second transistor 2353 (TXA) and the eighth transistor 2365 (TXB) are in linear mode during the active shutter signal. Note that this does not cause range measurement errors.

25 shows a flowchart of a method 2500 of time decomposition using a time-decomposition sensor 2300 according to the subject matter described herein. The method starts at 2501. At 2502, an active shutter signal is generated. At 2503, one or more photons incident on the at least one SPAD circuit 2301 are detected during the active shutter signal in which one or more detected photons are reflected from the object. At 2504, an output signal is generated based on the detection event DE. At 2505, based on the output signal for the detection event DE, a first activation signal (eg TXEN) and a second activation signal (eg TXENB) are generated. In one embodiment, the first active signal is activated in response to the start of the active shutter signal and deactivated in response to the output signal, and the second active signal is activated in response to the output signal and in response to the end of the active shutter signal. Deactivated.

At 2506, if the first active signal is active, charge of the capacitive device SC is transferred to the first floating diffusion FD1 node to form a first charge at the first floating diffusion FD1 node. At 2507, if the second active signal is active, residual charge of the capacitive device SC is transferred to the second floating diffusion FD2 node to form a second charge at the second floating diffusion FD2 node. At 2508, a first voltage based on the first charge and a second voltage based on the second charge are output. The first ratio of the first voltage to the sum of the first and second voltages is proportional to the flight time of one or more detected photons, and the second ratio of the second voltage to the sum of the first and second voltages Is proportional to the flight time of one or more detected photons. At 2509, the method ends.

In one embodiment, transmitting the first and second charges includes changing the VTX signal (or drive signal) in accordance with a ramp function, where the VTX signal (or drive signal) is one. Or more photons begin to change until the end of the active shutter signal in response to the start time of the detected light pulse. Additionally, transferring charge of the capacitive device to the first floating diffusion (FD1) node to form a first charge at the first floating diffusion (FD1) node may result in a VTX signal (or drive signal) when the first active signal is active. It may be further based on the level of, and to transfer the remaining charge of the capacitive device to the second floating diffusion (FD2) node to form a second charge at the second floating diffusion (FD2) node, the second active signal is active When based on the level of the VTX signal (or drive signal).

In another embodiment, the first ratio of the first voltage to the first and second voltages may be more proportional to the delay time subtracted from the flight time of one or more detected photons. Similarly, the second ratio of the second voltage to the first and second voltages may be more proportional to the delay time subtracted from the flight time of one or more detected photons, wherein the delay time is equal to the transmission time of the light pulse. It may include the time between the start and the time when the VTX signal (or drive signal) started to change.

LiDAR (light detection and distance measurement) systems using SPADs typically do not provide light intensity based imaging capabilities. Imaging intensity along with distance information can greatly improve object recognition performance in advanced driver-assisted systems (ADAS) and autonomous driving applications. The subject matter described herein provides an imaging system that provides both ranging and intensity imaging information. Both the distance image and the intensity image are generated from the same source, so there is no image alignment issue and / or no complicated blending algorithm is needed. Embodiments of the pixels described herein are configured to be TCCs. Additionally, pixels configured to be time-to-digital converters (TDCs) may also be used, but may provide an image with a spatially lower resolution than pixels configured to be TCCs. That is, pixels configured to be TCCs are smaller and may provide pixel arrays with higher resolution than pixel arrays using pixels configured to be TDCs.

One exemplary embodiment of an imaging system capable of providing distance and light intensity information is the imaging system 15 shown in FIG. 1. Pixel array 42 includes pixel 43 shown in FIG. 5, pixel 700 shown in FIG. 7, pixel 1300 shown in FIG. 13, pixel 2100 shown in FIG. 12, and / or FIG. It may include embodiments of the TCC pixels described herein, such as pixel 2300 shown in 23. The light source 22 is controlled to provide a point scan that is synchronized with the pixel processing unit 46, as described in connection with FIGS. 3 and 4. The point scan may be repeated a number of times, providing a statistical mean for both distance information and light intensity information. Light intensity information can be used to determine reflectance of objects within the field of view of imaging system 15. In alternative embodiments, light source 22 may be controlled to illuminate the entire scene. When multiple light pulses are irradiated and captured, a histogram can be formed for each pixel, and by combining bins around the peak in the histogram, greatly improving object recognition performance in ADASs and autonomous driving applications. A gray scale image can be generated that can be used to generate the image. It should be noted that alternative embodiments may use pixels that are configured to provide a TDC output.

The reflectance of each point captured by the image sensor unit 24 may be determined based on the distance and gray scale value for each point. Gray scale images can also be created without laser pulses. By combining the multiple frames together, the image sensor unit 24 can operate like a Quanta image sensor that captures at most one photon per frame per pixel. When combining multiple bit planes (ie, frames) together, high dynamic-distance imaging can be achieved. By using both 3D and 2D images generated from the same image sensor unit for object recognition, complicated image mixing processes can be avoided, and the performance of recognition can be improved.

In one embodiment, the gray scale value for the pixel may be generated by directly using the peak value of the histogram of the arrival time (ie, detection time) of the photons detected by the pixel. The window width may be equal to the full width at half-maximum (FWHM) of the irradiated laser, light or pulse. A histogram of light-detection times can be formed, and a bin corresponding to the peak number of detected lights can be used to measure the surface reflectance s of the object at the point where the light pulse is reflected. Alternatively, the histogram for the pixel can be convolved with the trigger waveform output from the SPAD, and the maximum detected photon count can then be selected.

26A shows an exemplary trigger waveform 2600 from SPAD. The horizontal axis in FIG. 26 is relative time (unitless), and the vertical axis in FIG. 26 is relative amplitude (unitless). 26B shows an example histogram 2601 of photo-detection times of an example pixel that may be formed according to the subject matter described herein. The horizontal axis in FIG. 26B is the relative normalized time, and the vertical axis indicates the photon detection event count.

FIG. 26C shows an exemplary histogram 2602, where a window width representing the FWHM of the irradiated pulse (not shown) is indicated at 2602a to indicate a window in which an event count maximum can be determined. FIG. 26D shows an exemplary histogram 2603, and in the histogram 2603, a trigger waveform output from the SPAD (FIG. 26) is wound into a histogram to determine an event count maximum.

The surface reflectance S can be estimated starting from the following equation (5).

Where P is the pixel value; α is a system dependent constant that converts lux into pixel values; S is the expressive reflectance and L is the light intensity. Light intensity L is expressed as the sum of L _amb and L _laser and is the ambient light intensity and laser light intensity reaching the sensor, respectively.

Therefore, the surface reflectance S can be measured by Equation 6.

The ambient light intensity L _amb may be expressed by Equation 7.

Where N is the number of events detected from the light pulse, M is the number of events detected (ie from the periphery) before the light pulse is detected, D is the measured distance, and β can be corrected Is another system-dependent variable.

27 shows an example histogram 2700 for example pixels in accordance with the subject matter described herein. The horizontal axis of the histogram 2700 is relative time (unitless), and the vertical axis of the histogram 2700 is the count of photon detection events. The number M of events detected before the light pulse is detected is indicated at 2701. The number N of events detected within the light pulse is indicated at 2702.

The relationship between the emitting laser power (indicated by I _laser ) and the received laser power (L _laser ) may be:

는 시스템 상수이다.From here,

Is a system constant.

Subsequently, the measurement of the reflectance S at the '<x, y>' position may be expressed by Equation 9.

Here, f (D) is a distance dependent function, i.e., derived from equation (10).

28 shows a flowchart 2800 of an example method of generating depth or distance, maps, and gray scale images of a scene according to the subject matter described herein. The method starts at 2801. At 2802, the scene is point scanned by the imaging system 150 shown in FIG. 1, for example. The pixel array 42 of the imaging system 15 may include exemplary pixels 43 (FIG. 5), 700 (FIG. 7), 1300 (FIG. 13), 2100 (FIG. 21), and / or 2300 (FIG. 23). It may include. Although a point scan may be performed only once, it should be understood that better results may be obtained by repeating the point scan multiple times. At 2803, light detection events are accumulated for the pixels of pixel array 42. At 2804, a depth or distance map is generated as described herein. The distance information may be determined by the pixel processing unit 46 and / or the processor 19. At 2805, a gray scale image of the scene is generated based on the measurement of the reflectance of the scene. The gray scale image may be determined by the pixel processing unit 46 and / or the processor 19. The method ends at 2806.

29A shows an example scene 2900. 29B and 29C show example depth maps 2901 and example gray scale images 2902 of the scene shown in FIG. 29A, respectively, formed according to the subject matter described herein. The scale on the right side of FIG. 29B is meters.

30 shows an exemplary embodiment of the overall layout of the imaging system 15 shown in FIGS. 1 and 2 according to the subject matter described herein. Imaging module 17 includes 2D / 3D imaging and TOF measurements in accordance with advanced aspects of the present disclosure, including the necessary hardware shown in the exemplary embodiments of FIGS. 2, 5, 7 (or 13). Can be achieved. The processor 19 may be configured to connect with a plurality of external devices. In one embodiment, the imaging module 17 may function as an input device for providing data inputs to the processor 19 in the form of processed pixel outputs such as P1 and P2 of FIG. 12 for further processing. Processor 19 may also receive inputs from other input devices (not shown) that may be part of system 15. Some examples of such input devices include computer keyboards, touch pads, touch screens, joysticks, physical or virtual 'clickable buttons', and / or computer mouse / pointing devices. In FIG. 30, processor 19 is shown coupled with system memory 20, peripheral storage unit 275, one or more output devices 277, and network interface 278. In FIG. 30, the display unit is shown as an output device 277. In some embodiments, system 15 may include more than one instance of the depicted devices. Some examples of system 15 are computer systems (desktop or laptop), tablet computers, mobile devices, mobile phones, video gaming units or consoles, machine-to-machine (M2M) communication units, robots, automobiles, virtual reality devices, states Stateless thin client systems, dash-cam or rearview camera systems of vehicles, autonomous driving systems, or any other type of computing or data processing device. In various embodiments, all of the components shown in FIG. 30 can be housed in a single housing. Thus, system 15 may be configured as a standalone system or any other suitable form factor. In some embodiments, system 15 may be configured as a client system rather than a server system.

In some embodiments, system 15 may include more than one processor (eg, in a distributed processing configuration). When system 15 is a multiprocessor system, there may be more than one instance of processor 19, or there may be multiple processors coupled with processor 19 through respective interfaces (not shown). The processor 19 may be a system on chip (SoC) and / or may include more than one central processing unit (CPU).

The system memory 20 may be, for example, a semiconductor based storage system such as DRAM, SRAM, PRAM, RRAM, CBRAM, MRAM, STT-MRAM, or the like. In some embodiments, memory unit 20 may include at least one 3DS memory module along with one or more non-3DS memory modules. Non-3DS memory modules can be either double data rate or double data rate 2, 3 or 4 synchronous dynamic random access memory (DDR / DDR2 / DDR3 / DDR4 SDRAM), or Rambus DRAM, flash memory, and various forms of read-only memory (ROM). And the like. In addition, in some embodiments, system memory 20 may include a number of different types of semiconductor memories as opposed to a single type memory. In other embodiments, system memory 20 may be a non-temporary data storage medium.

Peripheral storage unit 275 may, in various embodiments, comprise hard drives, optical disks (such as compact disks (CDs) or digital versatile disks (DVDs)), nonvolatile random access memory (RAM) devices. And support for magnetic, optical, magneto-optical or solid state storage media. In some embodiments, peripheral storage unit 275 may be configured with more complex storage devices / systems, such as disk arrays (which may be in an appropriate Redundant Array of Independent Disks (RAID) configuration) or storage area networks (SANs). The peripheral storage unit 275 may include a Small Computer System Interface (SCSI) interface, a Fiber Channel interface, a FireWire® (IEEE 1394) interface, a Peripheral Component Interconnect Express (PCIe) ^TM standard-based interface, a USB ( Universal Serial Bus) may be coupled with the processor 19 via a standard peripheral interface, such as a protocol based interface, or other suitable interface. These various storage devices may be non-temporary data storage media.

The display unit 277 may be an example of an output device. Other examples of output autonomy include graphics / display devices, computer screens, warning systems, Computer Aided Design / Computer Aided Machining (CAD / CAM) systems, video game stations, smartphone display screens, car dashboard display screens, or other It includes an appropriate type of data output device. In some embodiments, input device (s) such as imaging module 17 and output device (s) such as display unit 277 may be coupled with processor 19 via I / O or peripheral interface (s). have.

In one embodiment, network interface 278 may communicate with processor 19 to allow system 15 to be coupled with a network (not shown). In other embodiments, there may be no network interface 278 at all. Network interface 278 may include suitable devices, media and / or protocol content for connecting the system to the network, wireless or wired. In various embodiments, the network may include local area networks (LANs), wide area networks (WANs), wired or wireless Ethernet, wireless communications networks, satellite links, or other suitable type of networks.

System 15 may include an on-board power supply unit 280 to supply electrical power to the various system components shown in FIG. 30. The power supply unit 280 may have batteries or may be connected to an AC electrical power outlet or an automotive based power outlet. In one embodiment, the power supply unit 280 may convert solar energy or other renewable energy into electrical power.

In one embodiment, the imaging module 17 may be integrated into a high speed interface that can be plugged into any personal computer (PC) or laptop, such as, for example, the Universal Serial Bus (USB) 2.0 or 3.0 interface above. have. For example, a non-temporary, computer-readable storage medium, such as system memory 20 or a peripheral data storage unit such as a CD / DVD, may store program code or software. The processor 19 and / or pixel processing unit 46 (FIG. 2) of the imaging module 17 may be configured to execute program code, whereby the system 15 may have been previously described with reference to FIGS. 1 to 29. 2D imaging (eg, a gray scale image of a 3D object), TOF and distance measurements, and the generation of a 3D image of the object using pixel-specific distance / range values, such as the operations discussed may be performed. For example, in certain embodiments, upon execution of program code, processor 19 and / or pixel processing unit 46 may be associated with, for example, row decoder / driver 125 and pixel column unit 128 of FIG. 12. Properly configure (or activate) the circuit components to apply appropriate input signals, such as a shutter signal, an RST signal, a SEL signal, to the pixels 43 of the pixel array 42 to capture light from the returned laser pulses. And subsequently process pixel outputs for pixel-specific values P1, P2 required for TOF and distance measurements. The program code or software, when executed by a suitable processing entity, such as processor 19 and / or pixel processing unit 46, may process the various pixel-specific ADC outputs (P1 and P2 values) It can be registration software or open source software that determines the values and presents the results in various forms, including displaying a 3D image of the distance object based on, for example, TOF based distance measurements. In certain embodiments, pixel processing unit 46 of imaging module 17 may perform some of the processing of pixel outputs before pixel output data is sent to processor 19 for further processing and display. In other embodiments, the processor 19 may also perform some or all of the functionality of the pixel processing unit 46, in which case the pixel processing unit 46 may not be part of the imaging module 17. .

As will be appreciated by those skilled in the art, the innovative ideas described herein may be modified and varied over a wide range of applications. Thus, the scope of the claimed subject matter should not be limited to any particular exemplary teaching discussed above, but instead is defined by the following claims.

15: System
17: imaging module
19: Processor module
20: memory module
22: Projector Module
24: image sensor unit
26: destination
33: laser
34: laser controller
35: projection device
42: 2D pixel array
46: pixel array control and processing circuits
501: SPAD cores
502: PPD core
503: multiple SPADs
504: first control circuit
505: incident light
507: second control circuit
508: PPD
700: pixels
701: electronic shutter signal
702: second control circuit
703-707: first to fifth transistors
1200: image sensor unit
1202: pixels
1203: row decoder / driver
1204: thermal decoder
1205: pixel column unit
1500: time-decomposition sensor
1501: SPAD circuit
1503: logic circuit
1505: PPD circuit
2000: time-decomposition sensor
2001: SPAD circuit
2003: logic circuit
2005: second PPD circuit
2300: time-decomposition sensor
2301a-2301n: one or more SPAD circuits
2303: logic circuit
2305: third PPD circuit

Claims (20)

In the image sensor:
A time-decomposition sensor comprising at least one pixel; In response to detecting by the at least one pixel one or more photons reflected from the object corresponding to a light pulse directed towards the object, the time-resolved sensor detects a pair of first and second signals. Output; A first ratio of the amplitude of the first signal of the pair to the sum of the amplitude of the first signal of the pair and the amplitude of the second signal is proportional to the flight time of the one or more detected photons; The second ratio of the amplitude of the first signal of the pair and the amplitude of the second signal of the pair to the amplitude of the second signal is proportional to the flight time of the one or more detected photons and ; And
And a processor to determine a surface reflectance of the object on which the light pulse is reflected, based on the pair of the first signal and the second signal. The method of claim 1,
And the processor further determines a distance to the object based on the phase of the first signal and the second signal. The method of claim 1,
In response to detecting one or more photons reflected from the object in the pixel for a plurality of light pulses directed toward the object, the time-resolved sensor includes a plurality of first and second signal pairs. Each pair of first and second signals corresponds to a respective light pulse, and
And the processor determines a surface reflectance of the object on which the plurality of light pulses are reflected, based on the plurality of first and second signal pairs. The method of claim 3,
In each of the plurality of pixels, in response to detecting one or more photons reflected from the object in response to each light pulse directed toward the object, the time-resolved sensor is configured to generate the plurality of first signals and the first signal. Output each of the pairs of two signals,
The processor further generates a gray scale image of the object based on the pairs of the plurality of first signals and the second signal. The method of claim 4, wherein
And the processor generates the gray scale image by generating at least one histogram of arrival times of photons detected by a predetermined pixel of the plurality of pixels. The method of claim 3,
The time-resolved sensor comprises a plurality of pixels,
At least one pixel of the plurality of pixels is:
At least one single-photon avalanche diode (SPAD); Each SPAD, in response to an active shutter signal, outputs a signal based on detecting one or more photons reflected from the object and incident on the SPAD;
Logic circuitry coupled with the output signal of the at least one SPAD; The logic circuit generates a first active signal and a second active signal; The first activation signal is activated in response to the start of the active shutter signal and deactivated in response to an output signal of the at least one SPAD; And the second activation signal is activated in response to the output signal of the at least one SPAD and deactivated in response to the termination of the active shutter signal; And
A differential time-to-charge converter (DTCC) circuit coupled with the first and second active signals,
The DTCC circuit is:
A capacitive device comprising a first terminal and a second terminal coupled with a ground voltage;
A first switching device having first, second and third terminals; The first terminal of the first switching device is coupled with the first terminal of the capacitive device; The second terminal of the first switching device is coupled with a first floating diffusion node; The third terminal of the first switching device is coupled with the first active signal; And the first switching device sends a first charge of the capacitive device to the first floating diffusion node in response to the first active signal;
A second switching device having first, second and third terminals; The first terminal of the second switching device is coupled with the first terminal of the capacitive device; The second terminal of the second switching device is coupled with a second floating diffusion node; The third terminal of the second switching device is coupled with the second active signal; And the second switching device sends the remaining charge of the capacitive device to the second floating diffusion node in response to the second active signal; And
An output circuit outputting the pair of the first signal and the second signal; The first signal comprises a first voltage based on the first charge of the first floating diffusion node; And the second signal comprises a second voltage based on the remaining charge of the second floating diffusion node. The method of claim 6,
Further comprising a drive signal that changes based on a ramp function, the drive signal starting to change until the end of the active shutter signal in response to a start time of the light pulse at which the one or more photons are detected. And the driving signal is connected to the third terminal of the first switching device if the first active signal is active and to the third terminal of the second switching device if the second activation signal is active. sensor. The method of claim 7, wherein
The first ratio of the first voltage to the sum of the first voltage and the second voltage is more proportional to the delay time from the flight time of the one or more photons, and the first voltage and the first voltage The second ratio of the second voltage to the sum of two voltages is more proportional to the delay time subtracted from the flight time of the one or more photons, the delay time beginning of the transmission time of the light pulse To a time from when the drive signal starts to change. The method of claim 6,
The capacitive device comprises a capacitor or a pinned diode. In the imaging unit:
A light source illuminating the object with a series of light pulses directed towards a representation of the object;
A time-decomposition sensor comprising at least one pixel; The time-resolved sensor is synchronized with the light source and in response to detecting one or more photons corresponding to the light pulses reflected from the surface of the object in the at least one pixel of the first and second signals. Output the pair; A first ratio of the amplitude of the first signal of the pair to the sum of the amplitude of the first signal of the pair and the amplitude of the second signal is proportional to the flight time of the one or more detected photons; The second ratio of the amplitude of the first signal of the pair and the amplitude of the second signal of the pair to the amplitude of the second signal is proportional to the flight time of the one or more detected photons and ; And
Determine a distance to the object based on the pair of the first signal and the second signal, and a surface of the object on which the light pulse is reflected based on the pair of the first signal and the second signal An imaging unit comprising a processor for determining a reflectance. The method of claim 10,
In response to detecting one or more photons reflected from the object at the pixel, the time-resolved sensor outputs a plurality of pairs of first and second signals; Each of said pairs of said first signal and said second signal corresponds to each of a plurality of light pulses directed toward said object, and
And the processor further determines a plurality of surface reflectances of the object based on the pair of corresponding first and second signals. The method of claim 11,
And the processor further generates a gray scale image of the object based on the plurality of surface reflectances. The method of claim 12,
And the processor generates the gray scale image by generating at least one histogram of arrival times of photons detected by a predetermined pixel of the plurality of pixels. The method of claim 11,
The time-resolved sensor further comprises a plurality of pixels, wherein at least one pixel of the plurality of pixels is:
At least one single-photon avalanche diode (SPAD); Each SPAD, in response to an active shutter signal, outputs a signal based on detecting one or more photons reflected from the object and incident on the SPAD;
Logic circuitry coupled with the output signal of the at least one SPAD; The logic circuit generates a first active signal and a second active signal; The first activation signal is activated in response to the start of the active shutter signal and deactivated in response to an output signal of the at least one SPAD; And the second activation signal is activated in response to the output signal of the at least one SPAD and deactivated in response to the termination of the active shutter signal; And
And a differential time-to-charge converter (DTCC) circuit coupled with the first and second active signals that output the first signal and the second signal. The method of claim 14,
The DTCC circuit is:
A capacitive device comprising a first terminal and a second terminal coupled with a ground voltage;
A first switching device having first, second and third terminals; The first terminal of the first switching device is coupled with the first terminal of the capacitive device; The second terminal of the first switching device is coupled with a first floating diffusion node; The third terminal of the first switching device is coupled with the first active signal; And the first switching device sends a first charge of the capacitive device to the first floating diffusion node in response to the first active signal; And
A second switching device having first, second and third terminals; The first terminal of the second switching device is coupled with the first terminal of the capacitive device; The second terminal of the second switching device is coupled with a second floating diffusion node; The third terminal of the second switching device is coupled with the second active signal; And the second switching device sends the remaining charge of the capacitive device to the second floating diffusion node in response to the second active signal;
The first signal includes a first voltage based on the first charge of the first floating diffusion node, and the second signal includes a second voltage based on the remaining charge of the second floating diffusion node. and,
The drive signal that changes according to the gradient function begins to change until the end of the active shutter signal in response to the start time of the light pulse at which the one or more photons are detected, and the drive signal is active if the first active signal is active. An imaging unit coupled to the third terminal of the first switching device and to the third terminal of the second switching device if the second active signal is active. The method of claim 15,
The capacitive device includes a capacitor or a pinned diode. In a method for generating a grayscale image of an object:
Irradiating a series of light pulses from the light source toward the surface of the object;
Detecting one or more photons in the pixel corresponding to a light pulse reflected from the surface of the object;
Generating, by a time-resolved sensor, a pair of first and second signals in response to detecting the one or more photons; The time-decomposition sensor is synchronized with the light source; A first ratio of the amplitude of the first signal of the pair to the sum of the amplitude of the first signal of the pair and the amplitude of the second signal is proportional to the flight time of the one or more detected photons; And a second ratio of the amplitude of the first signal of the pair and the amplitude of the second signal of the pair to the amplitude of the second signal is proportional to the flight time of the one or more detected photons and;
Determining, by a processor, a distance to the object based on the pair of the first signal and the second signal; And
Determining, by the processor, a surface reflectance of the object based on the pair of the first signal and the second signal. The method of claim 17,
Detecting one or more photons reflected from the object in the pixel for a plurality of light pulses directed toward the object; Each of the plurality of first and second signal pairs corresponds to each light pulse of the series of light pulses; And
Determining, by the processor, a surface reflectance of the object based on at least one of the pairs of the plurality of first and second signals. The method of claim 17,
Detecting one or more photons in each of the plurality of pixels; Each of the detected one or more photons is reflected from the object for a corresponding light pulse of the series of light pulses irradiated from the light source toward the surface of the object for each scan line;
Generating, by the time-resolved sensor, in response to detecting the one or more photons, a pair of first and second signals for each pixel; And
Generating, by the processor, a gray scale image of the object based on the pairs of the plurality of first and second signals. The method of claim 19,
Generating, by the processor, the gray scale image by generating at least one histogram of arrival times of photons detected by a predetermined pixel of the plurality of pixels.

Images