US20150271476A1 - Structured light imaging system - Google Patents
Structured light imaging system Download PDFInfo
- Publication number
- US20150271476A1 US20150271476A1 US14/729,245 US201514729245A US2015271476A1 US 20150271476 A1 US20150271476 A1 US 20150271476A1 US 201514729245 A US201514729245 A US 201514729245A US 2015271476 A1 US2015271476 A1 US 2015271476A1
- Authority
- US
- United States
- Prior art keywords
- storage element
- photodiode
- charge
- pulse
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 18
- 238000005286 illumination Methods 0.000 claims abstract description 38
- 238000012546 transfer Methods 0.000 claims description 19
- 238000000034 method Methods 0.000 claims description 13
- 238000006243 chemical reaction Methods 0.000 claims description 10
- 230000008878 coupling Effects 0.000 claims 2
- 238000010168 coupling process Methods 0.000 claims 2
- 238000005859 coupling reaction Methods 0.000 claims 2
- 238000010586 diagram Methods 0.000 description 32
- 238000009792 diffusion process Methods 0.000 description 16
- 239000004065 semiconductor Substances 0.000 description 13
- 230000001360 synchronised effect Effects 0.000 description 8
- 230000000875 corresponding effect Effects 0.000 description 7
- 238000012545 processing Methods 0.000 description 5
- 239000002184 metal Substances 0.000 description 4
- 230000015556 catabolic process Effects 0.000 description 3
- 230000001276 controlling effect Effects 0.000 description 3
- 238000006731 degradation reaction Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000007796 conventional method Methods 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 229910021420 polycrystalline silicon Inorganic materials 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000003750 conditioning effect Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
Images
Classifications
-
- H04N13/0296—
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
- G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/296—Synchronisation thereof; Control thereof
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
- G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- H04N13/0253—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/74—Circuitry for compensating brightness variation in the scene by influencing the scene brightness using illuminating means
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/70—SSIS architectures; Circuits associated therewith
- H04N25/71—Charge-coupled device [CCD] sensors; Charge-transfer registers specially adapted for CCD sensors
- H04N25/75—Circuitry for providing, modifying or processing image signals from the pixel array
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/70—SSIS architectures; Circuits associated therewith
- H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
- H04N25/77—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
- H04N25/771—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising storage means other than floating diffusion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/70—SSIS architectures; Circuits associated therewith
- H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
- H04N25/78—Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters
-
- H04N5/37452—
-
- H04N5/378—
Definitions
- the present invention is directed to imaging systems and, more particularly, structured light imaging systems.
- Structured light imaging systems are commonly used for three-dimensional (3D) imaging.
- a structured light imaging system has two main parts: an illumination source and a sensing device including an array of pixels.
- the illumination source projects one or more light patterns onto an object being imaged and the pixels within the sensing device detect light reflected by the object. The detected light is then processed by the sensing device to generate a representation of the object.
- FIG. 1 illustrates a typical four transistor (4T) pixel 50 utilized in a pixel array of an sensing device, such as a complementary metal-oxide-semiconductor (CMOS) sensing device.
- the pixel 50 includes a photosensor 52 (e.g., a photodiode), a storage node N configured as a floating diffusion (FD) region, transfer transistor 54 , reset transistor 56 , charge conversion transistor 58 configured as a source follower transistor, and row select transistor 60 .
- the photosensor 52 is connected to the storage node N by the transfer transistor 54 when the transfer transistor 54 is activated by a transfer control signal TX.
- the reset transistor 56 is connected between the storage node N and an array pixel supply voltage VAA.
- a reset control signal RESET is used to activate the reset transistor 56 , which resets the storage node N to a known state.
- the charge conversion transistor 58 has its gate connected to the storage node N and is connected between the array pixel supply voltage VAA and the row select transistor 60 .
- the charge conversion transistor 58 converts the charge stored at the storage node N into an electrical output signal.
- the row select transistor 60 is controllable by a row select signal ROW for selectively outputting the output signal OUT from the charge conversion transistor 58 .
- two output signals are conventionally generated, one being a reset signal Vrst generated after the storage node N is reset, the other being an image or photo signal Vsig generated after charges are transferred from the photosensor 52 to the storage node N.
- FIG. 2 illustrates a sensing device 200 that includes an array 230 of pixels (such as the pixel 50 illustrated in FIG. 1 ) and a timing and control circuit 232 .
- the timing and control circuit 232 provides timing and control signals for enabling the reading out of signals from pixels of the pixel array 230 in a manner commonly known to those skilled in the art.
- the pixel array 230 has dimensions of M rows by N columns of pixels, with the size of the pixel array 230 depending on its application.
- Signals from the sensing device 200 are typically read out a row at a time using a column parallel readout architecture.
- the timing and control circuit 232 selects a particular row of pixels in the pixel array 230 by controlling the operation of a row addressing circuit 234 and row drivers 240 .
- Signals stored in the selected row of pixels are provided to a readout circuit 242 in the manner described above.
- the signals are read twice from each of the columns and then read out sequentially or in parallel using a column addressing circuit 244 .
- the pixel signals (Vrst, Vsig) corresponding to the reset pixel signal and image pixel signal are provided as outputs of the readout circuit 242 , and are typically subtracted by a differential amplifier 260 in a correlated double sampling operation and the result digitized by an analog to digital converter 264 to provide a digital pixel signal.
- the digital pixel signals represent an image captured by pixel array 230 .
- the digital pixel signals are processed in an image processing circuit 268 to produce an output image.
- An imaging method in accordance with one embodiment generates a stream of light pulses, converts the stream after reflection by a scene to charge, stores charge converted during the light pulses to a first storage element, and stores charge converted between light pulses to a second storage element.
- a structured light imaging system in accordance with one embodiment includes an illumination source that generates a stream of light pulses and an image sensor.
- the image sensor includes a photodiode, first and second storage elements, first and second switches, and a controller that synchronizes the image sensor to the illumination source and actuates the first and second switches to couple the first storage element to the photodiode to store charge converted during the light pulses and to couple the second storage element to the photodiode to store charge converted between the light pulses.
- FIG. 1 is a circuit diagram illustrating a prior art four transistor pixel for use in a pixel array of an imaging device
- FIG. 2 is a block diagram of a prior art imaging device
- FIG. 3A is an illustration representing the generation, reflection, and capture of structured light in accordance with an aspect of the present invention.
- FIG. 3B is a block diagram of a multiple storage node pixel in accordance with an aspect of the present invention.
- FIG. 4A is an illustration partly in circuit diagram form of a two storage node pixel in accordance with an aspect of the present invention.
- FIG. 4B is an illustration partly in circuit diagram form of a four storage node pixel in accordance with an aspect of the present invention.
- FIG. 5 is a timing diagram illustrating exposure and readout for a four storage node pixel in accordance with an aspect of the present invention
- FIG. 6 is an illustration partly in circuit diagram form of a N storage node pixel in accordance with an aspect of the present invention.
- FIG. 7A is a circuit diagram of a two storage node pixel in accordance with an aspect of the present invention.
- FIG. 7B is a semiconductor layer diagram and corresponding energy diagram for one storage node of a two storage node pixel in accordance with an aspect of the present invention.
- FIG. 7C is a circuit diagram of a two storage node pixel and readout circuitry in accordance with an aspect of the present invention.
- FIG. 8A is a circuit diagram of another two storage node pixel in accordance with an aspect of the present invention.
- FIG. 8B is a semiconductor layer diagram and corresponding energy diagram for one storage node of another two storage node pixel in accordance with an aspect of the present invention.
- FIG. 9A is a timing diagram for the circuit FIGS. 7A and 7C ;
- FIG. 9B is a timing diagram for the circuit of FIG. 8A implemented using a readout structure such as illustrated in FIG. 7C ;
- FIG. 10 is a flow chart depicting the use of a two storage node register.
- FIG. 3A depicts an imaging system 300 in accordance with an embodiment of the present invention.
- Systems in accordance with aspects of the present invention are able to capture depth information of a scene using an improved structured light approach.
- the depicted imaging system 300 includes an illumination source 302 and a sensing device 304 synchronized with the illumination source 302 .
- the illustrated illumination source 302 is an infrared (IR) illumination source that generates structured light; however, other spectrums of light such as visible light may be employed. Suitable illumination sources for use with the present invention will be understood by one of skill in the art from the description herein.
- IR infrared
- the imaging system 300 generates a 3D representation of an object/3D scene 306 .
- the illumination source 302 generates pulsed structured light 303 , which is directed toward the object 306 .
- the object reflects portions of the pulsed structured light 303 as reflected structured light 307 .
- the sensing device 304 which is in sync with the illumination source 302 , captures the reflected light 307 and generates a 3D representation of the object.
- the illustrated sensing device 304 includes an array of pixels 308 and each pixel includes multiple storage elements, which are discussed in further detail below. Through the use of multiple storage elements, the present invention facilitates the capture and storage of rapidly changing scenes.
- FIG. 3B depicts a pixel 308 in accordance with an aspect of the present invention.
- the illustrated pixel 308 includes a photodiode 310 for converting light impinging on the photodiode to charge.
- the illustrated pixel 308 also includes two storage elements, a first storage node 312 and a second storage note 314 , for storing charge converted by the photodiode 310 .
- the illumination source 302 illustrated in FIG. 3A emits pulses of structured light 303 . As depicted in FIG.
- the pixels 308 being synchronized to the illumination source 302 , are able to continuously capture reflected pulses of structured light 307 —with the bright frames 307 a (light pulses) stored in one storage element 312 while the background frames 307 b (between light pulses) are stored in another storage element 314 . Interleaving the bright and background frames in time strongly suppresses any degradation in depth resolution due to varying ambient lighting conditions.
- the time-interleaved frames acquired by the pixel 308 can be subtracted, e.g., either at the pixel or column level, upon read out in the analog domain to further improve performance.
- the charge stored in the storage elements 312 / 314 may be combined in a floating diffusion region of a pixel output amplifier 316 , which will be described in further detail below.
- FIG. 4A is a conceptual diagram depicted operation of a pixel 400 in accordance with an example of the present invention.
- the illustrated pixel 400 includes a photodiode 402 , a first switch 404 that selectively connects the photodiode 402 to a first storage element 406 (e.g., to store charge developed by the photodiode 402 during one or more light pulses), and a second switch 408 that selectively connects the photodiode 402 to a second storage element 410 (e.g., to store charge developed by the photodiode 402 between one or more light pulses).
- the two switches 404 / 408 of the pixel 400 are controlled to transfer photo-generated charges to the appropriate storage regions 406 / 410 where the new charges are added to the charges that were previously stored.
- charge is accumulated in the photodiode 402 and then transferred to a storage element 406 / 410 (see, for example, FIGS. 7A and 7B and the related description below).
- photo-generated charge is directly streamed and accumulated in a storage region 406 / 410 (see, for example, FIGS. 8A and 8B and the related description below).
- Pixel 400 may be synchronized and the switches 404 / 408 may be actuated under control of a conventional timing and control circuit such as timing and control circuit 232 depicted in FIG. 2 . Suitable modifications to timing and control circuit 232 for use with pixel 400 and other pixels described herein will be understood by one of skill in the art from the description herein.
- FIG. 4B is a conceptual diagram depicting operation of another pixel 420 in accordance with an example of the present invention that enables projection and capture of multiple patterns, e.g., for more accurate depth measurement results.
- the illustrated pixel 420 includes a photodiode 422 and four switches 424 a - d that selectively connect the photodiode 422 to four respective storage elements 426 a - d .
- three patterns projected by an illumination source and the background can all be captured in an exposure.
- continuous acquisition can be performed in which readout and exposure are pipelined as shown in FIG. 5 .
- FIG. 6 is a conceptual diagram depicting another pixel 600 having a photodiode 602 and N storage elements 604 a -N that can be selectively coupled to the photodiode 602 through N switches 606 a -N.
- N- 1 projected patterns and the background can be captured. Further, since all the frames are interleaved in time, any degradation in depth resolution due to varying ambient lighting conditions is suppressed.
- FIG. 7A depicts a conceptual circuit diagram of a pixel 700 along with descriptive regions associated with semiconductor fabrication in accordance with aspects of the present invention.
- the pixel 700 includes a photodiode (PD) 702 .
- the pixel 700 also includes a first storage device/element (SDI) 704 and a second storage device/element (SD 2 ) 706 .
- the first storage element 704 is selectively coupled to the photodiode 702 through a first storage gate (SG 1 ) 708 and the second storage element 706 is selectively coupled to the photodiode 702 through a second storage gate (SG 2 ) 710 .
- first storage element 704 is selectively coupled to a first floating diffusion region (FD 1 ) 712 through a first transfer switch (TX 1 ) 714 and the second storage element 706 is selectively coupled to a second floating diffusion region (FD 2 ) 716 through a second transfer switch (TX 2 ) 718 .
- the first and second floating diffusion regions 712 / 716 may be coupled together in one embodiment.
- FIG. 7B depicts an example of a semiconductor structure for implementing one storage element of a pixel 700 along with energy diagrams depicting the flow of electrons based on signals applied to the pixel 700 , e.g., under the control of timing and control device 232 ( FIG. 1 ).
- the photodiode 702 may be fabricated as a p-n junction formed from a p+ region and an n ⁇ region within a semiconductor.
- the first storage element 704 may also be fabricated as a p-n junction formed from a p+ region and an n ⁇ region.
- a metal shield 720 is positioned over regions of the semiconductor other than the region including the photodiode 702 to prevent unwanted noise from being introduced.
- the first storage gate 708 and the first transfer switch 714 may be fabricated as poly-silicon or metal deposits on a dielectric layer on top of the semiconductor.
- the first floating diffusion region 712 may be fabricated as a n+ region.
- applying a high signal level to the storage gate 708 , signal SG, and a low signal level to the transfer switch, signal TX allows charge to flow from the photodiode 702 to the storage element 704 , but prevents the flow of charge to the floating diffusion region.
- Applying a low signal level to the storage gate 708 and a low signal level to the transfer switch isolates the charge stored in the storage element 704 .
- Applying a low signal level to the storage gate 708 and a high signal level to the transfer switch allows charge to flow from the storage element to the floating diffusion region, but prevents the flow of charge from the photodiode 702 to the storage element 704 .
- FIG. 7C depicts a circuit diagram 750 implementing the pixel 700 of FIG. 7A .
- pixel 700 includes a photodiode conditioning transistor 775 , a reset transistor 780 , and a charge conversion transistor 785 configured as a source follower transistor.
- the pixel 700 is coupled to conventional readout circuitry 752 .
- the illustrated readout circuitry includes a row select transistor 790 and a voltage source, V AA .
- the row select transistor 790 is controlled by a row signal, ROW, to produce an output signal, OUT, from a selected row.
- FIG. 9A depicts a timing diagram for controlling the operation of pixel 700 .
- the pixel 700 is synchronized to structured light pulses produced by an illumination device (e.g., to illumination device signal, ID).
- the first storage gate signal, SG 1 is in phase with the illumination device signal, ID, and is used to control the first storage gate 708 .
- the second storage gate signal, SG 2 is 180 degrees out of phase with the illumination device signal, ID, and is used to control the second storage gate 710 .
- the first and second storage gate signals, SG 1 and SG 2 cycle a plurality of times (e.g., 10 times) to accumulate charge developed by the photodiode during the illumination device pulses in the first element 704 and to accumulate charge between the illumination device pulses in the second element 706 .
- the appropriate number of cycles is dependent on the particular system and one of skill in the art will understand how to determine the appropriate number of cycles from the description herein.
- the stored charge is read out of the storage registers and transferred to the floating diffusion regions 712 / 716 of the pixel during a Frame Valid period.
- the charge is first read out of the first storage register 704 and, then, the charge is read out of the second storage register 706 .
- Charge is read out of the first storage register by first applying a high row selection signal, ROW, to the row selection transistor 790 during the read out period, applying a reset signal, RST, pulse to the reset transistor 780 (which creates a Sample Reset pulse), and applying a transfer signal, TX 1 , pulse to the first transfer resistor 714 .
- the readout circuitry 752 then reads the transferred charge during a Sample Signal period.
- a similar technique is applied to read out the charge stored in the second storage register.
- FIG. 8A depicts a conceptual circuit diagram of another pixel 800 along with descriptive regions associated with semiconductor fabrication in accordance with aspects of the present invention.
- the pixel 800 includes a photodiode (PD) 802 .
- the pixel 800 also includes a first storage device/element (VB 1 ) 804 and a second storage device/element (VB 2 ) 806 .
- the first storage element 804 is selectively coupled to the photodiode 802 through a first storage gate (SG 1 ) 808 and the second storage element 806 is selectively coupled to the photodiode 802 through a second storage gate (SG 2 ) 810 .
- first storage element 804 is coupled to a first floating diffusion region (FD 1 ) 812 through the structure of the first storage element 804 and the second storage element 806 is coupled to a second floating diffusion region (FD 2 ) 816 through the structure of second storage element 806 .
- the first and second floating diffusion regions 812 / 816 may be coupled together in one embodiment.
- FIG. 8B depicts an example of a semiconductor structure for implementing one storage element of a pixel 800 along with energy diagrams depicting the flow of electrons based on signals applied to the pixel 800 , e.g., under the control of timing and control device 232 ( FIG. 1 ).
- the photodiode 802 may be fabricated as a p-n junction formed from a p+ region and an n ⁇ region within a semiconductor.
- the first storage element 804 may be fabricated as an n ⁇ region.
- a metal shield 820 is positioned over regions of the semiconductor other than the region including the photodiode 802 to prevent unwanted noise from being introduced.
- the first storage gate 808 may be fabricated as poly-silicon or metal deposits on a dielectric layer on top of the semiconductor.
- the first floating diffusion region 812 may be fabricated as a n+ region.
- applying a high signal level to the storage gate 808 allows charge to flow from the photodiode 802 to the storage element 804 , but prevents the flow of charge to the floating diffusion region.
- Applying a mid-level signal to the storage gate 808 isolates the charge stored in the storage element 804 .
- Applying a low signal level to the storage gate 808 allows charge to flow from the storage element to the floating diffusion region, but prevents the flow of charge from the photodiode 802 to the storage element 804 .
- the pixel 800 may be implemented using a circuit diagram such as the circuit diagram 750 and readout circuitry 752 described above with reference to FIG. 7A . Suitable modifications to circuit diagram 750 and readout circuitry 752 for use with pixel 800 will be understood by one of skill in the art from the description herein.
- FIG. 9B depicts a timing diagram for controlling the operation of pixel 800 implemented using a suitable modified circuit diagram 750 and readout circuitry 752 .
- the pixel 800 is synchronized to structured light pulses produced by an illumination device (e.g., to illumination device signal, ID).
- the first storage gate signal, SG 1 is in phase with the illumination device signal, ID, and is used to control the first storage gate 808 .
- the second storage gate signal, SG 2 is 180 degrees out of phase with the illumination device signal, ID, and is used to control the second storage gate 810 .
- the first and second storage gate signals, SG 1 and SG 2 cycle a plurality of times (e.g., 10 times) to accumulate charge developed by the photodiode during the illumination device pulses in the first element 804 and to accumulate charge between the illumination device pulses in the second storage registers 806 .
- the appropriate number of cycles is dependent on the particular system and one of skill in the art will understand how to determine the appropriate number of cycles from the description herein.
- the stored charge is read out of the storage registers and transferred to the floating diffusion regions 812 / 816 of the pixel during a Frame Valid period.
- the charge is first read out of the first storage register 804 and, then, the charge is read out of the second storage register 806 .
- Charge is read out of the first storage register by first applying a high row selection signal, ROW, to the row selection transistor 790 during the read out period, applying a reset signal, RST, pulse to the reset transistor 780 (which creates a Sample Reset pulse), and then applying a low pulse to the storage signal, SG 1 .
- the readout circuitry 752 then reads the transferred charge during a Sample Signal period.
- a similar technique is applied to read out the charge stored in the second storage register.
- FIG. 10 is a flow chart 1000 depicting steps for storing charge in accordance with one embodiment of an imaging system.
- a stream of light pulses is generated.
- the stream of light pulses is generated by an illumination source that generates a periodic stream of light pulses.
- the periodic stream of light pulses may have a rate of 5 kilo-Hertz.
- the stream of light pulses may be directed toward an object that is being imaged.
- the stream of light pulses includes a single pattern of light.
- the stream of light pulses include two or more distinct patterns.
- the patterns of light may include a pattern of random points or a zebra-stripe like pattern.
- the source of the light pulses and an image sensor are synchronized.
- the image sensor is synchronized to the illumination source using conventional techniques that will be understood by one of skill in the art from the description herein.
- a reflection of the stream of light pulses is converted to charge.
- the stream of light pulses is reflected by an object that is being images/scanned and the reflected light is converted to charge by photodiodes within an array of pixels of the image sensor.
- the converted charge is routed to storage elements. If the converted charge corresponds to a light pulse, processing proceeds at block 1090 . If the converted charge corresponds to a period of time between light pulses, processing proceeds at block 1100 .
- the photodiode is coupled to the first storage element to transfer accumulated charge converted during the one or more pulses of light and is coupled to the second storage element to transfer accumulated charge converted between the two or more pulses of light.
- the charge converted by the photodiode is streamed to the first storage element during conversion of the one or more pulses of light and streamed by the photodiode to the second storage element between conversion of the two or more pulses of light.
- the converted charge corresponding to charge converted during one or more light pulses is stored in a first storage element of a pixel.
- the converted charge corresponding to charge converted between two or more light pulses is stored in a second storage element of a pixel.
- charge converted from a first pulse may be stored to the first storage element
- charge converted between the first pulse and a second pulse may be stored in a second storage element
- charge converted from the second pulse may be stored to a third storage element
- charge converted between the second pulse and a third pulse may be stored to a fourth storage element.
- charge may be stored in different storage elements based on the pattern of the structured light and/or to enable simultaneous conversion and readout of previously converted/stored charge.
- the stream of light pulses includes pulse streams having different patterns (e.g., a pulse having a first pattern, another pulse having a second pattern, and another pulse having a third pattern)
- the storing step during the one or more pulses of light may include storing charge converted from at least one pulse having the first pattern to the first storage element, storing charge converted from at least one pulse having the second pattern to a third storage element, and storing charge converted from at least one pulse having the third pattern to a fourth storage element.
- step 1110 the system checks if additional cycles of the pulsed light are to be stored during an exposure period. If additional cycles are to be stored, processing proceeds at decision step 1080 . If no more cycles are to be stored during the current exposure period, processing proceeds to step 1120 .
- charge is processed and read out of the storage elements.
- charge stored in the first and second storage elements may be read out during the storage of charge in the third and/or fourth storage elements.
- Charge converted between light pulses may be subtracted from charge converted during light pulses.
- the charge stored in a second storage element may be subtracted from charge stored in a first storage element, e.g., either at the pixel or column level, upon read out in the analog domain to further improve performance.
- charge in one storage element e.g., a storage element for storing charge between light pulses may be subtracted from multiple storage elements (e.g., storage elements associated with pulses from different light patters. For example, if there are four storage elements, charge stored in a second storage element may be subtracted from the charge stored in a first storage element, the charge stored in a third storage element, and the charge stored in a fourth storage element.
- distance information may be obtained from the stored charge.
- the distortion of a projected pattern as imaged by the sensor can be used for an exact geometric reconstruction of the surface shape.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Electromagnetism (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Transforming Light Signals Into Electric Signals (AREA)
- Solid State Image Pick-Up Elements (AREA)
Abstract
Description
- This application claims priority of U.S. Provisional Patent Application No. 61/479,029, filed Apr. 26, 2011, which is incorporated fully herein by reference.
- The present invention is directed to imaging systems and, more particularly, structured light imaging systems.
- Structured light imaging systems are commonly used for three-dimensional (3D) imaging. A structured light imaging system has two main parts: an illumination source and a sensing device including an array of pixels. The illumination source projects one or more light patterns onto an object being imaged and the pixels within the sensing device detect light reflected by the object. The detected light is then processed by the sensing device to generate a representation of the object.
-
FIG. 1 illustrates a typical four transistor (4T)pixel 50 utilized in a pixel array of an sensing device, such as a complementary metal-oxide-semiconductor (CMOS) sensing device. Thepixel 50 includes a photosensor 52 (e.g., a photodiode), a storage node N configured as a floating diffusion (FD) region,transfer transistor 54,reset transistor 56,charge conversion transistor 58 configured as a source follower transistor, and rowselect transistor 60. Thephotosensor 52 is connected to the storage node N by thetransfer transistor 54 when thetransfer transistor 54 is activated by a transfer control signal TX. Thereset transistor 56 is connected between the storage node N and an array pixel supply voltage VAA. A reset control signal RESET is used to activate thereset transistor 56, which resets the storage node N to a known state. - The
charge conversion transistor 58 has its gate connected to the storage node N and is connected between the array pixel supply voltage VAA and the row selecttransistor 60. Thecharge conversion transistor 58 converts the charge stored at the storage node N into an electrical output signal. Therow select transistor 60 is controllable by a row select signal ROW for selectively outputting the output signal OUT from thecharge conversion transistor 58. For eachpixel 50, two output signals are conventionally generated, one being a reset signal Vrst generated after the storage node N is reset, the other being an image or photo signal Vsig generated after charges are transferred from thephotosensor 52 to the storage node N. -
FIG. 2 illustrates asensing device 200 that includes anarray 230 of pixels (such as thepixel 50 illustrated inFIG. 1 ) and a timing andcontrol circuit 232. The timing andcontrol circuit 232 provides timing and control signals for enabling the reading out of signals from pixels of thepixel array 230 in a manner commonly known to those skilled in the art. Thepixel array 230 has dimensions of M rows by N columns of pixels, with the size of thepixel array 230 depending on its application. - Signals from the
sensing device 200 are typically read out a row at a time using a column parallel readout architecture. The timing andcontrol circuit 232 selects a particular row of pixels in thepixel array 230 by controlling the operation of arow addressing circuit 234 androw drivers 240. Signals stored in the selected row of pixels are provided to areadout circuit 242 in the manner described above. The signals are read twice from each of the columns and then read out sequentially or in parallel using acolumn addressing circuit 244. The pixel signals (Vrst, Vsig) corresponding to the reset pixel signal and image pixel signal are provided as outputs of thereadout circuit 242, and are typically subtracted by adifferential amplifier 260 in a correlated double sampling operation and the result digitized by an analog todigital converter 264 to provide a digital pixel signal. The digital pixel signals represent an image captured bypixel array 230. The digital pixel signals are processed in animage processing circuit 268 to produce an output image. - An imaging method in accordance with one embodiment generates a stream of light pulses, converts the stream after reflection by a scene to charge, stores charge converted during the light pulses to a first storage element, and stores charge converted between light pulses to a second storage element. A structured light imaging system in accordance with one embodiment includes an illumination source that generates a stream of light pulses and an image sensor. The image sensor includes a photodiode, first and second storage elements, first and second switches, and a controller that synchronizes the image sensor to the illumination source and actuates the first and second switches to couple the first storage element to the photodiode to store charge converted during the light pulses and to couple the second storage element to the photodiode to store charge converted between the light pulses.
- The invention is best understood from the following detailed description when read in connection with the accompanying drawings, with like elements having the same reference numerals. When a plurality of similar elements are present, a single reference numeral may be assigned to the plurality of similar elements with a small letter designation referring to specific elements. When referring to the elements collectively or to a non-specific one or more of the elements, the small letter designation may be dropped. The letter “n” may represent a non-specific number of elements. Also, lines without arrows connecting components may represent a bi-directional exchange between these components. This emphasizes that according to common practice, the various features of the drawings are not drawn to scale. On the contrary, the dimensions of the various features are arbitrarily expanded or reduced for clarity. Included in the drawings are the following figures:
-
FIG. 1 is a circuit diagram illustrating a prior art four transistor pixel for use in a pixel array of an imaging device; -
FIG. 2 is a block diagram of a prior art imaging device; -
FIG. 3A is an illustration representing the generation, reflection, and capture of structured light in accordance with an aspect of the present invention; -
FIG. 3B is a block diagram of a multiple storage node pixel in accordance with an aspect of the present invention; -
FIG. 4A is an illustration partly in circuit diagram form of a two storage node pixel in accordance with an aspect of the present invention; -
FIG. 4B is an illustration partly in circuit diagram form of a four storage node pixel in accordance with an aspect of the present invention; -
FIG. 5 is a timing diagram illustrating exposure and readout for a four storage node pixel in accordance with an aspect of the present invention; -
FIG. 6 is an illustration partly in circuit diagram form of a N storage node pixel in accordance with an aspect of the present invention; -
FIG. 7A is a circuit diagram of a two storage node pixel in accordance with an aspect of the present invention; -
FIG. 7B is a semiconductor layer diagram and corresponding energy diagram for one storage node of a two storage node pixel in accordance with an aspect of the present invention; -
FIG. 7C is a circuit diagram of a two storage node pixel and readout circuitry in accordance with an aspect of the present invention; -
FIG. 8A is a circuit diagram of another two storage node pixel in accordance with an aspect of the present invention; -
FIG. 8B is a semiconductor layer diagram and corresponding energy diagram for one storage node of another two storage node pixel in accordance with an aspect of the present invention; -
FIG. 9A is a timing diagram for the circuitFIGS. 7A and 7C ; -
FIG. 9B is a timing diagram for the circuit ofFIG. 8A implemented using a readout structure such as illustrated inFIG. 7C ; and -
FIG. 10 is a flow chart depicting the use of a two storage node register. -
FIG. 3A depicts animaging system 300 in accordance with an embodiment of the present invention. Systems in accordance with aspects of the present invention are able to capture depth information of a scene using an improved structured light approach. The depictedimaging system 300 includes anillumination source 302 and asensing device 304 synchronized with theillumination source 302. The illustratedillumination source 302 is an infrared (IR) illumination source that generates structured light; however, other spectrums of light such as visible light may be employed. Suitable illumination sources for use with the present invention will be understood by one of skill in the art from the description herein. - In accordance with one use, the
imaging system 300 generates a 3D representation of an object/3D scene 306. Theillumination source 302 generates pulsed structuredlight 303, which is directed toward theobject 306. The object reflects portions of the pulsed structured light 303 as reflectedstructured light 307. Thesensing device 304, which is in sync with theillumination source 302, captures the reflectedlight 307 and generates a 3D representation of the object. The illustratedsensing device 304 includes an array ofpixels 308 and each pixel includes multiple storage elements, which are discussed in further detail below. Through the use of multiple storage elements, the present invention facilitates the capture and storage of rapidly changing scenes. -
FIG. 3B depicts apixel 308 in accordance with an aspect of the present invention. The illustratedpixel 308 includes aphotodiode 310 for converting light impinging on the photodiode to charge. The illustratedpixel 308 also includes two storage elements, afirst storage node 312 and asecond storage note 314, for storing charge converted by thephotodiode 310. In contrast to conventional structured light systems, theillumination source 302 illustrated inFIG. 3A emits pulses ofstructured light 303. As depicted inFIG. 3B , thepixels 308, being synchronized to theillumination source 302, are able to continuously capture reflected pulses of structured light 307—with the bright frames 307 a (light pulses) stored in onestorage element 312 while the background frames 307 b (between light pulses) are stored in anotherstorage element 314. Interleaving the bright and background frames in time strongly suppresses any degradation in depth resolution due to varying ambient lighting conditions. The time-interleaved frames acquired by thepixel 308 can be subtracted, e.g., either at the pixel or column level, upon read out in the analog domain to further improve performance. The charge stored in thestorage elements 312/314 may be combined in a floating diffusion region of apixel output amplifier 316, which will be described in further detail below. -
FIG. 4A is a conceptual diagram depicted operation of apixel 400 in accordance with an example of the present invention. The illustratedpixel 400 includes aphotodiode 402, afirst switch 404 that selectively connects thephotodiode 402 to a first storage element 406 (e.g., to store charge developed by thephotodiode 402 during one or more light pulses), and asecond switch 408 that selectively connects thephotodiode 402 to a second storage element 410 (e.g., to store charge developed by thephotodiode 402 between one or more light pulses). Being synchronized with the illumination source 302 (FIG. 3A ), the twoswitches 404/408 of thepixel 400 are controlled to transfer photo-generated charges to theappropriate storage regions 406/410 where the new charges are added to the charges that were previously stored. - In one embodiment, charge is accumulated in the
photodiode 402 and then transferred to astorage element 406/410 (see, for example,FIGS. 7A and 7B and the related description below). In another embodiment, photo-generated charge is directly streamed and accumulated in astorage region 406/410 (see, for example,FIGS. 8A and 8B and the related description below).Pixel 400 may be synchronized and theswitches 404/408 may be actuated under control of a conventional timing and control circuit such as timing andcontrol circuit 232 depicted inFIG. 2 . Suitable modifications to timing andcontrol circuit 232 for use withpixel 400 and other pixels described herein will be understood by one of skill in the art from the description herein. -
FIG. 4B is a conceptual diagram depicting operation of anotherpixel 420 in accordance with an example of the present invention that enables projection and capture of multiple patterns, e.g., for more accurate depth measurement results. The illustratedpixel 420 includes aphotodiode 422 and four switches 424 a-d that selectively connect thephotodiode 422 to four respective storage elements 426 a-d. By utilizing apixel 420 with four storage elements 426 as shown inFIG. 4B , three patterns projected by an illumination source and the background can all be captured in an exposure. In another embodiment, if one pattern is used, continuous acquisition can be performed in which readout and exposure are pipelined as shown inFIG. 5 . -
FIG. 6 is a conceptual diagram depicting anotherpixel 600 having aphotodiode 602 and N storage elements 604 a-N that can be selectively coupled to thephotodiode 602 through N switches 606 a-N. In accordance with this embodiment, N-1 projected patterns and the background can be captured. Further, since all the frames are interleaved in time, any degradation in depth resolution due to varying ambient lighting conditions is suppressed. -
FIG. 7A depicts a conceptual circuit diagram of apixel 700 along with descriptive regions associated with semiconductor fabrication in accordance with aspects of the present invention. Thepixel 700 includes a photodiode (PD) 702. Thepixel 700 also includes a first storage device/element (SDI) 704 and a second storage device/element (SD2) 706. Thefirst storage element 704 is selectively coupled to thephotodiode 702 through a first storage gate (SG1) 708 and thesecond storage element 706 is selectively coupled to thephotodiode 702 through a second storage gate (SG2) 710. Additionally, thefirst storage element 704 is selectively coupled to a first floating diffusion region (FD1) 712 through a first transfer switch (TX1) 714 and thesecond storage element 706 is selectively coupled to a second floating diffusion region (FD2) 716 through a second transfer switch (TX2) 718. The first and second floatingdiffusion regions 712/716 may be coupled together in one embodiment. -
FIG. 7B depicts an example of a semiconductor structure for implementing one storage element of apixel 700 along with energy diagrams depicting the flow of electrons based on signals applied to thepixel 700, e.g., under the control of timing and control device 232 (FIG. 1 ). As depicted inFIG. 7B , thephotodiode 702 may be fabricated as a p-n junction formed from a p+ region and an n− region within a semiconductor. Thefirst storage element 704 may also be fabricated as a p-n junction formed from a p+ region and an n− region. Ametal shield 720 is positioned over regions of the semiconductor other than the region including thephotodiode 702 to prevent unwanted noise from being introduced. Thefirst storage gate 708 and thefirst transfer switch 714 may be fabricated as poly-silicon or metal deposits on a dielectric layer on top of the semiconductor. The first floatingdiffusion region 712 may be fabricated as a n+ region. - As indicated in the corresponding energy diagrams, applying a high signal level to the
storage gate 708, signal SG, and a low signal level to the transfer switch, signal TX, allows charge to flow from thephotodiode 702 to thestorage element 704, but prevents the flow of charge to the floating diffusion region. Applying a low signal level to thestorage gate 708 and a low signal level to the transfer switch isolates the charge stored in thestorage element 704. Applying a low signal level to thestorage gate 708 and a high signal level to the transfer switch allows charge to flow from the storage element to the floating diffusion region, but prevents the flow of charge from thephotodiode 702 to thestorage element 704. -
FIG. 7C depicts a circuit diagram 750 implementing thepixel 700 ofFIG. 7A . In addition to the pixel components discussed above,pixel 700 includes aphotodiode conditioning transistor 775, areset transistor 780, and acharge conversion transistor 785 configured as a source follower transistor. - The
pixel 700 is coupled toconventional readout circuitry 752. The illustrated readout circuitry includes a rowselect transistor 790 and a voltage source, VAA. The rowselect transistor 790 is controlled by a row signal, ROW, to produce an output signal, OUT, from a selected row. -
FIG. 9A depicts a timing diagram for controlling the operation ofpixel 700. Thepixel 700 is synchronized to structured light pulses produced by an illumination device (e.g., to illumination device signal, ID). The first storage gate signal, SG1, is in phase with the illumination device signal, ID, and is used to control thefirst storage gate 708. The second storage gate signal, SG2, is 180 degrees out of phase with the illumination device signal, ID, and is used to control thesecond storage gate 710. The first and second storage gate signals, SG1 and SG2, cycle a plurality of times (e.g., 10 times) to accumulate charge developed by the photodiode during the illumination device pulses in thefirst element 704 and to accumulate charge between the illumination device pulses in thesecond element 706. The appropriate number of cycles is dependent on the particular system and one of skill in the art will understand how to determine the appropriate number of cycles from the description herein. - After the charge is accumulated in the first and second storage registers 704 and 706, the stored charge is read out of the storage registers and transferred to the floating
diffusion regions 712/716 of the pixel during a Frame Valid period. In the illustrated embodiment, the charge is first read out of thefirst storage register 704 and, then, the charge is read out of thesecond storage register 706. Charge is read out of the first storage register by first applying a high row selection signal, ROW, to therow selection transistor 790 during the read out period, applying a reset signal, RST, pulse to the reset transistor 780 (which creates a Sample Reset pulse), and applying a transfer signal, TX1, pulse to thefirst transfer resistor 714. Thereadout circuitry 752 then reads the transferred charge during a Sample Signal period. A similar technique is applied to read out the charge stored in the second storage register. -
FIG. 8A depicts a conceptual circuit diagram of anotherpixel 800 along with descriptive regions associated with semiconductor fabrication in accordance with aspects of the present invention. Thepixel 800 includes a photodiode (PD) 802. Thepixel 800 also includes a first storage device/element (VB1) 804 and a second storage device/element (VB2) 806. Thefirst storage element 804 is selectively coupled to thephotodiode 802 through a first storage gate (SG1) 808 and thesecond storage element 806 is selectively coupled to thephotodiode 802 through a second storage gate (SG2) 810. Additionally, thefirst storage element 804 is coupled to a first floating diffusion region (FD1) 812 through the structure of thefirst storage element 804 and thesecond storage element 806 is coupled to a second floating diffusion region (FD2) 816 through the structure ofsecond storage element 806. The first and second floatingdiffusion regions 812/816 may be coupled together in one embodiment. -
FIG. 8B depicts an example of a semiconductor structure for implementing one storage element of apixel 800 along with energy diagrams depicting the flow of electrons based on signals applied to thepixel 800, e.g., under the control of timing and control device 232 (FIG. 1 ). As depicted inFIG. 8B , thephotodiode 802 may be fabricated as a p-n junction formed from a p+ region and an n− region within a semiconductor. Thefirst storage element 804 may be fabricated as an n− region. Ametal shield 820 is positioned over regions of the semiconductor other than the region including thephotodiode 802 to prevent unwanted noise from being introduced. Thefirst storage gate 808 may be fabricated as poly-silicon or metal deposits on a dielectric layer on top of the semiconductor. The first floatingdiffusion region 812 may be fabricated as a n+ region. - As indicated in the corresponding energy diagrams, applying a high signal level to the
storage gate 808, signal SG, allows charge to flow from thephotodiode 802 to thestorage element 804, but prevents the flow of charge to the floating diffusion region. Applying a mid-level signal to thestorage gate 808 isolates the charge stored in thestorage element 804. Applying a low signal level to thestorage gate 808 allows charge to flow from the storage element to the floating diffusion region, but prevents the flow of charge from thephotodiode 802 to thestorage element 804. - The
pixel 800 may be implemented using a circuit diagram such as the circuit diagram 750 andreadout circuitry 752 described above with reference toFIG. 7A . Suitable modifications to circuit diagram 750 andreadout circuitry 752 for use withpixel 800 will be understood by one of skill in the art from the description herein. -
FIG. 9B depicts a timing diagram for controlling the operation ofpixel 800 implemented using a suitable modified circuit diagram 750 andreadout circuitry 752. Thepixel 800 is synchronized to structured light pulses produced by an illumination device (e.g., to illumination device signal, ID). The first storage gate signal, SG1, is in phase with the illumination device signal, ID, and is used to control thefirst storage gate 808. The second storage gate signal, SG2, is 180 degrees out of phase with the illumination device signal, ID, and is used to control thesecond storage gate 810. The first and second storage gate signals, SG1 and SG2, cycle a plurality of times (e.g., 10 times) to accumulate charge developed by the photodiode during the illumination device pulses in thefirst element 804 and to accumulate charge between the illumination device pulses in the second storage registers 806. The appropriate number of cycles is dependent on the particular system and one of skill in the art will understand how to determine the appropriate number of cycles from the description herein. - After the charge is accumulated in the first and second storage registers 804 and 806, the stored charge is read out of the storage registers and transferred to the floating
diffusion regions 812/816 of the pixel during a Frame Valid period. In the illustrated embodiment, the charge is first read out of thefirst storage register 804 and, then, the charge is read out of thesecond storage register 806. Charge is read out of the first storage register by first applying a high row selection signal, ROW, to therow selection transistor 790 during the read out period, applying a reset signal, RST, pulse to the reset transistor 780 (which creates a Sample Reset pulse), and then applying a low pulse to the storage signal, SG1. Thereadout circuitry 752 then reads the transferred charge during a Sample Signal period. A similar technique is applied to read out the charge stored in the second storage register. -
FIG. 10 is aflow chart 1000 depicting steps for storing charge in accordance with one embodiment of an imaging system. - At
step 1020, a stream of light pulses is generated. In one embodiment, the stream of light pulses is generated by an illumination source that generates a periodic stream of light pulses. The periodic stream of light pulses may have a rate of 5 kilo-Hertz. The stream of light pulses may be directed toward an object that is being imaged. In one embodiment, the stream of light pulses includes a single pattern of light. In other embodiments, the stream of light pulses include two or more distinct patterns. The patterns of light may include a pattern of random points or a zebra-stripe like pattern. - At
step 1040, the source of the light pulses and an image sensor are synchronized. In one embodiment, the image sensor is synchronized to the illumination source using conventional techniques that will be understood by one of skill in the art from the description herein. - At
step 1060, a reflection of the stream of light pulses is converted to charge. In one embodiment the stream of light pulses is reflected by an object that is being images/scanned and the reflected light is converted to charge by photodiodes within an array of pixels of the image sensor. - At
step 1080, the converted charge is routed to storage elements. If the converted charge corresponds to a light pulse, processing proceeds atblock 1090. If the converted charge corresponds to a period of time between light pulses, processing proceeds atblock 1100. In one embodiment, the photodiode is coupled to the first storage element to transfer accumulated charge converted during the one or more pulses of light and is coupled to the second storage element to transfer accumulated charge converted between the two or more pulses of light. In another embodiment, the charge converted by the photodiode is streamed to the first storage element during conversion of the one or more pulses of light and streamed by the photodiode to the second storage element between conversion of the two or more pulses of light. - At
step 1090, the converted charge corresponding to charge converted during one or more light pulses is stored in a first storage element of a pixel. Atstep 1100, the converted charge corresponding to charge converted between two or more light pulses is stored in a second storage element of a pixel. For example, charge converted from a first pulse may be stored to the first storage element, charge converted between the first pulse and a second pulse may be stored in a second storage element, charge converted from the second pulse may be stored to a third storage element, and charge converted between the second pulse and a third pulse may be stored to a fourth storage element. - In embodiments where each pixel includes more than two storage element, charge may be stored in different storage elements based on the pattern of the structured light and/or to enable simultaneous conversion and readout of previously converted/stored charge. For example, where the stream of light pulses includes pulse streams having different patterns (e.g., a pulse having a first pattern, another pulse having a second pattern, and another pulse having a third pattern), the storing step during the one or more pulses of light may include storing charge converted from at least one pulse having the first pattern to the first storage element, storing charge converted from at least one pulse having the second pattern to a third storage element, and storing charge converted from at least one pulse having the third pattern to a fourth storage element.
- At
step 1110, the system checks if additional cycles of the pulsed light are to be stored during an exposure period. If additional cycles are to be stored, processing proceeds atdecision step 1080. If no more cycles are to be stored during the current exposure period, processing proceeds to step 1120. - At
step 1120, charge is processed and read out of the storage elements. In an embodiment including four storage elements, charge stored in the first and second storage elements may be read out during the storage of charge in the third and/or fourth storage elements. Charge converted between light pulses may be subtracted from charge converted during light pulses. For example, the charge stored in a second storage element may be subtracted from charge stored in a first storage element, e.g., either at the pixel or column level, upon read out in the analog domain to further improve performance. In embodiments where each pixel includes additional storage element, charge in one storage element (e.g., a storage element for storing charge between light pulses may be subtracted from multiple storage elements (e.g., storage elements associated with pulses from different light patters. For example, if there are four storage elements, charge stored in a second storage element may be subtracted from the charge stored in a first storage element, the charge stored in a third storage element, and the charge stored in a fourth storage element. - Using conventional techniques that will be understood by one of skill in the art from the description herein, distance information may be obtained from the stored charge. In one example, the distortion of a projected pattern as imaged by the sensor can be used for an exact geometric reconstruction of the surface shape. By interleaving the captures of light and background frames and subtracting one from the other, the invention herein prevents degradation in depth resolution by rejecting the effects due to varying ambient illumination.
- Although the invention is illustrated and described herein with reference to specific embodiments, the invention is not intended to be limited to the details shown. Rather, various modifications may be made in the details without departing from the invention.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/729,245 US20150271476A1 (en) | 2011-04-26 | 2015-06-03 | Structured light imaging system |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201161479029P | 2011-04-26 | 2011-04-26 | |
US13/353,593 US9083905B2 (en) | 2011-04-26 | 2012-01-19 | Structured light imaging system |
US14/729,245 US20150271476A1 (en) | 2011-04-26 | 2015-06-03 | Structured light imaging system |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/353,593 Continuation US9083905B2 (en) | 2011-04-26 | 2012-01-19 | Structured light imaging system |
Publications (1)
Publication Number | Publication Date |
---|---|
US20150271476A1 true US20150271476A1 (en) | 2015-09-24 |
Family
ID=46049233
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/353,593 Active 2034-01-13 US9083905B2 (en) | 2011-04-26 | 2012-01-19 | Structured light imaging system |
US14/729,245 Abandoned US20150271476A1 (en) | 2011-04-26 | 2015-06-03 | Structured light imaging system |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/353,593 Active 2034-01-13 US9083905B2 (en) | 2011-04-26 | 2012-01-19 | Structured light imaging system |
Country Status (2)
Country | Link |
---|---|
US (2) | US9083905B2 (en) |
EP (1) | EP2519001B1 (en) |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3470773A4 (en) * | 2017-08-14 | 2019-04-17 | Shenzhen Goodix Technology Co., Ltd. | Three-dimensional image system, and electronic device |
US10545275B1 (en) | 2018-07-16 | 2020-01-28 | Shenzhen Guangjian Technology Co., Ltd. | Light projecting method and device |
EP3614174A1 (en) * | 2018-08-21 | 2020-02-26 | Omron Corporation | Distance measuring device and distance measuring method |
US10585194B1 (en) | 2019-01-15 | 2020-03-10 | Shenzhen Guangjian Technology Co., Ltd. | Switchable diffuser projection systems and methods |
US10585173B1 (en) * | 2019-01-15 | 2020-03-10 | Shenzhen Guangjian Technology Co., Ltd. | Systems and methods for enhanced ToF resolution |
US10641942B2 (en) | 2018-07-16 | 2020-05-05 | Shenzhen Guangjian Technology Co., Ltd. | Light projecting method and device |
US10690752B2 (en) | 2018-07-16 | 2020-06-23 | Shenzhen Guangjian Technology Co., Ltd. | Light projecting method and device |
US10690846B2 (en) | 2018-10-24 | 2020-06-23 | Shenzhen Guangjian Technology Co., Ltd. | Light projecting method and device |
US10931860B2 (en) | 2019-01-17 | 2021-02-23 | Shenzhen Guangjian Technology Co., Ltd. | Display device and electronic apparatus with 3D camera module |
US11438486B2 (en) * | 2019-08-26 | 2022-09-06 | Qualcomm Incorporated | 3D active depth sensing with laser pulse train bursts and a gated sensor |
WO2023238913A1 (en) * | 2022-06-08 | 2023-12-14 | 凸版印刷株式会社 | Distance image capturing device, and distance image capturing method |
Families Citing this family (40)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7057256B2 (en) | 2001-05-25 | 2006-06-06 | President & Fellows Of Harvard College | Silicon-based visible and near-infrared optoelectric devices |
US7442629B2 (en) | 2004-09-24 | 2008-10-28 | President & Fellows Of Harvard College | Femtosecond laser-induced formation of submicrometer spikes on a semiconductor substrate |
US9911781B2 (en) | 2009-09-17 | 2018-03-06 | Sionyx, Llc | Photosensitive imaging devices and associated methods |
US9673243B2 (en) | 2009-09-17 | 2017-06-06 | Sionyx, Llc | Photosensitive imaging devices and associated methods |
US8692198B2 (en) * | 2010-04-21 | 2014-04-08 | Sionyx, Inc. | Photosensitive imaging devices and associated methods |
WO2011160130A2 (en) | 2010-06-18 | 2011-12-22 | Sionyx, Inc | High speed photosensitive devices and associated methods |
US9496308B2 (en) | 2011-06-09 | 2016-11-15 | Sionyx, Llc | Process module for increasing the response of backside illuminated photosensitive imagers and associated methods |
CN103946867A (en) | 2011-07-13 | 2014-07-23 | 西奥尼克斯公司 | Biometric imaging devices and associated methods |
US20130027596A1 (en) * | 2011-07-27 | 2013-01-31 | Chung Chun Wan | Color imaging using time-multiplexed light sources and monochrome image sensors with multi-storage-node pixels |
US9118883B2 (en) | 2011-11-28 | 2015-08-25 | Semiconductor Components Industries, Llc | High dynamic range imaging with multi-storage pixels |
US9064764B2 (en) | 2012-03-22 | 2015-06-23 | Sionyx, Inc. | Pixel isolation elements, devices, and associated methods |
US9918017B2 (en) | 2012-09-04 | 2018-03-13 | Duelight Llc | Image sensor apparatus and method for obtaining multiple exposures with zero interframe time |
US9531961B2 (en) | 2015-05-01 | 2016-12-27 | Duelight Llc | Systems and methods for generating a digital image using separate color and intensity data |
US10558848B2 (en) | 2017-10-05 | 2020-02-11 | Duelight Llc | System, method, and computer program for capturing an image with correct skin tone exposure |
US9819849B1 (en) | 2016-07-01 | 2017-11-14 | Duelight Llc | Systems and methods for capturing digital images |
US9807322B2 (en) | 2013-03-15 | 2017-10-31 | Duelight Llc | Systems and methods for a digital image sensor |
GB2511868B (en) * | 2013-03-15 | 2020-07-15 | Tobii Ab | Eye/gaze tracker and method of tracking the position of an eye and/or a gaze point of a subject |
WO2014151093A1 (en) | 2013-03-15 | 2014-09-25 | Sionyx, Inc. | Three dimensional imaging utilizing stacked imager devices and associated methods |
US9209345B2 (en) | 2013-06-29 | 2015-12-08 | Sionyx, Inc. | Shallow trench textured regions and associated methods |
EP2887009A1 (en) * | 2013-12-23 | 2015-06-24 | Universität Zürich | Method for reconstructing a surface using spatially structured light and a dynamic vision sensor |
WO2015132726A1 (en) * | 2014-03-03 | 2015-09-11 | Photoneo S.R.O | Methods and apparatus for superpixel modulation |
CN106662433B (en) * | 2014-06-27 | 2019-09-06 | 新加坡恒立私人有限公司 | Structured light imaging system and method |
US10924688B2 (en) | 2014-11-06 | 2021-02-16 | Duelight Llc | Image sensor apparatus and method for obtaining low-noise, high-speed captures of a photographic scene |
US11463630B2 (en) | 2014-11-07 | 2022-10-04 | Duelight Llc | Systems and methods for generating a high-dynamic range (HDR) pixel stream |
US10313600B2 (en) * | 2015-10-13 | 2019-06-04 | Canon Kabushiki Kaisha | Imaging device capable of simultaneously capturing of a motion image and a static image and imaging method |
WO2017103781A1 (en) * | 2015-12-13 | 2017-06-22 | Photoneo S.R.O. | Methods and apparatus for superpixel modulation with ambient light suppression |
US10958885B2 (en) * | 2016-08-26 | 2021-03-23 | Mems Start, Llc | Filtering imaging system including a light source to output an optical signal modulated with a code |
US10557925B2 (en) * | 2016-08-26 | 2020-02-11 | Samsung Electronics Co., Ltd. | Time-of-flight (TOF) image sensor using amplitude modulation for range measurement |
CN109792478B (en) | 2016-09-01 | 2021-11-12 | 迪尤莱特公司 | Apparatus and method for adjusting focus based on focus target information |
US10582178B2 (en) * | 2016-11-02 | 2020-03-03 | Omnivision Technologies, Inc. | Systems and methods for active depth imager with background subtract |
US10306172B2 (en) * | 2017-09-08 | 2019-05-28 | Microsoft Technology Licensing, Llc | Time-of-flight sensor readout circuit |
US10458783B2 (en) | 2017-10-13 | 2019-10-29 | Faro Technologies, Inc. | Three-dimensional scanner having pixel memory |
US10598936B1 (en) * | 2018-04-23 | 2020-03-24 | Facebook Technologies, Llc | Multi-mode active pixel sensor |
US11067673B2 (en) * | 2018-05-17 | 2021-07-20 | Cmos Sensor, Inc. | Sensor, apparatus for lidar application and method of background cancellation |
US11054546B2 (en) | 2018-07-16 | 2021-07-06 | Faro Technologies, Inc. | Laser scanner with enhanced dymanic range imaging |
KR102615195B1 (en) * | 2018-07-19 | 2023-12-18 | 삼성전자주식회사 | 3D(dimension) image sensor based on ToF(Time of Flight), and electronic apparatus comprising the image sensor |
US10791282B2 (en) | 2018-12-13 | 2020-09-29 | Fenwick & West LLP | High dynamic range camera assembly with augmented pixels |
US10855896B1 (en) * | 2018-12-13 | 2020-12-01 | Facebook Technologies, Llc | Depth determination using time-of-flight and camera assembly with augmented pixels |
US10902623B1 (en) | 2019-11-19 | 2021-01-26 | Facebook Technologies, Llc | Three-dimensional imaging with spatial and temporal coding for depth camera assembly |
US11194160B1 (en) | 2020-01-21 | 2021-12-07 | Facebook Technologies, Llc | High frame rate reconstruction with N-tap camera sensor |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6522395B1 (en) * | 1999-04-30 | 2003-02-18 | Canesta, Inc. | Noise reduction techniques suitable for three-dimensional information acquirable with CMOS-compatible image sensor ICS |
US20080231737A1 (en) * | 2001-06-05 | 2008-09-25 | Weale Gareth P | Dual storage node pixel for CMOS sensor |
US20100039546A1 (en) * | 2008-07-17 | 2010-02-18 | Microsoft International Holdings B.V. | Cmos photogate 3d camera system having improved charge sensing cell and pixel geometry |
US20100265316A1 (en) * | 2009-04-16 | 2010-10-21 | Primesense Ltd. | Three-dimensional mapping and imaging |
US20110085043A1 (en) * | 2009-10-08 | 2011-04-14 | Honda Motor Co., Ltd. | Image capturing apparatus, image capturing system, and processing method |
US20120087573A1 (en) * | 2010-10-11 | 2012-04-12 | Vinay Sharma | Eliminating Clutter in Video Using Depth Information |
US20120154535A1 (en) * | 2010-12-15 | 2012-06-21 | Microsoft Corporation | Capturing gated and ungated light in the same frame on the same photosurface |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000292135A (en) * | 1999-04-07 | 2000-10-20 | Minolta Co Ltd | Three-dimensional information input camera |
US6906302B2 (en) * | 2002-07-30 | 2005-06-14 | Freescale Semiconductor, Inc. | Photodetector circuit device and method thereof |
WO2004017088A2 (en) * | 2002-08-15 | 2004-02-26 | Lc Technologies, Inc. | Motion clutter suppression for image-subtracting cameras |
JP4495449B2 (en) * | 2003-12-22 | 2010-07-07 | Hoya株式会社 | Lighting control device |
JP5395323B2 (en) * | 2006-09-29 | 2014-01-22 | ブレインビジョン株式会社 | Solid-state image sensor |
US8659698B2 (en) | 2007-05-17 | 2014-02-25 | Ilya Blayvas | Compact 3D scanner with fixed pattern projector and dual band image sensor |
US8730382B2 (en) * | 2008-06-04 | 2014-05-20 | Honda Motor Co., Ltd. | Charge accumulating and splitting imaging device |
DE102009037596B4 (en) * | 2009-08-14 | 2014-07-24 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Pixel structure, system and method for optical distance measurement and control circuit for the pixel structure |
US8654152B2 (en) * | 2010-06-21 | 2014-02-18 | Microsoft Corporation | Compartmentalizing focus area within field of view |
-
2012
- 2012-01-19 US US13/353,593 patent/US9083905B2/en active Active
- 2012-04-25 EP EP12165535.1A patent/EP2519001B1/en active Active
-
2015
- 2015-06-03 US US14/729,245 patent/US20150271476A1/en not_active Abandoned
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6522395B1 (en) * | 1999-04-30 | 2003-02-18 | Canesta, Inc. | Noise reduction techniques suitable for three-dimensional information acquirable with CMOS-compatible image sensor ICS |
US20080231737A1 (en) * | 2001-06-05 | 2008-09-25 | Weale Gareth P | Dual storage node pixel for CMOS sensor |
US20100039546A1 (en) * | 2008-07-17 | 2010-02-18 | Microsoft International Holdings B.V. | Cmos photogate 3d camera system having improved charge sensing cell and pixel geometry |
US20100265316A1 (en) * | 2009-04-16 | 2010-10-21 | Primesense Ltd. | Three-dimensional mapping and imaging |
US20110085043A1 (en) * | 2009-10-08 | 2011-04-14 | Honda Motor Co., Ltd. | Image capturing apparatus, image capturing system, and processing method |
US20120087573A1 (en) * | 2010-10-11 | 2012-04-12 | Vinay Sharma | Eliminating Clutter in Video Using Depth Information |
US20120154535A1 (en) * | 2010-12-15 | 2012-06-21 | Microsoft Corporation | Capturing gated and ungated light in the same frame on the same photosurface |
Cited By (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10659765B2 (en) * | 2017-08-14 | 2020-05-19 | Shenzhen GOODIX Technology Co., Ltd. | Three-dimensional (3D) image system and electronic device |
EP3470773A4 (en) * | 2017-08-14 | 2019-04-17 | Shenzhen Goodix Technology Co., Ltd. | Three-dimensional image system, and electronic device |
US11460547B2 (en) | 2018-07-16 | 2022-10-04 | Shenzhen Guangjian Technology Co., Ltd. | Light projecting method and device |
US10545275B1 (en) | 2018-07-16 | 2020-01-28 | Shenzhen Guangjian Technology Co., Ltd. | Light projecting method and device |
US11914073B2 (en) * | 2018-07-16 | 2024-02-27 | Shenzhen Guangjian Technology Co., Ltd. | Light projecting method and device |
US10641942B2 (en) | 2018-07-16 | 2020-05-05 | Shenzhen Guangjian Technology Co., Ltd. | Light projecting method and device |
US10690752B2 (en) | 2018-07-16 | 2020-06-23 | Shenzhen Guangjian Technology Co., Ltd. | Light projecting method and device |
US11592607B2 (en) | 2018-07-16 | 2023-02-28 | Shenzhen Guangjian Technology Co., Ltd. | Light projecting method and device |
EP3614174A1 (en) * | 2018-08-21 | 2020-02-26 | Omron Corporation | Distance measuring device and distance measuring method |
US11762070B2 (en) | 2018-08-21 | 2023-09-19 | Omron Corporation | Distance measuring device and distance measuring method |
US10690846B2 (en) | 2018-10-24 | 2020-06-23 | Shenzhen Guangjian Technology Co., Ltd. | Light projecting method and device |
US11422262B2 (en) | 2019-01-15 | 2022-08-23 | Shenzhen Guangjian Technology Co., Ltd. | Switchable diffuser projection systems and methods |
US10585173B1 (en) * | 2019-01-15 | 2020-03-10 | Shenzhen Guangjian Technology Co., Ltd. | Systems and methods for enhanced ToF resolution |
US10585194B1 (en) | 2019-01-15 | 2020-03-10 | Shenzhen Guangjian Technology Co., Ltd. | Switchable diffuser projection systems and methods |
US10931860B2 (en) | 2019-01-17 | 2021-02-23 | Shenzhen Guangjian Technology Co., Ltd. | Display device and electronic apparatus with 3D camera module |
US11438486B2 (en) * | 2019-08-26 | 2022-09-06 | Qualcomm Incorporated | 3D active depth sensing with laser pulse train bursts and a gated sensor |
WO2023238913A1 (en) * | 2022-06-08 | 2023-12-14 | 凸版印刷株式会社 | Distance image capturing device, and distance image capturing method |
Also Published As
Publication number | Publication date |
---|---|
US9083905B2 (en) | 2015-07-14 |
EP2519001A3 (en) | 2016-12-07 |
EP2519001A2 (en) | 2012-10-31 |
EP2519001B1 (en) | 2019-06-26 |
US20120274744A1 (en) | 2012-11-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9083905B2 (en) | Structured light imaging system | |
US10638064B2 (en) | Methods and apparatus for true high dynamic range (THDR) time-delay-and-integrate (TDI) imaging | |
JP6374690B2 (en) | Imaging apparatus, control method therefor, program, and storage medium | |
CN105144699B (en) | The imaging sensor and its operating method of the resetting of having ready conditions of threshold monitor | |
Cho et al. | A 3-D camera with adaptable background light suppression using pixel-binning and super-resolution | |
US10225441B2 (en) | Time delay and integration (TDI) imaging sensor and method | |
CN108989717A (en) | Imaging sensor including phase-detection pixel | |
US9544513B2 (en) | Image sensor having pixel architecture for capturing depth image and color image | |
CN108600661A (en) | Separated grid is had ready conditions the imaging sensor of resetting | |
JP7297850B2 (en) | LED flicker mitigation for video | |
US9781364B2 (en) | Active pixel image sensor operating in global shutter mode, subtraction of the reset noise and non-destructive read | |
US9843757B2 (en) | Optical electronic device including enhanced global shutter pixel array and related methods | |
JP2009272820A (en) | Solid-state imaging device | |
US20180376083A1 (en) | Solid-state imaging device and imaging apparatus | |
US20170195574A1 (en) | Motion compensation for image sensor with a block based analog-to-digital converter | |
CN111466030B (en) | Intensity standardized image sensor | |
JP6101210B2 (en) | Linear image sensor with two lines and shared pixels | |
JP2011244452A (en) | Image sensor for cmos technology with high video capture rate | |
JP2016090785A (en) | Imaging equipment and control method thereof | |
Cho et al. | A 5.9 µm-pixel 2d/3d image sensor with background suppression over 100klx | |
CN103517003B (en) | Solid state image pickup device and its driving method and electronic equipment | |
CN105704401B (en) | The driving method and photographic device of photographic device | |
JP2015008343A (en) | Imaging device, and method for forming imaging image | |
TWI626619B (en) | Method and system for implementing dynamic ground sharing in an image sensor with pipeline architecture | |
WO2017038052A1 (en) | Imaging device and solid-state imaging device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SEMICONDUCTOR COMPONENTS INDUSTRIES, LLC, ARIZONA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:APTINA IMAGING CORPORATION;REEL/FRAME:035775/0051 Effective date: 20141217 Owner name: APTINA IMAGING CORPORATION, CAYMAN ISLANDS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WAN, CHUNG CHUN;LI, XIANGLI;AGRANOV, GENNADIY;REEL/FRAME:035774/0829 Effective date: 20120118 |
|
AS | Assignment |
Owner name: DEUTSCHE BANK AG NEW YORK BRANCH, NEW YORK Free format text: SECURITY INTEREST;ASSIGNOR:SEMICONDUCTOR COMPONENTS INDUSTRIES, LLC;REEL/FRAME:038620/0087 Effective date: 20160415 |
|
AS | Assignment |
Owner name: DEUTSCHE BANK AG NEW YORK BRANCH, AS COLLATERAL AG Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE INCORRECT PATENT NUMBER 5859768 AND TO RECITE COLLATERAL AGENT ROLE OF RECEIVING PARTY IN THE SECURITY INTEREST PREVIOUSLY RECORDED ON REEL 038620 FRAME 0087. ASSIGNOR(S) HEREBY CONFIRMS THE SECURITY INTEREST;ASSIGNOR:SEMICONDUCTOR COMPONENTS INDUSTRIES, LLC;REEL/FRAME:039853/0001 Effective date: 20160415 Owner name: DEUTSCHE BANK AG NEW YORK BRANCH, AS COLLATERAL AGENT, NEW YORK Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE INCORRECT PATENT NUMBER 5859768 AND TO RECITE COLLATERAL AGENT ROLE OF RECEIVING PARTY IN THE SECURITY INTEREST PREVIOUSLY RECORDED ON REEL 038620 FRAME 0087. ASSIGNOR(S) HEREBY CONFIRMS THE SECURITY INTEREST;ASSIGNOR:SEMICONDUCTOR COMPONENTS INDUSTRIES, LLC;REEL/FRAME:039853/0001 Effective date: 20160415 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE |
|
AS | Assignment |
Owner name: FAIRCHILD SEMICONDUCTOR CORPORATION, ARIZONA Free format text: RELEASE OF SECURITY INTEREST IN PATENTS RECORDED AT REEL 038620, FRAME 0087;ASSIGNOR:DEUTSCHE BANK AG NEW YORK BRANCH, AS COLLATERAL AGENT;REEL/FRAME:064070/0001 Effective date: 20230622 Owner name: SEMICONDUCTOR COMPONENTS INDUSTRIES, LLC, ARIZONA Free format text: RELEASE OF SECURITY INTEREST IN PATENTS RECORDED AT REEL 038620, FRAME 0087;ASSIGNOR:DEUTSCHE BANK AG NEW YORK BRANCH, AS COLLATERAL AGENT;REEL/FRAME:064070/0001 Effective date: 20230622 |