US20220034651A1 - Energy optimized imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor - Google Patents
Energy optimized imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor Download PDFInfo
- Publication number
- US20220034651A1 US20220034651A1 US17/495,000 US202117495000A US2022034651A1 US 20220034651 A1 US20220034651 A1 US 20220034651A1 US 202117495000 A US202117495000 A US 202117495000A US 2022034651 A1 US2022034651 A1 US 2022034651A1
- Authority
- US
- United States
- Prior art keywords
- sensor
- scene
- light source
- sequence
- pixels
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003384 imaging method Methods 0.000 title abstract description 15
- 230000001360 synchronised effect Effects 0.000 title abstract description 4
- 239000011159 matrix material Substances 0.000 abstract description 5
- 238000005286 illumination Methods 0.000 description 15
- 238000000034 method Methods 0.000 description 7
- 238000005096 rolling process Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 5
- 239000004973 liquid crystal related substance Substances 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 239000012141 concentrate Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000000779 smoke Substances 0.000 description 2
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 1
- 101150013568 US16 gene Proteins 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001427 coherent effect Effects 0.000 description 1
- 230000002301 combined effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000000386 microscopy Methods 0.000 description 1
- 239000003595 mist Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000010287 polarization Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 229910052710 silicon Inorganic materials 0.000 description 1
- 239000010703 silicon Substances 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
- G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
- G01B11/2513—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object with several lines being projected in more than one direction, e.g. grids, patterns
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
- G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
- G01B11/2518—Projection by scanning of the object
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
- G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
- G01B11/2545—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object with one projection direction and several detection directions, e.g. stereo
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
- G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/42—Simultaneous measurement of distance and other co-ordinates
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/481—Constructional features, e.g. arrangements of optical elements
- G01S7/4817—Constructional features, e.g. arrangements of optical elements relating to scanning
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B15/00—Special procedures for taking photographs; Apparatus therefor
- G03B15/02—Illuminating scene
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B17/00—Details of cameras or camera bodies; Accessories therefor
- G03B17/48—Details of cameras or camera bodies; Accessories therefor adapted for combination with other photographic or optical apparatus
- G03B17/54—Details of cameras or camera bodies; Accessories therefor adapted for combination with other photographic or optical apparatus with projector
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B21/00—Projectors or projection-type viewers; Accessories therefor
- G03B21/14—Details
- G03B21/20—Lamp housings
- G03B21/2006—Lamp housings characterised by the light source
- G03B21/2033—LED or laser light sources
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B7/00—Control of exposure by setting shutters, diaphragms or filters, separately or conjointly
- G03B7/16—Control of exposure by setting shutters, diaphragms or filters, separately or conjointly in accordance with both the intensity of the flash source and the distance of the flash source from the object, e.g. in accordance with the "guide number" of the flash bulb and the focusing of the camera
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/65—Control of camera operation in relation to power supply
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/74—Circuitry for compensating brightness variation in the scene by influencing the scene brightness using illuminating means
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/30—Transforming light or analogous information into electric information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/12—Picture reproducers
- H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
- H04N9/3129—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM] scanning a light beam on the display screen
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
Definitions
- the brightness of the source is just one way to control the amount of light that reaches the sensor of a computational imaging system.
- Modern systems use an arrangement of devices to transport light from a source to the scene (or from the scene to sensor) and these devices (galvanometers, digital micro-mirror devices, liquid-crystal panels, phase modulators, etc.) are often programmable. However, these devices in the currently used configurations are also inefficient in their power consumption.
- a light source In this ubiquitous configuration, the light source is always turned on, and, while the energy efficiency of light sources have greatly increased over time, having the light source always turned on is still a major source of power consumption.
- the controllable light-blocking mask is used to selectively light the scene to be captured by the photo-sensor. This practice is not energy efficient, however, due to the fact that photons that are being generated are simply blocked within the system.
- This invention introduces an energy optimized imaging system that captures images using one or more directable beam light sources synchronized with one or more active pixel selectable photosensors.
- this invention can perform numerous imaging techniques that are either not possible with current technology, or are possible with current technology but would require far more optical components, far more expense, far more computation, and/or far more power consumption.
- This invention can demonstrate several never-seen-before capabilities. For instance, live structured-light video of very bright scenes can be captured, even, for example, a light bulb that is on.
- the invention can also capture epipolar-only and indirect-only live video with optimal energy efficiency.
- the invention can measure the 3D shape of objects in challenging conditions, such as strong indirect light, strong ambient light, and smoke while using a low-power projector as the light source. Also, the invention can record live video from the point of view of a projector, rather than a photosensor.
- FIG. 1 shows a functional block diagram of the imaging system.
- FIGS. 2A-2B illustrate the extremes of projector technologies.
- FIG. 2A is a representation of the low end of the efficiency spectrum, where the light source is always on, blocked with mask p.
- FIG. 2B represents the most efficient projector, able to concentrate all the energy of the light source to just one pixel.
- FIG. 3 illustrates the preferred embodiment of the invention wherein a light source and a sensor are arranged in a rectified stereo configuration.
- FIG. 4A shows the epipolar component of an image of an object
- FIG. 4B shows the non-epipolar component of an image of the same object.
- a widely known truth in the field of image capture is that to optimally capture images with the most detail and least noise, the light throughput between the light source and the photosensor must be optimized.
- This invention implements this maxim while at the same time allowing for selective blocking of light paths between the light source and photosensor.
- the system topology that results from this optimization also allows for never-seen-before imaging techniques and energy efficiency.
- FIG. 1 There are three main parts to the invention as currently implemented, interconnected as shown in FIG. 1 : a directable light source 10 , a sensor 15 and, crucial to this invention, a synchronizing controller 20 that synchronizes the active region of sensor 15 to the beam direction of light source 10 in accordance with an optimizing equation, discussed below.
- the term “directable light source” is a controllable light source that emits different amounts of light in different directions, where each pixel in the projector corresponds to a direction along which a slightly diverging beam is emitted. By changing the amount of light emitted along each direction, the projected pattern can be changed.
- SLM spatial light modulator
- scanning projectors There are two broad classes of projectors, spatial light modulator (SLM) based projectors and scanning projectors.
- SLM projectors are of the type shown in FIG. 2A , and consists of a 2D array of controllable elements, one element for each pixel.
- the elements can be controlled to either block/reflect light (a digital micro-mirror device (DMD)) or block/transmit light (liquid crystal display (LCD) or liquid crystal on silicon (LCoS)).
- DMD digital micro-mirror device
- LCD liquid crystal display
- LCD liquid crystal on silicon
- the projector has a light source inside along with optics to focus that light onto the entire field of the SLM.
- the SLM elements By controlling the SLM elements, one can change the spatial light pattern the projector sends out.
- Scanning projectors are of the type shown in FIG. 2B and internally have a light source, some optics that focuses the light into a narrow beam and a some type of controllable mirror that deflects the beam.
- the projector By rapidly changing the mirror deflection and simultaneously modulating the light power of the internal source, the projector is able to ‘draw’ a spatial pattern. With this type of projector only one direction (or pixel) is being illuminated at any point of time, but this direction changes very quickly.
- the mirror deflects the beam along a rasterized path (i.e., scans across one row at a time and proceeds row-by-row). This is a raster scanning laser projector.
- light source As used herein, the terms “light source”, “directable light source” and “projector” are used interchangeably.
- Phase measuring light sensors can be used for measuring distance based on continuous wave time-of-flight
- Dynamic vision Sensors DVS
- photodiode arrays and avalanche photodiode arrays are high speed, high sensitivity light sensors that are often used for impulse time-of-flight measurements (flash LIDARS).
- flash LIDARS impulse time-of-flight measurements
- basic CMOS and CCD sensors may be used.
- a scanning projector of the type using a LASER-based projector with a beam steering mechanism, for example, a MEMS mirror, is used as the directable light source, and the sensor is preferably a light sensitive photosensor with a rolling shutter.
- light source 10 can be controlled to illuminate a specific point, line, or area of a scene to be imaged.
- Sensor 15 contains a grid of light-sensitive pixels.
- the rolling shutter of the photosensor captures an image by rapidly scanning the scene one row (or column) of pixels at a time. By synchronizing these, at any point of time, the light can selectively illuminate just the part of the scene that will be scanned by the photo-sensor. This optimizes the energy required to image the scene.
- the goal of the invention is to maximize the total energy that can be transmitted from individual projector pixels to individual sensor pixels.
- the illumination vector l is used to describe how the total energy of a projector is distributed over N individual pixels.
- each element of l measures the total energy emitted by the source through a specific projector pixel during the exposure time.
- the l 1 -norm of l is therefore equal to the total “useful” energy of the source, i.e., the energy actually used for scene illumination. This energy cannot be larger than the energy generated by the source:
- ⁇ 1 is the l 1 -norm, giving the sum of all elements of a vector.
- the energy efficiency of a projector depends critically on its ability to direct a maximum amount of the energy generated by the light source 10 to individual pixels. This ability is expressed as an upper bound on the individual elements of l:
- ⁇ is a projector-specific parameter defined as the spatial spread. This parameter takes values between 1 and N and models energy redistribution. The larger its value, the lower the energy that can be sent through any one pixel, and the more energy wasted when projecting a pattern with just few pixels turned on.
- ⁇ depends on the projection technology.
- ⁇ ⁇ is the max of two norms and therefore also a norm.
- mask m which is bounded from 0 to 1.
- the combined effect of the mask and illumination pattern can be represented as the outer product matrix of two vectors:
- matrix ⁇ can be thought of as defining a non-uniform spatial light distribution that concentrates energy usable for imaging in some parts of space and not in others. Energy utilization is maximized when both the illumination pattern and the mask reach their norm upper bounds, ⁇ m ⁇ ⁇ ⁇ I ⁇ ⁇ .
- ⁇ is a regularization parameter that balances energy efficiency and the reproduction of ⁇ . This allows for finding M & L that will saturate their upper-bound constraints, and hence a fully illuminated matrix ⁇ .
- Illumination codes that maximize the energy efficiency are the impulse illuminations, like those of FIG. 2B .
- This is replicated in the invention by the use of the a LASER-based projector, preferably an ultra-low power LASER, with a MEMS mirror to provide beam steering as depicted in FIG. 1 as light source 10 .
- the light source and the sensor must be configured using a rectified stereo configuration, as shown in FIG. 3 .
- the sensor masks are ideally chosen such that for each projector scan line, only the corresponding row in the sensor is exposed. In reality, a single illuminated line in the projector may be mapped to multiple lines in the sensor.
- the masks can be implemented easily using the rolling shutter setup by placing the projector and sensor in an approximate stereo configuration such that their optical axes are parallel to each other, perpendicular to the baseline and the scanlines of the projector and rows of the sensor are parallel to the baseline.
- the epipolar lines between the projector and sensor are horizontal lines. For any projector scanline the row of pixels in the sensor image along which the corresponding points must lie can be identified.
- the exposure t e for each sensor row is matched to the time the projector stays on a scanline (t p ) and the other timing parameters are chosen so that the line scanned by the projector is synchronized to the row being exposed in the sensor.
- the sensor exposure time is set to be t p less than the projector cycle time and the trigger is offset by t p so that every row is exposed for the entire projector cycle except during the time it is illuminated directly by the projector.
- This energy optimized imaging system also has unique capabilities that are not possible in other imaging systems.
- the rolling shutter of sensor 15 is tuned by synchronization controller 20 for the impulse illuminations of light source 10 , very little ambient light is let into the sensor. This allows the invention to image extremely bright objects and scenes under bright ambient illumination. With current technology imaging systems, light from a controlled light source would be overwhelmed by ambient light and would not be detectable at the photosensor.
- the rolling shutter of sensor 15 is aligned solely to the light source 10 , reflections and scattered light that are caused by the object (such as if the object was mirrored, shiny, metallic, translucent, etc.) are not captured in the frame. Note that the rolling shutter of sensor 15 can purposely be offset from the source illumination so that only the reflections are captured.
- This ability to not image reflections, scattered light and ambient light also gives the invention the ability to image and recover the shape of objects that are in challenging lighting conditions, specifically smoke or mist filled surroundings.
- Using the source illumination-to-photo-sensor disparity offset can allow for three-dimensional reconstruction within such lighting challenged areas.
- controller 20 could be implemented as circuitry, as an ASIC, as a microprocessor running software, or by any other means known in the art.
- the invention is not intended to be limited to one method of implementing the functions of the controller.
- Dual photography a technique where the image generated is from the viewpoint of the light source rather than the photosensor, is also possible, even in a live video context, with no processing required.
- the illumination technique used in this invention can be expanded to multiple photosensors. This allows for highly power efficient active illumination stereo using two or more photosensors.
- the technique also extends naturally to configurations with multiple light sources. Different light sources interfere with each other minimally when used with the proposed technique. With inter-source synchronization, interference can be eliminated completely.
- the proposed technique can be realized with a time-of-flight (ToF) photosensor.
- a rolling shutter ToF photosensor combined with a modulated scanning laser light source using our technique would allow for a power efficient ToF depth sensor that works under bright ambient light conditions and suppresses indirect lighting effects.
- the invention can be used with other imaging modalities including, but not limited to, light field imaging, microscopy, polarization, coherent, nonlinear, fluorescent and non-linear imaging.
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Optics & Photonics (AREA)
- Electromagnetism (AREA)
- Computer Networks & Wireless Communication (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Theoretical Computer Science (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
Description
- This application is a divisional application filed under 35 U.S.C. § 121 of U.S. patent application Ser. No. 16/516,664, filed Jul. 19, 2019, which is a continuation of U.S. patent application Ser. No. 15/545,391, filed Jul. 21, 2017, which is a filing under section 371 of PCT Application No. PCT/US16/17942, filed Feb. 15, 2016, which claims the benefit of U.S. Provisional Patent Application No. 62/176,352, filed Feb. 13, 2015.
- This invention was made with government support under National Science Foundation No. IIS317749 and Army/ARO No. W911NF-10-2-0016. The government has certain rights in this invention.
- When an image is captured under controlled lighting, the power of the light source has great impact on the result. All things being equal, brighter sources will send more photons to the sensor during an exposure, producing a brighter and less noisy image.
- The brightness of the source, however, is just one way to control the amount of light that reaches the sensor of a computational imaging system. Modern systems use an arrangement of devices to transport light from a source to the scene (or from the scene to sensor) and these devices (galvanometers, digital micro-mirror devices, liquid-crystal panels, phase modulators, etc.) are often programmable. However, these devices in the currently used configurations are also inefficient in their power consumption.
- Studies of this problem began in the 1960s for arrangements of just the three main active components; a light source, a controllable light-blocking mask, and a photo-sensor. In this ubiquitous configuration, the light source is always turned on, and, while the energy efficiency of light sources have greatly increased over time, having the light source always turned on is still a major source of power consumption. The controllable light-blocking mask is used to selectively light the scene to be captured by the photo-sensor. This practice is not energy efficient, however, due to the fact that photons that are being generated are simply blocked within the system.
- This invention introduces an energy optimized imaging system that captures images using one or more directable beam light sources synchronized with one or more active pixel selectable photosensors.
- By synchronizing control of both a light source that has the ability to illuminate a specific area(s) along with a photosensor having a configurable mask having the ability to mask specific pixels, this invention can perform numerous imaging techniques that are either not possible with current technology, or are possible with current technology but would require far more optical components, far more expense, far more computation, and/or far more power consumption.
- This invention can demonstrate several never-seen-before capabilities. For instance, live structured-light video of very bright scenes can be captured, even, for example, a light bulb that is on. The invention can also capture epipolar-only and indirect-only live video with optimal energy efficiency. The invention can measure the 3D shape of objects in challenging conditions, such as strong indirect light, strong ambient light, and smoke while using a low-power projector as the light source. Also, the invention can record live video from the point of view of a projector, rather than a photosensor.
-
FIG. 1 shows a functional block diagram of the imaging system. -
FIGS. 2A-2B illustrate the extremes of projector technologies.FIG. 2A is a representation of the low end of the efficiency spectrum, where the light source is always on, blocked with mask p.FIG. 2B represents the most efficient projector, able to concentrate all the energy of the light source to just one pixel. -
FIG. 3 illustrates the preferred embodiment of the invention wherein a light source and a sensor are arranged in a rectified stereo configuration. -
FIG. 4A shows the epipolar component of an image of an object, whileFIG. 4B shows the non-epipolar component of an image of the same object. - A widely known truth in the field of image capture is that to optimally capture images with the most detail and least noise, the light throughput between the light source and the photosensor must be optimized. This invention implements this maxim while at the same time allowing for selective blocking of light paths between the light source and photosensor. The system topology that results from this optimization also allows for never-seen-before imaging techniques and energy efficiency.
- There are three main parts to the invention as currently implemented, interconnected as shown in
FIG. 1 : adirectable light source 10, asensor 15 and, crucial to this invention, a synchronizingcontroller 20 that synchronizes the active region ofsensor 15 to the beam direction oflight source 10 in accordance with an optimizing equation, discussed below. - As used herein, the term “directable light source” is a controllable light source that emits different amounts of light in different directions, where each pixel in the projector corresponds to a direction along which a slightly diverging beam is emitted. By changing the amount of light emitted along each direction, the projected pattern can be changed. There are two broad classes of projectors, spatial light modulator (SLM) based projectors and scanning projectors.
- SLM projectors are of the type shown in
FIG. 2A , and consists of a 2D array of controllable elements, one element for each pixel. Depending on the type of SLM, the elements can be controlled to either block/reflect light (a digital micro-mirror device (DMD)) or block/transmit light (liquid crystal display (LCD) or liquid crystal on silicon (LCoS)). The projector has a light source inside along with optics to focus that light onto the entire field of the SLM. By controlling the SLM elements, one can change the spatial light pattern the projector sends out. In the equations shown herein, these projectors have σ=N, where N is the number of pixels. This type of projector is very inefficient when projecting patterns where only a small number of pixels are illuminated (a pattern containing a single line for instance) because most of the light needs to be blocked by the SLM to create the pattern. - Scanning projectors are of the type shown in
FIG. 2B and internally have a light source, some optics that focuses the light into a narrow beam and a some type of controllable mirror that deflects the beam. By rapidly changing the mirror deflection and simultaneously modulating the light power of the internal source, the projector is able to ‘draw’ a spatial pattern. With this type of projector only one direction (or pixel) is being illuminated at any point of time, but this direction changes very quickly. In the projector used in the preferred embodiment of the invention, the mirror deflects the beam along a rasterized path (i.e., scans across one row at a time and proceeds row-by-row). This is a raster scanning laser projector. In the equations shown herein, scanning projectors can be thought of as projectors with σ=1. This type of projector is efficient when projecting patterns where only a small number of pixels are illuminated. - As used herein, the terms “light source”, “directable light source” and “projector” are used interchangeably.
- Also, in the preferred embodiments of the invention, various types of sensors may be used. Phase measuring light sensors (example photonic mixing devices or PMDs) can be used for measuring distance based on continuous wave time-of-flight; Dynamic vision Sensors (DVS) are sensors that are sensitive to changes in light levels; and photodiode arrays and avalanche photodiode arrays are high speed, high sensitivity light sensors that are often used for impulse time-of-flight measurements (flash LIDARS). In addition, basic CMOS and CCD sensors may be used.
- In the preferred embodiment of the invention, a scanning projector of the type using a LASER-based projector with a beam steering mechanism, for example, a MEMS mirror, is used as the directable light source, and the sensor is preferably a light sensitive photosensor with a rolling shutter.
- With reference to
FIG. 1 ,light source 10 can be controlled to illuminate a specific point, line, or area of a scene to be imaged.Sensor 15 contains a grid of light-sensitive pixels. In a preferred embodiment, the rolling shutter of the photosensor captures an image by rapidly scanning the scene one row (or column) of pixels at a time. By synchronizing these, at any point of time, the light can selectively illuminate just the part of the scene that will be scanned by the photo-sensor. This optimizes the energy required to image the scene. The goal of the invention is to maximize the total energy that can be transmitted from individual projector pixels to individual sensor pixels. - The mathematical framework for this energy-optimized imaging system follows. If
light source 10 is always on, and emits at the constant rate of Φ watts, illuminating a scene for exposure time T means that the total energy generated bylight source 10 is ΦT. - The illumination vector l is used to describe how the total energy of a projector is distributed over N individual pixels. In particular, each element of l measures the total energy emitted by the source through a specific projector pixel during the exposure time. The l1-norm of l is therefore equal to the total “useful” energy of the source, i.e., the energy actually used for scene illumination. This energy cannot be larger than the energy generated by the source:
-
0≤l, ∥l∥ 1 ≤ΦT - where ∥∥1 is the l1-norm, giving the sum of all elements of a vector.
- The energy efficiency of a projector depends critically on its ability to direct a maximum amount of the energy generated by the
light source 10 to individual pixels. This ability is expressed as an upper bound on the individual elements of l: -
∥l∥ 1 ≤ΦT/σ - where σ is a projector-specific parameter defined as the spatial spread. This parameter takes values between 1 and N and models energy redistribution. The larger its value, the lower the energy that can be sent through any one pixel, and the more energy wasted when projecting a pattern with just few pixels turned on.
- The specific value of σ depends on the projection technology. At the far end of the range, with σ=N, are conventional projectors, as shown in
FIG. 2A . These projectors use mirrors and lenses to distribute light evenly over a spatial light modulator, which may then block all or part of the light. The near end of the range, with σ=1, represents an idealized projector that is perfectly efficient, as shown inFIG. 2B . This projector can send all its light through just one pixel for the entire exposure time T but can also distribute it, without any blocking, according to an arbitrary illumination l. - The l1 and l∞ constraints on l can be written more concisely as
-
- where ∥⋅∥†σ is the max of two norms and therefore also a norm. These constraints are useful in three ways. First, arrangements can be optimized with very different light redistribution properties by adjusting the spatial spread parameter. Second, the dependence on exposure time makes a distinction between systems that conserve energy and those that merely conserve power. Third, they explicitly account for timescale-dependent behavior, for example raster-scan laser projectors can act like a beam, light sheet, or point source depending on T.
- For masks that can control light attenuation at individual pixels on a sensor, we consider mask m, which is bounded from 0 to 1. The combined effect of the mask and illumination pattern can be represented as the outer product matrix of two vectors:
-
Π=ml T - Intuitively, matrix Π can be thought of as defining a non-uniform spatial light distribution that concentrates energy usable for imaging in some parts of space and not in others. Energy utilization is maximized when both the illumination pattern and the mask reach their norm upper bounds, ∥m∥∞∥I∥†σ.
- It is also possible to use more than one mask and illumination pattern for the frame exposure time. Suppose for instance that K masks and illuminations were used. The optimization equation could then be written as:
-
- There may be sequences that distribute light exactly like M and L but with greater total energy. Finding the most energy-efficient sequences requires solving a homogeneous factorization problem, where the goal is to produce a matrix Π with the largest possible scale factor:
-
- The optimization equations above are hard to solve directly. But the equation can be relaxed into the following form:
-
- where λ is a regularization parameter that balances energy efficiency and the reproduction of Π. This allows for finding M & L that will saturate their upper-bound constraints, and hence a fully illuminated matrix Π.
- Illumination codes that maximize the energy efficiency are the impulse illuminations, like those of
FIG. 2B . This is replicated in the invention by the use of the a LASER-based projector, preferably an ultra-low power LASER, with a MEMS mirror to provide beam steering as depicted inFIG. 1 aslight source 10. In this embodiment, the light source and the sensor must be configured using a rectified stereo configuration, as shown inFIG. 3 . To capture the epipolar component, as shown inFIG. 4A , the sensor masks are ideally chosen such that for each projector scan line, only the corresponding row in the sensor is exposed. In reality, a single illuminated line in the projector may be mapped to multiple lines in the sensor. This blocks all light that does not satisfy the constraint between the projector and sensor. The masks can be implemented easily using the rolling shutter setup by placing the projector and sensor in an approximate stereo configuration such that their optical axes are parallel to each other, perpendicular to the baseline and the scanlines of the projector and rows of the sensor are parallel to the baseline. In this configuration, the epipolar lines between the projector and sensor are horizontal lines. For any projector scanline the row of pixels in the sensor image along which the corresponding points must lie can be identified. - To capture the epipolar component, the exposure te for each sensor row is matched to the time the projector stays on a scanline (tp) and the other timing parameters are chosen so that the line scanned by the projector is synchronized to the row being exposed in the sensor. Conversely, to capture non-epipolar light, the sensor exposure time is set to be tp less than the projector cycle time and the trigger is offset by tp so that every row is exposed for the entire projector cycle except during the time it is illuminated directly by the projector.
- This energy optimized imaging system also has unique capabilities that are not possible in other imaging systems.
- Because the rolling shutter of
sensor 15 is tuned bysynchronization controller 20 for the impulse illuminations oflight source 10, very little ambient light is let into the sensor. This allows the invention to image extremely bright objects and scenes under bright ambient illumination. With current technology imaging systems, light from a controlled light source would be overwhelmed by ambient light and would not be detectable at the photosensor. - Also, since the rolling shutter of
sensor 15 is aligned solely to thelight source 10, reflections and scattered light that are caused by the object (such as if the object was mirrored, shiny, metallic, translucent, etc.) are not captured in the frame. Note that the rolling shutter ofsensor 15 can purposely be offset from the source illumination so that only the reflections are captured. - This ability to not image reflections, scattered light and ambient light also gives the invention the ability to image and recover the shape of objects that are in challenging lighting conditions, specifically smoke or mist filled surroundings. Using the source illumination-to-photo-sensor disparity offset can allow for three-dimensional reconstruction within such lighting challenged areas.
- It should be understood by one of skill in the art that
controller 20 could be implemented as circuitry, as an ASIC, as a microprocessor running software, or by any other means known in the art. The invention is not intended to be limited to one method of implementing the functions of the controller. - Dual photography, a technique where the image generated is from the viewpoint of the light source rather than the photosensor, is also possible, even in a live video context, with no processing required.
- The illumination technique used in this invention can be expanded to multiple photosensors. This allows for highly power efficient active illumination stereo using two or more photosensors.
- The technique also extends naturally to configurations with multiple light sources. Different light sources interfere with each other minimally when used with the proposed technique. With inter-source synchronization, interference can be eliminated completely.
- The proposed technique can be realized with a time-of-flight (ToF) photosensor. A rolling shutter ToF photosensor combined with a modulated scanning laser light source using our technique would allow for a power efficient ToF depth sensor that works under bright ambient light conditions and suppresses indirect lighting effects.
- In other embodiments, the invention can be used with other imaging modalities including, but not limited to, light field imaging, microscopy, polarization, coherent, nonlinear, fluorescent and non-linear imaging.
- Although the invention is illustrated and described herein with reference to specific embodiments, the invention is not intended to be limited to the details shown. Rather, various modifications may be made in the implementation without departing from the invention.
Claims (19)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/495,000 US20220034651A1 (en) | 2015-02-13 | 2021-10-06 | Energy optimized imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562176352P | 2015-02-13 | 2015-02-13 | |
PCT/US2016/017942 WO2016131036A1 (en) | 2015-02-13 | 2016-02-15 | Imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor |
US201715545391A | 2017-07-21 | 2017-07-21 | |
US16/516,664 US11747135B2 (en) | 2015-02-13 | 2019-07-19 | Energy optimized imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor |
US17/495,000 US20220034651A1 (en) | 2015-02-13 | 2021-10-06 | Energy optimized imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/516,664 Division US11747135B2 (en) | 2015-02-13 | 2019-07-19 | Energy optimized imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220034651A1 true US20220034651A1 (en) | 2022-02-03 |
Family
ID=69139106
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/516,664 Active 2036-12-27 US11747135B2 (en) | 2015-02-13 | 2019-07-19 | Energy optimized imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor |
US17/495,000 Pending US20220034651A1 (en) | 2015-02-13 | 2021-10-06 | Energy optimized imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/516,664 Active 2036-12-27 US11747135B2 (en) | 2015-02-13 | 2019-07-19 | Energy optimized imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor |
Country Status (1)
Country | Link |
---|---|
US (2) | US11747135B2 (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111149350B (en) * | 2017-09-28 | 2022-02-22 | 苹果公司 | Generating still images using event cameras |
CN112019835B (en) * | 2020-08-08 | 2022-11-15 | 江西欧迈斯微电子有限公司 | Frame rate verification device and method for dynamic vision sensor module and storage medium |
JP2024068423A (en) * | 2022-11-08 | 2024-05-20 | ソニーセミコンダクタソリューションズ株式会社 | Sensing system, sensing control device, and sensing method |
Citations (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4236819A (en) * | 1974-07-29 | 1980-12-02 | The United States Of America As Represented By The Secretary Of The Air Force | Imagery with constant range lines |
US4687326A (en) * | 1985-11-12 | 1987-08-18 | General Electric Company | Integrated range and luminance camera |
US4687325A (en) * | 1985-03-28 | 1987-08-18 | General Electric Company | Three-dimensional range camera |
US20050088644A1 (en) * | 2001-04-04 | 2005-04-28 | Morcom Christopher J. | Surface profile measurement |
US20050128196A1 (en) * | 2003-10-08 | 2005-06-16 | Popescu Voicu S. | System and method for three dimensional modeling |
US20050219552A1 (en) * | 2002-06-07 | 2005-10-06 | Ackerman Jermy D | Methods and systems for laser based real-time structured light depth extraction |
US20070268398A1 (en) * | 2006-05-17 | 2007-11-22 | Ramesh Raskar | Apparatus and method for illuminating a scene with multiplexed illumination for motion capture |
US20080118143A1 (en) * | 2006-11-21 | 2008-05-22 | Mantis Vision Ltd. | 3D Geometric Modeling And Motion Capture Using Both Single And Dual Imaging |
US20100046801A1 (en) * | 2006-12-25 | 2010-02-25 | Rui Ishiyama | Apparatus, method and program for distance measurement |
US20100074532A1 (en) * | 2006-11-21 | 2010-03-25 | Mantisvision Ltd. | 3d geometric modeling and 3d video content creation |
US20100085425A1 (en) * | 2008-10-07 | 2010-04-08 | Kar-Han Tan | One-Touch Projector Alignment for 3D Stereo Display |
US20110026783A1 (en) * | 2008-04-03 | 2011-02-03 | Kyushu Institute Of Technology | Personal authentication method using subcutaneous bloodstream measurement and personal authentication device |
US20110071675A1 (en) * | 2009-09-22 | 2011-03-24 | Gm Global Technology Operations, Inc. | Visual perception system and method for a humanoid robot |
US20110164805A1 (en) * | 2010-01-07 | 2011-07-07 | De La Rue North America Inc. | Systems and Methods for Detecting an Optically Variable Material |
US7978928B2 (en) * | 2007-09-18 | 2011-07-12 | Seiko Epson Corporation | View projection for dynamic configurations |
US8294762B2 (en) * | 2008-10-10 | 2012-10-23 | Fujifilm Corporation | Three-dimensional shape measurement photographing apparatus, method, and program |
US8339616B2 (en) * | 2009-03-31 | 2012-12-25 | Micrometric Vision Technologies | Method and apparatus for high-speed unconstrained three-dimensional digitalization |
US20130021474A1 (en) * | 2011-07-20 | 2013-01-24 | Raytheon Company | Rolling-shutter imaging system with synchronized scanning illumination and methods for higher-resolution imaging |
US20130121560A1 (en) * | 2011-11-14 | 2013-05-16 | Ryusuke Hirai | Image processing device, method of processing image, and image display apparatus |
US20130250272A1 (en) * | 2012-03-21 | 2013-09-26 | David Ludwig | Threshold Detection Method, Module and Readout Integrated Circuit Layer for LIDAR Time of Flight System Using Differentiated Gaussian Signal |
US20140211193A1 (en) * | 2012-09-24 | 2014-07-31 | Alces Technology, Inc. | Structured light and time of flight depth capture with a MEMS ribbon linear array spatial light modulator |
US20140232566A1 (en) * | 2011-06-17 | 2014-08-21 | Leddartech Inc. | System and method for traffic side detection and characterization |
US20140253724A1 (en) * | 2013-03-08 | 2014-09-11 | Mitutoyo Corporation | Shape measuring apparatus |
US20140327746A1 (en) * | 2013-05-06 | 2014-11-06 | Iphoton Solutions, Llc | Volume reconstruction of an object using a 3d sensor and robotic coordinates |
US20150285625A1 (en) * | 2014-04-07 | 2015-10-08 | Samsung Electronics Co., Ltd. | High resolution, high frame rate, low power image sensor |
US20150381962A1 (en) * | 2014-06-27 | 2015-12-31 | Electronics And Telecommunications Research Institute | Method and apparauts for implementing active imaging system |
US20160124203A1 (en) * | 2014-11-04 | 2016-05-05 | Olympus Corporation | Microscope and microscope image acquisition method |
US20160182788A1 (en) * | 2014-12-22 | 2016-06-23 | Google Inc. | Time-of-flight camera system with scanning illuminator |
US20160182889A1 (en) * | 2014-12-19 | 2016-06-23 | Datalogic ADC, Inc. | Depth camera system using coded structured light |
US20160349369A1 (en) * | 2014-01-29 | 2016-12-01 | Lg Innotek Co., Ltd. | Device for extracting depth information and method thereof |
US20170064235A1 (en) * | 2015-08-27 | 2017-03-02 | Samsung Electronics Co., Ltd. | Epipolar plane single-pulse indirect tof imaging for automotives |
US20170142406A1 (en) * | 2015-11-16 | 2017-05-18 | Samsung Electronics Co., Ltd. | Apparatus for and method of illumination control for acquiring image information and depth information simultaneously |
US20170176575A1 (en) * | 2015-12-18 | 2017-06-22 | Gerard Dirk Smits | Real time position sensing of objects |
US20170353707A1 (en) * | 2016-06-03 | 2017-12-07 | Samsung Electronics Co., Ltd. | Timestamp error correction with double readout for the 3d camera with epipolar line laser point scanning |
US20170366801A1 (en) * | 2016-06-20 | 2017-12-21 | Intel Corporation | Depth image provision apparatus and method |
US20220245762A1 (en) * | 2021-02-03 | 2022-08-04 | Qualcomm Incorporated | High resolution time-of-flight depth imaging |
Family Cites Families (68)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5128753A (en) * | 1954-12-24 | 1992-07-07 | Lemelson Jerome H | Method and apparatus for scaning objects and generating image information |
US4621185A (en) | 1985-02-25 | 1986-11-04 | Caterpillar, Inc. | Adaptive welding apparatus having fill control correction for curvilinear weld grooves |
JP2884561B2 (en) * | 1993-06-02 | 1999-04-19 | ブラザー工業株式会社 | Image processing device |
JPH08159756A (en) | 1994-12-12 | 1996-06-21 | Nikon Corp | Obstacle detector |
US5717390A (en) | 1995-03-20 | 1998-02-10 | Hasselbring; Richard E. | Doppler-radar based automatic vehicle-classification system |
US5852672A (en) * | 1995-07-10 | 1998-12-22 | The Regents Of The University Of California | Image system for three dimensional, 360 DEGREE, time sequence surface mapping of moving objects |
US20020014533A1 (en) | 1995-12-18 | 2002-02-07 | Xiaxun Zhu | Automated object dimensioning system employing contour tracing, vertice detection, and forner point detection and reduction methods on 2-d range data maps |
JPH11132713A (en) | 1997-10-30 | 1999-05-21 | Oki Electric Ind Co Ltd | Distance measuring device and automatic focus control and photographing device, and recessed and projecting part detector |
US6556307B1 (en) * | 1998-09-11 | 2003-04-29 | Minolta Co., Ltd. | Method and apparatus for inputting three-dimensional data |
WO2000032451A1 (en) * | 1998-11-27 | 2000-06-08 | Hamamatsu Photonics K.K. | Driver sensor and apparatus for controlling air bag |
US6415051B1 (en) * | 1999-06-24 | 2002-07-02 | Geometrix, Inc. | Generating 3-D models using a manually operated structured light source |
JP2002039716A (en) | 2000-07-25 | 2002-02-06 | Olympus Optical Co Ltd | Depth map input device |
DE10135342C1 (en) | 2001-07-20 | 2002-10-31 | Schneider Laser Technologies | Projection device for automobile head-up display has projected light beams for each image point reflected by rear mirror at projection surface for convergence in planar viewing field infront of driver |
JP2004150810A (en) | 2002-10-28 | 2004-05-27 | Sony Corp | Three-dimensional image measuring instrument |
US7440084B2 (en) | 2004-12-16 | 2008-10-21 | Arete' Associates | Micromechanical and related lidar apparatus and method, and fast light-routing components |
WO2007043036A1 (en) | 2005-10-11 | 2007-04-19 | Prime Sense Ltd. | Method and system for object reconstruction |
US8014588B2 (en) | 2005-11-04 | 2011-09-06 | Cognitech, Inc. | System and method for three-dimensional estimation based on image data |
US7804585B2 (en) | 2006-01-08 | 2010-09-28 | Visiontools Bildanalyse Systeme Gmbh | Creation of a range image |
DE102006055746A1 (en) | 2006-11-25 | 2008-05-29 | Lavision Gmbh | Method for correcting a volume mapping equation for determining a velocity field of particles in a volume |
US7710545B2 (en) | 2008-02-13 | 2010-05-04 | The Boeing Company | Scanned laser detection and ranging apparatus |
US8194233B2 (en) | 2008-04-11 | 2012-06-05 | Microsoft Corporation | Method and system to reduce stray light reflection error in time-of-flight sensor arrays |
WO2010094135A1 (en) | 2009-02-23 | 2010-08-26 | 2G Robotics Inc. | Laser scanner assembly |
JP5243996B2 (en) | 2009-03-02 | 2013-07-24 | スタンレー電気株式会社 | Object detection device |
US8681216B2 (en) | 2009-03-12 | 2014-03-25 | Hewlett-Packard Development Company, L.P. | Depth-sensing camera system |
KR101668869B1 (en) * | 2009-05-29 | 2016-10-28 | 삼성전자주식회사 | Depth sensor, 3D image sensor and Depth sensing method there-of |
US20110102763A1 (en) | 2009-10-30 | 2011-05-05 | Microvision, Inc. | Three Dimensional Imaging Device, System and Method |
US8305561B2 (en) | 2010-03-25 | 2012-11-06 | Hokuyo Automatic Co., Ltd. | Scanning-type distance measuring apparatus |
US8485668B2 (en) * | 2010-05-28 | 2013-07-16 | Microsoft Corporation | 3D interaction for mobile device |
JP5528910B2 (en) | 2010-06-02 | 2014-06-25 | 株式会社Pfu | Overhead image reader |
KR101710514B1 (en) * | 2010-09-03 | 2017-02-27 | 삼성전자주식회사 | Depth sensor and method of estimating distance using the same |
WO2012125706A2 (en) * | 2011-03-15 | 2012-09-20 | Board Of Trustees Of Michigan State University | Real-time 3d shape measurement system |
DE102011106453A1 (en) | 2011-07-04 | 2013-01-10 | Carl Zeiss Ag | Method and device for time sequential recording of three-dimensional images |
JP5858688B2 (en) | 2011-08-30 | 2016-02-10 | スタンレー電気株式会社 | Distance image generator |
WO2013121267A1 (en) | 2012-02-15 | 2013-08-22 | Mesa Imaging Ag | Time of flight camera with stripe illumination |
KR102008233B1 (en) * | 2012-06-29 | 2019-08-07 | 삼성전자주식회사 | Depth measurement device and method of measuring a distance to an object using the depth estimation device |
JP6001960B2 (en) | 2012-08-23 | 2016-10-05 | 大塚電子株式会社 | Light distribution characteristic measuring apparatus and light distribution characteristic measuring method |
US9813618B2 (en) | 2012-11-02 | 2017-11-07 | Diversified Innovations Fund, Lllp | Wide area imaging system and method |
US9386298B2 (en) | 2012-11-08 | 2016-07-05 | Leap Motion, Inc. | Three-dimensional image sensors |
US9702977B2 (en) | 2013-03-15 | 2017-07-11 | Leap Motion, Inc. | Determining positional information of an object in space |
US20160065945A1 (en) | 2013-04-12 | 2016-03-03 | The Trustees Of Columbia University In The City Of New York | Systems, methods, and media for generating structured light |
US9412172B2 (en) * | 2013-05-06 | 2016-08-09 | Disney Enterprises, Inc. | Sparse light field representation |
US9476695B2 (en) | 2013-07-03 | 2016-10-25 | Faro Technologies, Inc. | Laser tracker that cooperates with a remote camera bar and coordinate measurement device |
US9188775B2 (en) | 2013-08-28 | 2015-11-17 | United Sciences, Llc | Optical scanning and measurement |
NO20131296A1 (en) | 2013-09-27 | 2015-01-19 | Megalink As | System and method for determining the position of a bullet projectile on a flashing plane |
JP2015135463A (en) | 2013-12-19 | 2015-07-27 | オリンパス株式会社 | Microscope apparatus and microscope system |
US20150176977A1 (en) | 2013-12-20 | 2015-06-25 | Lemoptix Sa | Methods and devices for determining position or distance |
US9386236B2 (en) * | 2014-01-26 | 2016-07-05 | Muller Imaging Technologies, Inc | Periodic fringe imaging with structured pattern illumination and electronic rolling shutter detection |
US11265534B2 (en) | 2014-02-08 | 2022-03-01 | Microsoft Technology Licensing, Llc | Environment-dependent active illumination for stereo matching |
US20150294496A1 (en) | 2014-04-14 | 2015-10-15 | GM Global Technology Operations LLC | Probabilistic person-tracking using multi-view fusion |
US9389069B2 (en) | 2014-03-26 | 2016-07-12 | Alces Technology, Inc. | Compact 3D depth capture systems |
JP6406856B2 (en) * | 2014-04-07 | 2018-10-17 | キヤノン株式会社 | Imaging apparatus and control method thereof |
US9383548B2 (en) | 2014-06-11 | 2016-07-05 | Olympus Corporation | Image sensor for depth estimation |
US9377533B2 (en) | 2014-08-11 | 2016-06-28 | Gerard Dirk Smits | Three-dimensional triangulation and time-of-flight based tracking systems and methods |
JP6552043B2 (en) | 2014-11-04 | 2019-07-31 | オリンパス株式会社 | Sheet illumination microscope |
US9536320B1 (en) | 2014-12-23 | 2017-01-03 | John H. Prince | Multiple coordinated detectors for examination and ranging |
US10061029B2 (en) | 2015-01-06 | 2018-08-28 | Samsung Electronics Co., Ltd. | Correction of depth images from T-O-F 3D camera with electronic-rolling-shutter for light modulation changes taking place during light integration |
WO2018136709A1 (en) | 2017-01-20 | 2018-07-26 | Carnegie Mellon University | Method for epipolar time of flight imaging |
WO2016131036A1 (en) | 2015-02-13 | 2016-08-18 | Carnegie Mellon University | Imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor |
US10679370B2 (en) | 2015-02-13 | 2020-06-09 | Carnegie Mellon University | Energy optimized imaging system with 360 degree field-of-view |
EP3274986A4 (en) | 2015-03-21 | 2019-04-17 | Mine One GmbH | Virtual 3d methods, systems and software |
US10043282B2 (en) | 2015-04-13 | 2018-08-07 | Gerard Dirk Smits | Machine vision for ego-motion, segmenting, and classifying objects |
US10145678B2 (en) * | 2015-04-20 | 2018-12-04 | Samsung Electronics Co., Ltd. | CMOS image sensor for depth measurement using triangulation with point scan |
GB201516701D0 (en) | 2015-09-21 | 2015-11-04 | Innovation & Business Dev Solutions Ltd | Time of flight distance sensor |
KR102486385B1 (en) | 2015-10-29 | 2023-01-09 | 삼성전자주식회사 | Apparatus and method of sensing depth information |
JP6668763B2 (en) * | 2016-01-13 | 2020-03-18 | セイコーエプソン株式会社 | Image recognition device, image recognition method, and image recognition unit |
US10531073B2 (en) | 2016-03-17 | 2020-01-07 | Samsung Electronics Co., Ltd. | Method and apparatus for automatic calibration of RGBZ sensors utilizing epipolar geometry and scanning beam projector |
US10451740B2 (en) | 2016-04-26 | 2019-10-22 | Cepton Technologies, Inc. | Scanning lidar systems for three-dimensional sensing |
KR102674646B1 (en) * | 2016-10-18 | 2024-06-13 | 포토닉 센서즈 앤드 알고리즘즈 에스.엘. | Apparatus and method for obtaining distance information from a view |
-
2019
- 2019-07-19 US US16/516,664 patent/US11747135B2/en active Active
-
2021
- 2021-10-06 US US17/495,000 patent/US20220034651A1/en active Pending
Patent Citations (37)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4236819A (en) * | 1974-07-29 | 1980-12-02 | The United States Of America As Represented By The Secretary Of The Air Force | Imagery with constant range lines |
US4687325A (en) * | 1985-03-28 | 1987-08-18 | General Electric Company | Three-dimensional range camera |
US4687326A (en) * | 1985-11-12 | 1987-08-18 | General Electric Company | Integrated range and luminance camera |
US20050088644A1 (en) * | 2001-04-04 | 2005-04-28 | Morcom Christopher J. | Surface profile measurement |
US20050219552A1 (en) * | 2002-06-07 | 2005-10-06 | Ackerman Jermy D | Methods and systems for laser based real-time structured light depth extraction |
US20050128196A1 (en) * | 2003-10-08 | 2005-06-16 | Popescu Voicu S. | System and method for three dimensional modeling |
US20070268398A1 (en) * | 2006-05-17 | 2007-11-22 | Ramesh Raskar | Apparatus and method for illuminating a scene with multiplexed illumination for motion capture |
US20100074532A1 (en) * | 2006-11-21 | 2010-03-25 | Mantisvision Ltd. | 3d geometric modeling and 3d video content creation |
US20080118143A1 (en) * | 2006-11-21 | 2008-05-22 | Mantis Vision Ltd. | 3D Geometric Modeling And Motion Capture Using Both Single And Dual Imaging |
US20100046801A1 (en) * | 2006-12-25 | 2010-02-25 | Rui Ishiyama | Apparatus, method and program for distance measurement |
US7978928B2 (en) * | 2007-09-18 | 2011-07-12 | Seiko Epson Corporation | View projection for dynamic configurations |
US20110026783A1 (en) * | 2008-04-03 | 2011-02-03 | Kyushu Institute Of Technology | Personal authentication method using subcutaneous bloodstream measurement and personal authentication device |
US20100085425A1 (en) * | 2008-10-07 | 2010-04-08 | Kar-Han Tan | One-Touch Projector Alignment for 3D Stereo Display |
US8294762B2 (en) * | 2008-10-10 | 2012-10-23 | Fujifilm Corporation | Three-dimensional shape measurement photographing apparatus, method, and program |
US8339616B2 (en) * | 2009-03-31 | 2012-12-25 | Micrometric Vision Technologies | Method and apparatus for high-speed unconstrained three-dimensional digitalization |
US20110071675A1 (en) * | 2009-09-22 | 2011-03-24 | Gm Global Technology Operations, Inc. | Visual perception system and method for a humanoid robot |
US20110164805A1 (en) * | 2010-01-07 | 2011-07-07 | De La Rue North America Inc. | Systems and Methods for Detecting an Optically Variable Material |
US20140232566A1 (en) * | 2011-06-17 | 2014-08-21 | Leddartech Inc. | System and method for traffic side detection and characterization |
US20130021474A1 (en) * | 2011-07-20 | 2013-01-24 | Raytheon Company | Rolling-shutter imaging system with synchronized scanning illumination and methods for higher-resolution imaging |
US20130121560A1 (en) * | 2011-11-14 | 2013-05-16 | Ryusuke Hirai | Image processing device, method of processing image, and image display apparatus |
US20130250272A1 (en) * | 2012-03-21 | 2013-09-26 | David Ludwig | Threshold Detection Method, Module and Readout Integrated Circuit Layer for LIDAR Time of Flight System Using Differentiated Gaussian Signal |
US20140211193A1 (en) * | 2012-09-24 | 2014-07-31 | Alces Technology, Inc. | Structured light and time of flight depth capture with a MEMS ribbon linear array spatial light modulator |
US20140253724A1 (en) * | 2013-03-08 | 2014-09-11 | Mitutoyo Corporation | Shape measuring apparatus |
US20140327746A1 (en) * | 2013-05-06 | 2014-11-06 | Iphoton Solutions, Llc | Volume reconstruction of an object using a 3d sensor and robotic coordinates |
US20160349369A1 (en) * | 2014-01-29 | 2016-12-01 | Lg Innotek Co., Ltd. | Device for extracting depth information and method thereof |
US20150285625A1 (en) * | 2014-04-07 | 2015-10-08 | Samsung Electronics Co., Ltd. | High resolution, high frame rate, low power image sensor |
US20150381962A1 (en) * | 2014-06-27 | 2015-12-31 | Electronics And Telecommunications Research Institute | Method and apparauts for implementing active imaging system |
US20160124203A1 (en) * | 2014-11-04 | 2016-05-05 | Olympus Corporation | Microscope and microscope image acquisition method |
US20160182889A1 (en) * | 2014-12-19 | 2016-06-23 | Datalogic ADC, Inc. | Depth camera system using coded structured light |
US20160182788A1 (en) * | 2014-12-22 | 2016-06-23 | Google Inc. | Time-of-flight camera system with scanning illuminator |
US20170064235A1 (en) * | 2015-08-27 | 2017-03-02 | Samsung Electronics Co., Ltd. | Epipolar plane single-pulse indirect tof imaging for automotives |
US10021284B2 (en) * | 2015-08-27 | 2018-07-10 | Samsung Electronics Co., Ltd. | Epipolar plane single-pulse indirect TOF imaging for automotives |
US20170142406A1 (en) * | 2015-11-16 | 2017-05-18 | Samsung Electronics Co., Ltd. | Apparatus for and method of illumination control for acquiring image information and depth information simultaneously |
US20170176575A1 (en) * | 2015-12-18 | 2017-06-22 | Gerard Dirk Smits | Real time position sensing of objects |
US20170353707A1 (en) * | 2016-06-03 | 2017-12-07 | Samsung Electronics Co., Ltd. | Timestamp error correction with double readout for the 3d camera with epipolar line laser point scanning |
US20170366801A1 (en) * | 2016-06-20 | 2017-12-21 | Intel Corporation | Depth image provision apparatus and method |
US20220245762A1 (en) * | 2021-02-03 | 2022-08-04 | Qualcomm Incorporated | High resolution time-of-flight depth imaging |
Non-Patent Citations (3)
Title |
---|
Blais, "A Review of 20 Years of Range Sensor Development˙" , Proceedings of SPIE-IS&T Electronic Imaging, SPIE Vol. 5013 (2003) pages 62-76 (Year: 2003) * |
Horaud et al "An overview of depth cameras and range scanners based on time-of-flight technologies" Machine Vision and Applications (2016) (Year: 2016) * |
S. B. Gokturk, H. Yalcin and C. Bamji, "A Time-Of-Flight Depth Sensor - System Description, Issues and Solutions," 2004 Conference on Computer Vision and Pattern Recognition Workshop, Washington, DC, USA, 2004, pp. 35-35, doi: 10.1109/CVPR.2004.291. (Year: 2004) * |
Also Published As
Publication number | Publication date |
---|---|
US20200018592A1 (en) | 2020-01-16 |
US11747135B2 (en) | 2023-09-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10359277B2 (en) | Imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor | |
US10679370B2 (en) | Energy optimized imaging system with 360 degree field-of-view | |
US20220034651A1 (en) | Energy optimized imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor | |
US11531257B2 (en) | Method, apparatus, and manufacture for a tracking camera or detector with fast asynchronous triggering | |
US10073336B2 (en) | Projection system with safety detection | |
US7428997B2 (en) | Method and apparatus for illuminating a field-of-view and capturing an image | |
US9047514B2 (en) | Apparatus, system and method for projecting images onto predefined portions of objects | |
KR101696630B1 (en) | Lighting system and method for image and object enhancement | |
WO2009142015A1 (en) | Projector | |
JP4709571B2 (en) | Visual information processing system and visual information processing method thereof | |
WO2019184184A1 (en) | Target image acquisition system and method | |
WO2019184185A1 (en) | Target image acquisition system and method | |
WO2019184183A1 (en) | Target image acquisition system and method | |
Kitajima et al. | Simultaneous projection and positioning of laser projector pixels | |
JP2018204976A (en) | Distance measuring apparatus, distance measuring method, and imaging apparatus | |
Henderson et al. | Design and calibration of a fast flying-dot projector for dynamic light transport acquisition | |
Wang | High resolution 2D imaging and 3D scanning with line sensors | |
JP6969606B2 (en) | Projector with detection function | |
Chan et al. | Light-Efficient Holographic Illumination for Continuous-Wave Time-of-Flight Imaging | |
Liu et al. | Improved flying-spot projector design for dense dynamic light-transport capture |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: THE GOVERNING COUNCIL OF THE UNIVERSITY OF TORONTO, CANADA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:O'TOOLE, MATTHEW;KUTULAKOS, KIRIAKOS NEOKLIS;SIGNING DATES FROM 20171116 TO 20180116;REEL/FRAME:057714/0726 Owner name: CARNEGIE MELLON UNIVERSITY, PENNSYLVANIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:NARASIMHAN, SRINIVASA;ACHAR, SUPREETH;SIGNING DATES FROM 20171026 TO 20180725;REEL/FRAME:057714/0633 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION COUNTED, NOT YET MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |