US9380224B2 - Depth sensing using an infrared camera - Google Patents
Depth sensing using an infrared camera Download PDFInfo
- Publication number
- US9380224B2 US9380224B2 US14/193,686 US201414193686A US9380224B2 US 9380224 B2 US9380224 B2 US 9380224B2 US 201414193686 A US201414193686 A US 201414193686A US 9380224 B2 US9380224 B2 US 9380224B2
- Authority
- US
- United States
- Prior art keywords
- machine learning
- image
- depth
- learning component
- trained machine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 claims abstract description 120
- 238000007637 random forest analysis Methods 0.000 claims abstract description 48
- 238000010801 machine learning Methods 0.000 claims abstract description 42
- 230000008569 process Effects 0.000 claims description 17
- 238000012545 processing Methods 0.000 claims description 7
- 238000005286 illumination Methods 0.000 claims description 6
- 239000000523 sample Substances 0.000 claims description 5
- 230000006835 compression Effects 0.000 claims description 2
- 238000007906 compression Methods 0.000 claims description 2
- 230000011218 segmentation Effects 0.000 claims description 2
- 230000002123 temporal effect Effects 0.000 claims 1
- 238000012549 training Methods 0.000 description 34
- 238000003066 decision tree Methods 0.000 description 27
- 238000012360 testing method Methods 0.000 description 26
- 239000010410 layer Substances 0.000 description 24
- 238000003860 storage Methods 0.000 description 21
- 230000006870 function Effects 0.000 description 18
- 238000010586 diagram Methods 0.000 description 16
- 238000004891 communication Methods 0.000 description 14
- 238000009826 distribution Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- 230000000644 propagated effect Effects 0.000 description 5
- 238000003384 imaging method Methods 0.000 description 4
- 238000004590 computer program Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 210000004247 hand Anatomy 0.000 description 2
- 238000010191 image analysis Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 239000002356 single layer Substances 0.000 description 2
- 101000822695 Clostridium perfringens (strain 13 / Type A) Small, acid-soluble spore protein C1 Proteins 0.000 description 1
- 101000655262 Clostridium perfringens (strain 13 / Type A) Small, acid-soluble spore protein C2 Proteins 0.000 description 1
- 241000699666 Mus <mouse, genus> Species 0.000 description 1
- 241000699670 Mus sp. Species 0.000 description 1
- 101000655256 Paraclostridium bifermentans Small, acid-soluble spore protein alpha Proteins 0.000 description 1
- 101000655264 Paraclostridium bifermentans Small, acid-soluble spore protein beta Proteins 0.000 description 1
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000007177 brain activity Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000005684 electric field Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 210000003813 thumb Anatomy 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
- 230000037303 wrinkles Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/30—Transforming light or analogous information into electric information
- H04N5/33—Transforming infrared radiation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/231—Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- G06K9/00201—
-
- G06K9/2018—
-
- G06K9/6219—
-
- G06K9/6282—
-
- G06T7/0057—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/143—Sensing or illuminating at different wavelengths
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
- G06V10/7625—Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/20—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from infrared radiation only
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Definitions
- a method of sensing depth using an infrared camera is described.
- an infrared image of a scene is received from an infrared camera.
- the infrared image is applied to a trained machine learning component which uses the intensity of image elements to assign all or some of the image elements a depth value which represents the distance between the surface depicted by the image element and the infrared camera.
- the machine learning component comprises one or more random decision forests.
- FIG. 1 is a schematic diagram of a depth sensing system
- FIG. 2 is a schematic diagram of the IR camera and computing-based device of FIG. 1 ;
- FIG. 3 is a flow chart of a method of converting an RGB camera into an IR camera
- FIG. 4 is a schematic diagram of the depth estimation engine of FIG. 2 ;
- FIG. 5 is a schematic diagram of method of generating a depth map from an IR image using one or more random decision forests
- FIG. 6 is a schematic diagram of an apparatus for generating training data for a random decision forest
- FIG. 7 is a schematic diagram of a random decision forest
- FIG. 8 is a flow diagram of generating a depth map from an IR image using a multi-layer decision forest
- FIG. 9 is a schematic diagram of an example multi-layer decision forest
- FIG. 10 is a flow diagram of a method of performing depth classification using a random decision forest
- FIG. 11 is a flow diagram of a method of performing depth regression using a random decision forests
- FIG. 12 is a flow diagram of a method of training a random decision forest
- FIG. 13 is a schematic diagram of a reference image element and two probe image elements.
- FIG. 14 illustrates an exemplary computing-based device in which embodiments of the systems and methods described herein may be implemented.
- Described herein are systems and methods for obtaining depth information for a scene using a simple infrared (IR) camera.
- the system implements machine learning to correlate the intensity of an image element (and that of its neighbors) and its three-dimensional (3D) depth.
- the IR camera may be a specialized IR camera or may be a modified conventional, monocular camera. Such a system allows depth information to be obtained in a very cost-effective and efficient manner allowing it to be implemented in low cost and low power devices such as mobile phones.
- the system 100 comprises an IR camera 102 arranged to capture one or more IR images of a scene comprising one or more surfaces; and a computing-based device 104 in communication with the IR camera 102 configured to generate depth maps from the IR images without input from a depth camera.
- the system 100 may generate one depth map for each IR image generated by the IR camera 102 .
- the system 100 may combine information from multiple IR images to generate a single depth map; interpolate between successive IR images to generate multiple depth maps; and/or combine IR images from multiple IR cameras to produce a single depth map.
- the IR camera 102 is mounted on a display screen 108 above and pointing horizontally at the scene including the user 110 .
- the IR camera 102 may be embedded within or mounted on any other suitable object in the environment (e.g. within display screen 108 or computing-based device 104 ).
- the computing-based device 104 shown in FIG. 1 is a traditional desktop computer with a separate processor component 106 and display screen 108 , however, the methods and systems described herein may equally be applied to computing-based devices 102 wherein the processor component 106 and display screen 108 are integrated such as in a laptop computer, tablet computer or smart phone.
- the processor component 106 and display screen 108 are integrated such as in a laptop computer, tablet computer or smart phone.
- FIG. 1 comprises a person 110 at a desk
- a person of skill in the art will appreciate that the methods and systems described herein may be equally applied to other scenes.
- FIG. 2 illustrates a schematic diagram of an IR camera 102 that may be used in the system 100 of FIG. 1 .
- the IR camera 102 comprises at least one IR imaging sensor 202 for capturing IR images of the scene 104 and a diffuse IR illuminant 204 arranged to actively illuminate the scene.
- the diffuse IR illuminate comprises multiple (e.g. 6 ) IR light emitting diodes (LED) around the IR imaging sensor 202 . This reduces shadowing with a minimal baseline and improves lighting uniformity.
- the IR illuminant 204 may be pulsed so that differences between successive images may be used to remove ambient illumination.
- the IR camera 102 may also comprise at least one processor 206 , which is in communication with the IR imaging sensor 202 and the IR illuminant 204 .
- the processor 206 may be a general purpose microprocessor or a specialized signal/image processor.
- the processor 206 is arranged to execute instructions to control the IR imaging sensor 202 and IR illuminant 204 to capture IR images.
- the processor 206 may optionally be arranged to perform processing on these images and signals, as outlined in more detail below.
- the IR camera 102 may also include memory 208 arranged to store the instructions for execution by the processor 206 , images or frames captured by the IR camera 202 , or any suitable information, images or the like.
- the memory 208 can include random access memory (RAM), read only memory (ROM), cache, Flash memory, a hard disk, or any other suitable storage component.
- RAM random access memory
- ROM read only memory
- cache Flash memory
- hard disk or any other suitable storage component.
- the memory 208 can be a separate component in communication with the processor 206 or integrated into the processor 206 .
- the IR camera 102 may also include an output interface 210 in communication with the processor 206 .
- the output interface 210 is arranged to provide the image data to the computing-based device 104 via a communication link.
- the communication link can be, for example, a wired connection (e.g. USBTM, FirewireTM, EthernetTM or similar) and/or a wireless connection (e.g. WiFiTM, BluetoothTM or similar).
- the output interface 210 can interface with one or more communication networks (e.g. the Internet) and provide data to the computing-based device 104 via these networks.
- the computing-based device 104 may comprise a depth estimation engine 212 that is configured to generate a depth map from the image data received from the IR camera 102 .
- the depth map is generated from a trained machine learning component that has been trained to map the intensity of an image element (and surrounding image elements) to a depth value.
- the depth map comprises, for each image element of the IR image, a depth value that represents the absolute distance between the surface in the scene depicted by the individual image element and the IR camera 102 .
- An image element is a unit of an image such as a pixel, a voxel, a group of pixels or voxels.
- Application software 214 may also be executed on the computing-based device 108 which may use the output of the depth estimation engine 212 (e.g. depth map).
- the computing-based device 104 may comprise a gesture recognition engine which uses the depth map to identify gestures performed by the user which then may be used to control the operation of the computing-based device.
- the IR camera 102 may be a dedicated IR camera or it may be a regular commodity camera, such as an RGB camera, that has been modified to allow it to operate as an IR camera.
- FIG. 3 illustrates an example method 300 for modifying a regular commodity camera, such as an RGB camera, to operate as an IR camera.
- the IR cut filter that is typically present is removed. Once the IR cut filter is removed, the method 300 proceeds to block 304 .
- an IR bandpass filter is added. This turns the regular commodity camera into an IR camera (i.e. a camera capable of generating IR images of a scene). Once the IR bandpass filter is added, the method 300 proceeds to block 306 .
- a diffuse IR illuminant is added to illuminate the scene.
- a ring of LEDs are built around the camera lens. Since a typical LED has a limited beam angle with significant attenuation away from its main optical direction, a ring of LEDs reduces shadowing and improves uniformity of lighting.
- the modified camera 102 of FIG. 1 comprises six diffuse IR LEDs, however, any suitable number of LEDs or other diffuse IR illuminants may be used.
- This conversion method produces an extremely inexpensive depth camera as compared to stereo, structured light of time of flight (ToF) cameras. It also allows the IR camera to have a very small form factor which allows it to be embedded into relatively small devices, such as mobile phones.
- FIG. 4 is a schematic diagram of the depth estimation engine 212 of FIG. 2 .
- the depth estimation engine 212 receives an IR image 402 and applies it to a trained machine learning component 404 to produce a depth map 406 .
- the trained machine learning component may comprise one or more random decision forests.
- other suitable machine learning components may be used such as, but not limited to, a deep neural network, a support vector regressor, and a Gaussian process regressor.
- the depth map comprises a depth value for each image element of the IR image 402 .
- the depth value represents the absolute distance between the surface represented by the image element in the image and the IR camera 102 .
- the depth estimation engine 212 of FIG. 2 may be integral with, or in communication with functionality that uses the depth image.
- any of a video compression 408 , segmentation 410 (e.g. for background removal); gesture recognition (e.g. to identify gestures performed by the user); natural user interface 414 (e.g. to control the operation of a computer in conjunction with for example gesture recognition), surface reconstruction 416 and object scanning 418 may be configured to use the depth map generated by the depth estimation engine 212 .
- FIG. 5 illustrates an example method 500 for generating a depth map 502 from an IR image 504 wherein the trained machine learning component 404 comprises one or more trained random decision forests 506 .
- the random decision forests 506 are trained to map a given image element (e.g. pixel) in an IR image to an absolute depth value. As described above the depth value represents the distance between the surface represented by the image element and the IR camera 102 .
- the random decision forests 506 may be created and trained in an offline process 508 that receives as training data 510 pairs 512 of IR images 514 and corresponding depth maps 516 .
- An example method for training a random decision forest 506 will be described with reference to FIG. 12 .
- the random decision forests 506 may be stored at the computing-based device 104 or any other entity in the system or elsewhere in communication with the computing-based device 104 .
- the trained random decision forests 506 can be applied 518 to an IR image 504 to produce a depth map 502 .
- the trained random decision forests 506 generate a depth map in a fast, simple manner which is not computationally expensive and which may be performed in real time or near real time on a live video feed from the IR camera 102 of FIG. 1 even using conventional computing hardware in a single-threaded implementation or in parallel on, for example a mobile graphics processing unit (GPU).
- GPU mobile graphics processing unit
- a training data generator 602 which is computer implemented, generates training data which comprises many pairs 604 of data, each pair 604 comprising an IR image 606 and a ground truth depth map 608 corresponding to the IR image 606 where each image element of the IR image 606 has an associated depth value in the depth map 608 .
- the variety of objects in the training images and configuration and orientations of those objects is as wide as possible according to the application domain, storage and computing resources available.
- the pairs of IR image and depth maps 604 may be generated from a real physical setup.
- the corresponding IR image and depth maps may be obtained from an IR camera 610 and a depth camera 612 that are mounted side by side so as to capture intensity (IR) and ground truth depth information simultaneously.
- the corresponding IR images and depth maps may be obtained from a time-of-flight depth sensor that provides registered and synchronized infrared and depth images.
- the real infrared images are pre-processed by applying a fixed intensity threshold to segment objects (e.g. hand or face) from the background. This removes the need to train with varied backgrounds, reduces the compute load, and works well in modulo extreme ambient illumination.
- the pairs of IR image and depth maps 604 may also, or alternatively, be synthetically generated using computer graphics techniques.
- a computer system 602 may have access to a virtual 3D model 614 of an object and to a rendering tool 616 .
- the rendering tool 616 may be arranged to automatically generate a plurality of high quality IR images and ground truth depth maps.
- the computer system 602 may be used to simulate ambient lighting conditions to allow the system 100 to learn invariance to these conditions.
- the computer system 602 may also be used to simulate variations in the surface texture of the objects (e.g. hands and face) that are being rendered to allow the system 100 to learn invariance to different skin colors, wrinkles and/or facial hair.
- FIG. 7 is a schematic diagram of a random decision forest comprising three random decision trees 702 , 704 and 706 . Two or more random decision trees may be used. Three are shown in this example for clarity.
- a random decision tree is a type of data structure used to store data accumulated during a training phase so that it may be used to make predictions about examples previously unseen by the random decision tree.
- a random decision tree is usually used as part of an ensemble of random decision trees (referred to as a forest) trained for a particular application domain in order to achieve generalization (that is being able to make good predictions about examples which are unlike those used to train the forest).
- a random decision tree has a root node 708 , a plurality of split nodes 710 and a plurality of leaf nodes 712 .
- the structure of the tree (the number of nodes and how they are connected) is learned as well as split functions to be used at each of the split nodes.
- data is accumulated at the leaf nodes during training.
- Image elements of an IR image may be pushed through trees of a random decision forest from the root to a leaf node in a process whereby a decision is made at each split node.
- the decision is made according to characteristics of the image element being classified and characteristics of image elements displaced from the original image element by spatial offsets specified by the parameters of the split node.
- parameter values also referred to as features
- data comprising part and state label votes are accumulated at the leaf nodes.
- the machine learning component comprises one or more of the following to reduce the amount of memory required by the machine learning component: a random decision forest with merged nodes, a random decision forest with auto-context, an entangled random decision forest and a multi-layered decision forest.
- an entangled random decision forest is a random decision forest where at least one decision tree has split nodes at a specified level which accumulate data during training which is issued to derive features for making decision at at least one lower level of the tree.
- a plurality of random decision forests have auto-context where output from one random decision forest is available to enable decisions to be made at split nodes in at least one other random decision forest.
- the output of one random decision forest is used to select subsequent random decision forests and/or outputs from subsequent random decision forests.
- the memory required for the machine learning component can be reduced or its accuracy can be increased by using a multi-layered decision tree.
- the problem can be significantly simplified by restricting the depths of the objects to a certain range.
- an expert forest can be trained to regress continuous and absolute depth values more efficiently.
- the machine learning component comprises a two-layer decision tree where the first layer classifies the image element into one of a plurality of depth ranges or bins.
- the second layer one or more expert regression decision forests which are trained specifically for the estimated depth range are applied to the image element. The results may then be aggregated to obtain a final estimation for the absolute depth of the image element.
- FIG. 8 illustrates a method 800 for estimating the depth value for an IR image element using a multi-layered decision forest.
- the depth estimation engine 212 receives an IR image element. Once the IR image element has been received the method 800 proceeds to block 804 .
- the received IR image element is applied to a classification forest to classify the image element into one of a plurality of depth ranges or bins.
- the classification forest at the first layer infers a probability distribution p(c
- the forest learns to map the image element and its spatial context (i.e. image elements surrounding the image element) into one of the depth bins for each image element.
- the number of depth ranges or bins C may be manually selected, for example, by experimenting on synthetic and/or real images.
- the classification forest output the probability distribution p which specifies the probability that the image element received in block 802 has a depth value within each of the depth ranges or bins.
- the expert regression forests for the second layer of the multi-layered decision forest method are selected.
- the expert forests for the second layer are chosen based on the local estimate of c (the estimate for the particular image element). This is referred to as the local expert network (LEN).
- x, I) are aggregated (and optionally averaged) over all the image elements to form a more robust estimate p(c
- the expert forests are then selected based on the GEN. Once the expert regression forests have been selected, the method 800 proceeds to block 810 .
- the received IR image element is applied to the expert regression forests selected at block 808 . Specifically each selected expert regression forest is evaluated to form a set of absolute depth estimates. Once the image element has been pushed through the selected expert forests the method 800 proceeds to block 812 .
- a depth value is assigned to the image element.
- the depth value y is a weighted sum over the estimates y c of the expert regression forests, where the weights ⁇ c are the posterior probabilities estimated in blocks 804 and 806 by the first layer.
- the depth value may be calculated from formula (1) shown below.
- ⁇ c 1 C ⁇ ⁇ c ⁇ y c ⁇ ( x
- ⁇ c can either be the local posterior p(c
- GEN is typically more costly than LEN due to the extra pooling step after the first layer, but is generally more robust (i.e. it produces more consistently accurate results).
- a threshold may be applied to the posteriors to select a subset of the expert regression trees instead of triggering all the expert regression trees.
- k expert regression trees are selected where k is a fixed number. This may make GEN faster than LEN, if for example only k forests can fit into the cache at once since GEN evaluates the same forests for all the image elements, whereas LEN may choose a different set of k forests for each image element.
- a multi-layer decision forest may reduce the memory required to achieve a certain accuracy level over a single layer decision forest.
- the primary task is simplified which increases the accuracy or reduced memory consumption compared to a single-layer forest.
- a multi-layered forest completes the same task as single-layered forest with C+1 forests instead of one where C is the number of depth ranges or bins.
- the task is simplified for the second-layer or expert forests they are typically more shallow (i.e. have less levels) than a single-layered forest.
- the reduction in complexity typically more than compensates for the increase in the number of trees.
- FIG. 9 is a schematic diagram of an example multi-layer decision forest 900 .
- the multi-layer forest comprises a first layer that has a single depth classification forest 902 and a second layer that has C depth regression forests 904 , 906 where C is the number of depth ranges or bins.
- the output of each leaf of the depth classification forest 902 is the local posterior p(c
- the local posterior will have four values, one for each depth range or bin, indicating the likelihood that the depth of the image element falls in the associated depth range or bin.
- the local posterior may be represented by a histogram as shown in FIG. 9 .
- the output of each leaf of the depth regression forests 904 , 906 is an absolute depth value.
- the depth value represents the depth in millimeters (mm). However, it will be evident to a person of skill in the art that the depth value may be represented in other measurement units.
- FIG. 10 illustrates an example method 1000 for implementing the first layer of the multi-layer decision forest to classify an IR image element into one of a plurality of depth ranges or bins (e.g. block 804 of method 800 ).
- the method 1000 is described as being executed by the classifier engine 212 of FIG. 2 , in other examples all or part of the method may be executed by another component of the system described herein.
- the depth estimation engine 212 receives an image element to be classified.
- the depth estimation engine 212 may be configured to classify each image element in the IR image.
- the depth estimation engine 212 may be configured to classify only a subset of the image elements. In these examples, the depth estimation engine 212 may use a predetermined set of criteria for selecting the image elements to be classified.
- the depth estimation engine 214 selects a decision tree from the classifier decision forest. Once a decision tree has been selected, the method 1000 proceeds to block 1006 .
- the depth estimation engine 212 pushes the image element through the decision tree selected in block 1004 , such that it is tested against the trained parameters at a node, and then passed to the appropriate child in dependence on the outcome of the test, and the process is repeated until the image element reaches a leaf node. Once the image element reaches a leaf node, the method 1000 proceeds to block 1008 .
- the classifier engine 214 stores the accumulated votes for each depth range or bin associated with the end leaf node.
- the votes may be in the form of a histogram or any other suitable form. Once the accumulated votes are stored the method 1000 proceeds to block 1010 .
- the depth estimation engine 212 determines whether there are more decision trees in the forest. If it is determined that there are more decision trees in the forest then the method 1000 proceeds back to block 1004 where another decision tree is selected. This is repeated until it has been performed for all the decision trees in the forest and then the method ends 1012 .
- the individual tree distributions are averaged together to form the forest output. In other cases the individual tree distributions may be combined in another suitable manner.
- process for pushing an image element through the plurality of tress in the decision forest may be performed in parallel, instead of in sequence as shown in FIG. 10 .
- FIG. 11 illustrates an example method 1100 for implementing the second layer of the multi-layer decision forest to determine the depth of an image element (e.g. block 810 of FIG. 8 ).
- the method 1100 is described as being executed by the classifier engine 212 of FIG. 2 , in other examples all or part of the method may be executed by another component of the system described herein.
- the depth estimation engine 212 receives an image element of an IR image. Once the depth estimation engine receives the image element the method 1100 proceeds to blocks 1104 .
- the depth estimation engine 212 selects an expert regression forest and at a block 1106 , the depth estimation engine 212 selects a decision tree from the selected expert regression forest. Once a forest and tree have been selected, the method 1000 proceeds to block 1108 .
- the depth estimation engine 212 pushes the image element through the selected decision tree, such that it is tested against the trained parameters at a node, and then passed to the appropriate child in dependence on the outcome of the test, and the process repeated until the image element reaches a leaf node. Once the image element reaches a leaf node, the method 1100 proceeds to block 1110 .
- the classifier engine 214 stores the depth value y associated with the end leaf node. Once the depth value is stored the method 1100 proceeds to block 1112 .
- the depth estimation engine 212 determines whether there are more decision trees in the selected expert forest. If it is determined that there are more decision trees in the selected expert forest then the method 1100 proceeds back to block 1106 where another tree is selected. This is repeated until it has been performed for all the decision trees in the forest and then the method 1100 proceeds to block 1114 .
- the depth estimation engine 212 determines whether there are any more expert forests. If it is determined that there are more expert forests to be applied to the image element then the method proceeds back to block 1104 . This is repeated until each decision forest has been applied to the image element and then the method 1100 ends at block 1116 .
- the mean individual tree depth value y is output for each tree.
- a median filter may then be applied over these predictions within a small patch around the image element x across all trees in the forest resulting in the final image element prediction.
- the image element predictions from each tree may be locally or globally weighted as described above.
- process for pushing an image element through the plurality of trees in the decision forest may be performed in parallel, instead of in sequence as shown in FIG. 11 .
- each forest may be evaluated in parallel instead of in sequence as shown in FIG. 11 .
- FIG. 12 illustrates a flow chart of a method 1200 for training a multi-layer random decision forest to estimate the depth value for an IR image element.
- the random decision forest is trained using a set of training pairs of IR images and depth maps as described above with reference to FIG. 6 .
- the training set of pairs of IR images and depth maps as described above is received. Once the training data has been received, the method 1200 proceeds to block 1204 .
- the number of decision trees to be used in the random decision forest is selected.
- a random decision forest is a collection of deterministic decision trees. Decision tress can suffer from over-fitting, i.e. poor generalization. However, an ensemble of many randomly trained decision trees (a random forest) yields improved generalization. Each tree of the forest is trained. During the training process the number of trees is fixed. Once the number of decision trees has been selected, the method 1200 proceeds to block 1206 .
- a tree from the forest is selected for training. Once a tree has been selected for training, the method 1200 proceeds to block 1208 .
- the root node of the tree selected in block 1206 is selected. Once the root node has been selected, the method 1200 proceeds to block 1210 .
- At block 1210 at least a subset of the image elements from each pair of IR image and depth map is selected for training the tree. Once the image elements from the training pairs to be used for training have been selected, the method 1200 proceeds to block 1212 .
- a random set of test parameters are then used for the binary test performed at the root node as candidate features.
- each root and split node of each tree performs a binary test on the input data and based on the results directs the data to the left (L) or right (R) child node.
- the leaf nodes do not perform any action; they store probability distributions or depth values depending on whether they are part of a depth classifier forest or a depth regression forest.
- the binary test performed at the root node is of the form shown in equation (2). ⁇ ( F ) ⁇ T (2)
- a function ⁇ (F) evaluates a feature F of an image element x to determine if it is greater than a threshold value T. If the function is greater than the threshold value then the result of the binary test is true. Otherwise the result of the binary test is false.
- the binary test of equation (2) is an example only and other suitable binary tests may be used.
- the binary test performed at the root node may evaluate the function to determine if it is greater than a first threshold value T and less than a second threshold value ⁇ .
- a candidate function ⁇ (F) can only make use of image element information which is available at test time.
- the parameter F for the function ⁇ (F) is randomly generated during training.
- the process for generating the parameter F can comprise generating random spatial offset values in the form of a two dimensional displacement.
- the result of the function ⁇ (F) is then computed by observing an image element value for a test data point which is displaced from the data point of interest x in the IR image by the spatial offset.
- equation (3) may be used as the basis of the binary test where ⁇ is as shown in equation (4)
- This function determines the difference in intensity between two image elements spatially offset from the image element of interest x by offsets u and v respectively.
- the image element of interest x will be referred to as the reference image element and the spatially offset image elements (x+u) and (x+v) will be referred to as probe image elements.
- An example of a reference image element x 1302 in an IR image 1300 and its probe image elements 1304 and 1306 are illustrated in FIG. 13 .
- the offsets u and v can be quite large (up to +/ ⁇ 128 pixels in a 640 ⁇ 480 image) and allow the forests to learn about the spatial context in the image.
- the IR image is pre-processed to compute a pseudo depth map D′ using the inverse square law.
- the pseudo depth for image element x may be calculated using equation (5):
- the features may then be selected based on the pseudo depth map D′ rather than (or in addition to) the raw IR image.
- the random set of test parameters comprises a plurality of random values for the function parameter F and the threshold value T.
- a plurality of random values for u, v, and T are generated.
- the function parameters F of each split node are optimized only over a randomly sampled subset of all possible parameters. This is an effective and simple way of injecting randomness into the trees, and increases generalization.
- the method 1200 proceeds to block 1214 .
- every combination of test parameters is applied to each data point selected for training.
- available values for F i.e. u, v
- available values of T for each image element selected for training.
- optimizing criteria are calculated for each combination of test parameters.
- the calculated criteria comprise the information gain (also known as the relative entropy) of the histogram for the depth bins or the density of the depth values.
- the gain Q of a particular combination of test parameters may be calculated using equation (6) where ⁇ is as set out in equation (7):
- E(S) is the Shannon entropy of the empirical distribution of the quantized depth labels c in S as shown in equations (8) and (9):
- E(S) is the differential entropy of the empirical continuous density p(y
- S) is modeled as a one-dimensional Gaussian.
- Other criteria that may be used to assess the quality of the parameters include, but is not limited to, Gini entropy or the ‘two-ing’ criterion.
- the parameters that maximized the criteria e.g. gain
- the method 1200 proceeds to block 1218 .
- the method 1200 proceeds to block 1220 where the current node is set as a leaf node. Similarly, the current depth of the trees is determined (i.e. how many levels of nodes are between the root node and the current node). If this is greater than a predefined maximum value, then the method 1200 proceeds to block 1220 where the current node is set as a leaf node. Once the current node is set to the leaf node, the method 1200 proceeds to block 1228 .
- the value for the calculated criteria e.g. gain
- the method 1200 proceeds to block 1222 where the current node is set to a split node. Once the current node is set to a split node the method 1200 moves to block 1224 .
- the subset of data points sent to each child node of the split nodes is determined using the parameters that optimized the criteria (e.g. gain). Specifically, these parameters are used in the binary test and the binary test is performed on all the training data points. The data points that pass the binary test form a first subset sent to a first child node, and the data points that fail the binary test form a second subset sent to a second child node. Once the subsets of data points have been determined, the method 1200 proceeds to block 1226 .
- the process outlined in blocks 1212 to 1224 is recursively executed for the subset of data points directed to the respective child node.
- new random test parameters are generated, applied to the respective subset of data points, parameters optimizing the criteria selected and the type of node (split or leaf) is determined. Therefore, this process recursively moves through the tree, training each node until leaf nodes are reached at each branch.
- votes may be accumulated at the leaf nodes of the trees. This is the training stage and so particular image elements which reach a given leaf node have depth values known from the ground truth training data. Once the votes are accumulated, the method 1200 proceeds to block 1232 .
- a representation of the accumulated votes may be stored using various different methods. Once the accumulated votes have been stored, the method 1200 proceeds to block 1234 .
- the method 1200 proceeds to block 1206 where the next tree in the decision forest is selected and the process repeats. If all the trees in the forest have been trained, and no others remain, then the training process is complete and the method 1200 terminates at block 1236 .
- FIG. 14 illustrates various components of an exemplary computing-based device 104 which may be implemented as any form of a computing and/or electronic device, and in which embodiments of the systems and methods described herein may be implemented.
- Computing-based device 104 comprises one or more processors 1402 which may be microprocessors, controllers or any other suitable type of processors for processing computer executable instructions to control the operation of the device in order to classify objects in image.
- the processors 1402 may include one or more fixed function blocks (also referred to as accelerators) which implement a part of the method of controlling the computing-based device in hardware (rather than software or firmware).
- Platform software comprising an operating system 1404 or any other suitable platform software may be provided at the computing-based device to enable application software 214 to be executed on the device.
- Computer-readable media may include, for example, computer storage media such as memory 1406 and communications media.
- Computer storage media, such as memory 1406 includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
- Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing-based device.
- communication media may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transport mechanism.
- computer storage media does not include communication media. Therefore, a computer storage medium should not be interpreted to be a propagating signal per se. Propagated signals may be present in a computer storage media, but propagated signals per se are not examples of computer storage media.
- the computer storage media memory 1406
- the storage may be distributed or located remotely and accessed via a network or other communication link (e.g. using communication interface 1408 ).
- the computing-based device 104 also comprises an input/output controller 1410 arranged to output display information to a display device 108 ( FIG. 1 ) which may be separate from or integral to the computing-based device 108 .
- the display information may provide a graphical user interface.
- the input/output controller 1410 is also arranged to receive and process input from one or more devices, such as a user input device (e.g. a mouse, keyboard, camera, microphone or other sensor).
- the user input device may detect voice input, user gestures or other user actions and may provide a natural user interface (NUI).
- NUI natural user interface
- the display device 108 may also act as the user input device if it is a touch sensitive display device.
- the input/output controller 1410 may also output data to devices other than the display device, e.g. a locally connected printing device (not shown in FIG. 14 ).
- the input/output controller 1410 , display device 108 and optionally the user input device may comprise NUI technology which enables a user to interact with the computing-based device in a natural manner, free from artificial constraints imposed by input devices such as mice, keyboards, remote controls and the like.
- NUI technology examples include but are not limited to those relying on voice and/or speech recognition, touch and/or stylus recognition (touch sensitive displays), gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, and machine intelligence.
- NUI technology examples include intention and goal understanding systems, motion gesture detection systems using depth cameras (such as stereoscopic camera systems, infrared camera systems, RGB camera systems and combinations of these), motion gesture detection using accelerometers/gyroscopes, facial recognition, 3D displays, head, eye and gaze tracking, immersive augmented reality and virtual reality systems and technologies for sensing brain activity using electric field sensing electrodes (EEG and related methods).
- depth cameras such as stereoscopic camera systems, infrared camera systems, RGB camera systems and combinations of these
- motion gesture detection using accelerometers/gyroscopes such as stereoscopic camera systems, infrared camera systems, RGB camera systems and combinations of these
- motion gesture detection using accelerometers/gyroscopes such as stereoscopic camera systems, infrared camera systems, RGB camera systems and combinations of these
- accelerometers/gyroscopes such as stereoscopic camera systems, infrared camera systems, RGB camera systems and combinations of these
- accelerometers/gyroscopes such
- the functionality described herein can be performed, at least in part, by one or more hardware logic components.
- illustrative types of hardware logic components include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs).
- FPGAs Field-programmable Gate Arrays
- ASICs Program-specific Integrated Circuits
- ASSPs Program-specific Standard Products
- SOCs System-on-a-chip systems
- CPLDs Complex Programmable Logic Devices
- computer or ‘computing-based device’ is used herein to refer to any device with processing capability such that it can execute instructions.
- processors including smart phones
- tablet computers or tablet computers
- set-top boxes media players
- games consoles personal digital assistants and many other devices.
- the methods described herein may be performed by software in machine readable form on a tangible storage medium e.g. in the form of a computer program comprising computer program code means adapted to perform all the steps of any of the methods described herein when the program is run on a computer and where the computer program may be embodied on a computer readable medium.
- tangible storage media include computer storage devices comprising computer-readable media such as disks, thumb drives, memory etc. and do not include propagated signals. Propagated signals may be present in a tangible storage media, but propagated signals per se are not examples of tangible storage media.
- the software can be suitable for execution on a parallel processor or a serial processor such that the method steps may be carried out in any suitable order, or simultaneously.
- a remote computer may store an example of the process described as software.
- a local or terminal computer may access the remote computer and download a part or all of the software to run the program.
- the local computer may download pieces of the software as needed, or execute some software instructions at the local terminal and some at the remote computer (or computer network).
- a dedicated circuit such as a DSP, programmable logic array, or the like.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Databases & Information Systems (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Software Systems (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Optics & Photonics (AREA)
- Image Analysis (AREA)
Abstract
Description
Here ωc can either be the local posterior p(c|x, I) in the case of LEN or the aggregated posterior p(c|I) in the case of GEN. GEN is typically more costly than LEN due to the extra pooling step after the first layer, but is generally more robust (i.e. it produces more consistently accurate results).
ƒ(F)<T (2)
where I is the input infrared image and u and v are 2D image element offsets.
where S is the sample set (the subset of training data used to train the tree), and SL and SR are the two sets of examples formed by the split.
E(S)=log(σs) (9)
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/193,686 US9380224B2 (en) | 2014-02-28 | 2014-02-28 | Depth sensing using an infrared camera |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/193,686 US9380224B2 (en) | 2014-02-28 | 2014-02-28 | Depth sensing using an infrared camera |
Publications (2)
Publication Number | Publication Date |
---|---|
US20150248764A1 US20150248764A1 (en) | 2015-09-03 |
US9380224B2 true US9380224B2 (en) | 2016-06-28 |
Family
ID=54007018
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/193,686 Active US9380224B2 (en) | 2014-02-28 | 2014-02-28 | Depth sensing using an infrared camera |
Country Status (1)
Country | Link |
---|---|
US (1) | US9380224B2 (en) |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108985119A (en) * | 2017-05-31 | 2018-12-11 | 华为技术有限公司 | The method and apparatus of structured light decoding |
US10743389B2 (en) | 2016-09-29 | 2020-08-11 | Signify Holding B.V. | Depth queue by thermal sensing |
US10925465B2 (en) | 2019-04-08 | 2021-02-23 | Activ Surgical, Inc. | Systems and methods for medical imaging |
US11179218B2 (en) | 2018-07-19 | 2021-11-23 | Activ Surgical, Inc. | Systems and methods for multi-modal sensing of depth in vision systems for automated surgical robots |
US11543526B2 (en) | 2020-06-09 | 2023-01-03 | Stmicroelectronics (Research & Development) Limited | Compact depth sensor module |
US11676368B2 (en) | 2020-06-30 | 2023-06-13 | Optum Services (Ireland) Limited | Identifying anomalous activity from thermal images |
US11977218B2 (en) | 2019-08-21 | 2024-05-07 | Activ Surgical, Inc. | Systems and methods for medical imaging |
US12100194B1 (en) | 2021-06-24 | 2024-09-24 | Apple Inc. | Image enhancement |
Families Citing this family (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10062201B2 (en) | 2015-04-21 | 2018-08-28 | Microsoft Technology Licensing, Llc | Time-of-flight simulation of multipath light phenomena |
US10282623B1 (en) * | 2015-09-25 | 2019-05-07 | Apple Inc. | Depth perception sensor data processing |
US9916524B2 (en) * | 2016-02-17 | 2018-03-13 | Microsoft Technology Licensing, Llc | Determining depth from structured light using trained classifiers |
US9760837B1 (en) * | 2016-03-13 | 2017-09-12 | Microsoft Technology Licensing, Llc | Depth from time-of-flight using machine learning |
JP6216842B1 (en) * | 2016-07-08 | 2017-10-18 | Idein株式会社 | Image processing apparatus, image processing method, program, and system |
CN108460464A (en) * | 2017-02-22 | 2018-08-28 | 中兴通讯股份有限公司 | Deep learning training method and device |
US11723579B2 (en) | 2017-09-19 | 2023-08-15 | Neuroenhancement Lab, LLC | Method and apparatus for neuroenhancement |
US11717686B2 (en) | 2017-12-04 | 2023-08-08 | Neuroenhancement Lab, LLC | Method and apparatus for neuroenhancement to facilitate learning and performance |
WO2019133997A1 (en) | 2017-12-31 | 2019-07-04 | Neuroenhancement Lab, LLC | System and method for neuroenhancement to enhance emotional response |
US11429807B2 (en) | 2018-01-12 | 2022-08-30 | Microsoft Technology Licensing, Llc | Automated collection of machine learning training data |
US11481571B2 (en) | 2018-01-12 | 2022-10-25 | Microsoft Technology Licensing, Llc | Automated localized machine learning training |
US10706505B2 (en) * | 2018-01-24 | 2020-07-07 | GM Global Technology Operations LLC | Method and system for generating a range image using sparse depth data |
JP7418340B2 (en) * | 2018-03-13 | 2024-01-19 | マジック リープ, インコーポレイテッド | Image augmented depth sensing using machine learning |
US11364361B2 (en) | 2018-04-20 | 2022-06-21 | Neuroenhancement Lab, LLC | System and method for inducing sleep by transplanting mental states |
WO2020056418A1 (en) | 2018-09-14 | 2020-03-19 | Neuroenhancement Lab, LLC | System and method of improving sleep |
GB2578769B (en) | 2018-11-07 | 2022-07-20 | Advanced Risc Mach Ltd | Data processing systems |
US10602270B1 (en) | 2018-11-30 | 2020-03-24 | Microsoft Technology Licensing, Llc | Similarity measure assisted adaptation control |
US11036615B2 (en) | 2018-12-06 | 2021-06-15 | Microsoft Technology Licensing, Llc | Automatically performing and evaluating pilot testing of software |
US10776000B2 (en) | 2018-12-19 | 2020-09-15 | Microsoft Technology Licensing, Llc. | System and method of receiving and converting digital ink input |
GB2583061B (en) * | 2019-02-12 | 2023-03-15 | Advanced Risc Mach Ltd | Data processing systems |
EP3706268B1 (en) * | 2019-03-07 | 2022-06-29 | ABB Schweiz AG | Artificial intelligence monitoring system using infrared images to identify hotspots in a switchgear |
US11328004B2 (en) | 2019-03-22 | 2022-05-10 | Microsoft Technology Licensing, Llc | Method and system for intelligently suggesting tags for documents |
US11786694B2 (en) | 2019-05-24 | 2023-10-17 | NeuroLight, Inc. | Device, method, and app for facilitating sleep |
US12079569B2 (en) | 2019-06-07 | 2024-09-03 | Microsoft Technology Licensing, Llc | Document editing models and localized content management |
USD963407S1 (en) | 2019-06-24 | 2022-09-13 | Accenture Global Solutions Limited | Beverage dispensing machine |
US10947103B2 (en) | 2019-06-24 | 2021-03-16 | Accenture Global Solutions Limited | Beverage dispensing machine for achieving target pours for different beverages |
US10726246B1 (en) | 2019-06-24 | 2020-07-28 | Accenture Global Solutions Limited | Automated vending machine with customer and identification authentication |
US11308430B2 (en) | 2019-10-11 | 2022-04-19 | Microsoft Technology Licensing, Llc | Keeping track of important tasks |
US11710247B2 (en) | 2020-01-30 | 2023-07-25 | Unity Technologies Sf | System for image compositing including training with synthetic data |
WO2021154099A1 (en) * | 2020-01-30 | 2021-08-05 | Weta Digital Limited | System for image compositing including training with synthetic data |
EP3869395A1 (en) | 2020-02-21 | 2021-08-25 | Accenture Global Solutions Limited | Identity and liveness verification |
US11488317B2 (en) | 2020-11-23 | 2022-11-01 | Sony Group Corporation | Neural network model based depth estimation |
US11475631B2 (en) | 2020-11-23 | 2022-10-18 | Sony Corporation | Training dataset generation for depth measurement |
Citations (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040101043A1 (en) * | 2002-11-25 | 2004-05-27 | Dynamic Digital Depth Research Pty Ltd | Image encoding system |
US6862564B1 (en) * | 2000-10-26 | 2005-03-01 | Sycamore Networks, Inc. | Network emulator |
US20090086046A1 (en) * | 2007-08-31 | 2009-04-02 | Historx, Inc. | Automatic exposure time selection for imaging tissue |
US20120280897A1 (en) * | 2011-05-02 | 2012-11-08 | Microsoft Corporation | Attribute State Classification |
US20120287401A1 (en) * | 2011-05-09 | 2012-11-15 | Carl Zeiss Meditec, Inc. | Integration and fusion of data from diagnostic measurements for glaucoma detection and progression analysis |
US20130342527A1 (en) * | 2012-06-21 | 2013-12-26 | Microsoft Corporation | Avatar construction using depth camera |
EP2701138A1 (en) * | 2012-08-22 | 2014-02-26 | Technische Hochshule Mittelhessen | Sensor based interactive visual aid |
US20140104387A1 (en) * | 2012-10-17 | 2014-04-17 | DotProduct LLC | Handheld portable optical scanner and method of using |
US20140122381A1 (en) * | 2012-10-25 | 2014-05-01 | Microsoft Corporation | Decision tree training in machine learning |
US20140172753A1 (en) * | 2012-12-14 | 2014-06-19 | Microsoft Corporation | Resource allocation for machine learning |
US8872111B2 (en) * | 2011-02-04 | 2014-10-28 | Raytheon Company | Infrared spatial modulator for scene-based non-uniformity image correction and systems and methods related thereto |
US20150058337A1 (en) * | 2013-08-20 | 2015-02-26 | Microsoft Corporation | Database access |
US20150062558A1 (en) * | 2013-09-05 | 2015-03-05 | Texas Instruments Incorporated | Time-of-Flight (TOF) Assisted Structured Light Imaging |
US20150123901A1 (en) * | 2013-11-04 | 2015-05-07 | Microsoft Corporation | Gesture disambiguation using orientation information |
US20150145985A1 (en) * | 2013-11-26 | 2015-05-28 | Michael Jason Gourlay | Large-Scale Surface Reconstruction That Is Robust Against Tracking And Mapping Errors |
US20150169169A1 (en) * | 2013-12-18 | 2015-06-18 | Flir Systems Ab | Processing an infrared (ir) image based on swipe gestures |
US20150181099A1 (en) * | 2012-08-30 | 2015-06-25 | Softkinetic Sensors Nv | Tof illumination system and tof camera and method for operating, with control means for driving electronic devices located in the scene |
-
2014
- 2014-02-28 US US14/193,686 patent/US9380224B2/en active Active
Patent Citations (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6862564B1 (en) * | 2000-10-26 | 2005-03-01 | Sycamore Networks, Inc. | Network emulator |
US20040101043A1 (en) * | 2002-11-25 | 2004-05-27 | Dynamic Digital Depth Research Pty Ltd | Image encoding system |
US20090086046A1 (en) * | 2007-08-31 | 2009-04-02 | Historx, Inc. | Automatic exposure time selection for imaging tissue |
US8872111B2 (en) * | 2011-02-04 | 2014-10-28 | Raytheon Company | Infrared spatial modulator for scene-based non-uniformity image correction and systems and methods related thereto |
US20120280897A1 (en) * | 2011-05-02 | 2012-11-08 | Microsoft Corporation | Attribute State Classification |
US20120287401A1 (en) * | 2011-05-09 | 2012-11-15 | Carl Zeiss Meditec, Inc. | Integration and fusion of data from diagnostic measurements for glaucoma detection and progression analysis |
US20130342527A1 (en) * | 2012-06-21 | 2013-12-26 | Microsoft Corporation | Avatar construction using depth camera |
EP2701138A1 (en) * | 2012-08-22 | 2014-02-26 | Technische Hochshule Mittelhessen | Sensor based interactive visual aid |
US20150181099A1 (en) * | 2012-08-30 | 2015-06-25 | Softkinetic Sensors Nv | Tof illumination system and tof camera and method for operating, with control means for driving electronic devices located in the scene |
US20140104387A1 (en) * | 2012-10-17 | 2014-04-17 | DotProduct LLC | Handheld portable optical scanner and method of using |
US20140122381A1 (en) * | 2012-10-25 | 2014-05-01 | Microsoft Corporation | Decision tree training in machine learning |
US20140172753A1 (en) * | 2012-12-14 | 2014-06-19 | Microsoft Corporation | Resource allocation for machine learning |
US20150058337A1 (en) * | 2013-08-20 | 2015-02-26 | Microsoft Corporation | Database access |
US20150062558A1 (en) * | 2013-09-05 | 2015-03-05 | Texas Instruments Incorporated | Time-of-Flight (TOF) Assisted Structured Light Imaging |
US20150123901A1 (en) * | 2013-11-04 | 2015-05-07 | Microsoft Corporation | Gesture disambiguation using orientation information |
US20150145985A1 (en) * | 2013-11-26 | 2015-05-28 | Michael Jason Gourlay | Large-Scale Surface Reconstruction That Is Robust Against Tracking And Mapping Errors |
US20150169169A1 (en) * | 2013-12-18 | 2015-06-18 | Flir Systems Ab | Processing an infrared (ir) image based on swipe gestures |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10743389B2 (en) | 2016-09-29 | 2020-08-11 | Signify Holding B.V. | Depth queue by thermal sensing |
CN108985119A (en) * | 2017-05-31 | 2018-12-11 | 华为技术有限公司 | The method and apparatus of structured light decoding |
US11179218B2 (en) | 2018-07-19 | 2021-11-23 | Activ Surgical, Inc. | Systems and methods for multi-modal sensing of depth in vision systems for automated surgical robots |
US11857153B2 (en) | 2018-07-19 | 2024-01-02 | Activ Surgical, Inc. | Systems and methods for multi-modal sensing of depth in vision systems for automated surgical robots |
US10925465B2 (en) | 2019-04-08 | 2021-02-23 | Activ Surgical, Inc. | Systems and methods for medical imaging |
US11389051B2 (en) | 2019-04-08 | 2022-07-19 | Activ Surgical, Inc. | Systems and methods for medical imaging |
US11754828B2 (en) | 2019-04-08 | 2023-09-12 | Activ Surgical, Inc. | Systems and methods for medical imaging |
US11977218B2 (en) | 2019-08-21 | 2024-05-07 | Activ Surgical, Inc. | Systems and methods for medical imaging |
US11543526B2 (en) | 2020-06-09 | 2023-01-03 | Stmicroelectronics (Research & Development) Limited | Compact depth sensor module |
US11676368B2 (en) | 2020-06-30 | 2023-06-13 | Optum Services (Ireland) Limited | Identifying anomalous activity from thermal images |
US12100194B1 (en) | 2021-06-24 | 2024-09-24 | Apple Inc. | Image enhancement |
Also Published As
Publication number | Publication date |
---|---|
US20150248764A1 (en) | 2015-09-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9380224B2 (en) | Depth sensing using an infrared camera | |
US9626766B2 (en) | Depth sensing using an RGB camera | |
US11710309B2 (en) | Camera/object pose from predicted coordinates | |
JP6333844B2 (en) | Resource allocation for machine learning | |
CN107466411B (en) | Two-dimensional infrared depth sensing | |
US9373087B2 (en) | Decision tree training in machine learning | |
EP3191989B1 (en) | Video processing for motor task analysis | |
US8571263B2 (en) | Predicting joint positions | |
CN106796656B (en) | Depth from time-of-flight camera | |
US10110881B2 (en) | Model fitting from raw time-of-flight images | |
US20170262768A1 (en) | Depth from time-of-flight using machine learning | |
US20140204013A1 (en) | Part and state detection for gesture recognition | |
US10554957B2 (en) | Learning-based matching for active stereo systems | |
US20150296152A1 (en) | Sensor data filtering | |
US20150199592A1 (en) | Contour-based classification of objects | |
US8879831B2 (en) | Using high-level attributes to guide image processing | |
Favorskaya et al. | Image inpainting based on self-organizing maps by using multi-agent implementation | |
Wu | Depth estimation for indoor single omnidirectional images | |
Klinghoffer | Towards Automated Design of Machine Perception Systems |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MICROSOFT CORPORATION, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:FANELLO, SEAN RYAN FRANCESCO;IZADI, SHAHRAM;KANG, SING BING;AND OTHERS;SIGNING DATES FROM 20140226 TO 20140305;REEL/FRAME:033341/0913 |
|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034747/0417 Effective date: 20141014 Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:039025/0454 Effective date: 20141014 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |