US9270974B2 - Calibration between depth and color sensors for depth cameras - Google Patents

Calibration between depth and color sensors for depth cameras Download PDF

Info

Publication number
US9270974B2
US9270974B2 US13/178,494 US201113178494A US9270974B2 US 9270974 B2 US9270974 B2 US 9270974B2 US 201113178494 A US201113178494 A US 201113178494A US 9270974 B2 US9270974 B2 US 9270974B2
Authority
US
United States
Prior art keywords
color camera
depth sensor
image
depth
planar object
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US13/178,494
Other versions
US20130010079A1 (en
Inventor
Cha Zhang
Zhengyou Zhang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Priority to US13/178,494 priority Critical patent/US9270974B2/en
Assigned to MICROSOFT CORPORATION reassignment MICROSOFT CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ZHANG, CHA, ZHANG, ZHENGYOU
Publication of US20130010079A1 publication Critical patent/US20130010079A1/en
Assigned to MICROSOFT TECHNOLOGY LICENSING, LLC reassignment MICROSOFT TECHNOLOGY LICENSING, LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MICROSOFT CORPORATION
Application granted granted Critical
Publication of US9270974B2 publication Critical patent/US9270974B2/en
Application status is Active legal-status Critical
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • H04N13/0246
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/246Calibration of cameras
    • G06T7/002
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/80Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G06T7/85Stereo camera calibration
    • H04N13/0207
    • H04N13/025
    • H04N13/0271
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/207Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/25Image signal generators using stereoscopic image cameras using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/271Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • H04N13/02
    • H04N13/0257
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/257Colour aspects

Abstract

A system described herein includes a receiver component that receives a first digital image from a color camera, wherein the first digital image comprises a planar object, and a second digital image from a depth sensor, wherein the second digital image comprises the planar object. The system also includes a calibrator component that jointly calibrates the color camera and the depth sensor based at least in part upon the first digital image and the second digital image.

Description

BACKGROUND

Recently there have been an increasing number of depth sensors that are available at relatively low prices. In an example, a sensor unit that communicates with a video game console includes a depth sensor. In another example, computing devices (desktops, laptops, tablet computing devices) are being manufactured with depth sensors therein. A sensor unit that includes both a color camera as well as a depth sensor can be referred to herein as a depth camera. Depth cameras have created a significant amount of interest in applications such as three-dimensional shape scanning, foreground-background segmentation, facial expression tracking, amongst others.

Depth cameras generate simultaneous streams of color images and depth images. To facilitate the applications discussed above (and other applications that employ color images and depth images), the depth sensor and color camera may be desirably calibrated. More specifically, both the color camera and the depth sensor have their own respective coordinate systems, and how such coordinate systems are aligned with respect to one another may be desirably determined to allow pixels in a color image generated by the color camera to be effectively mapped to pixels in a depth image generated by the depth sensor and vice versa.

Many difficulties exist with respect to calibrating a color camera and depth sensor. For example, color cameras have been calibrated utilizing colored patterns. Colored patterns, however, cannot be analyzed in a depth image, as such image does not include captured colors (e.g., corners of a pattern are often indistinguishable from other surface points in a depth image). Furthermore, although depth discontinuity can be observed in a depth image, boundary points of an object are generally unreliable due to unknown depth reconstruction mechanisms utilized in the depth sensor.

An exemplary approach to calibrate a color camera and depth sensor is to co-center an infrared image with a depth image. This may require, however, external infrared illumination. Additionally, commodity depth cameras typically produce relatively noisy depth images, rendering it difficult to calibrate the depth sensor with the color camera.

SUMMARY

The following is a brief summary of subject matter that is described in greater detail herein. This summary is not intended to be limiting as to the scope of the claims.

Described herein are various technologies pertaining to jointly calibrating a color camera and a depth sensor based at least in part upon images of a scene captured by the color camera and the depth sensor, wherein the scene includes a planar object. For instance, the planar object may be a checkerboard. Further, the depth sensor may be any suitable type of depth sensing system, including a triangulation system (such as stereo vision or structured light system), a depth from focus system, a depth from shape system, a depth from motion system, a time of flight system, or other suitable type of depth sensor system.

As will be described in greater detail herein, jointly calibrating the color camera and the depth sensor includes ascertaining a rotation and a translation between coordinate systems of the color camera and the depth sensor, respectively. In connection with computing these values, instructions can be output to a user that instructs the user to move a planar object, such as a checkerboard, to different positions in front of the color camera and the depth sensor. The color camera and the depth sensor may be synchronized, such that an image pair (an image from the color camera and an image from the depth sensor) include the planar object at a particular position and orientation. Rotation and translation between the coordinate systems of the color camera and the depth sensor can be ascertained based at least in part upon a plurality of such image pairs that include the planar object at various positions and orientations.

Two exemplary techniques for ascertaining the rotation and translation between the coordinate systems of the color camera and the depth sensor are described herein. In a first exemplary technique, an image generated by the color camera can be analyzed to locate the known pattern of the planar object that has been captured in such image. Because the pattern in the planar object is known, such planar object can be automatically located in the color image, and the three-dimensional orientation and position of the planar object in the color image can be computed relative to the color camera. A corresponding plane may be then fit into a corresponding image generated by the depth sensor. The plane can be fit based at least in part upon depth values in the image generated by the depth sensor. The plane fit in the image generated by the depth sensor corresponds to the observed plane in the color image after application of a rotation and translation to the plane in the depth image. Through such approach the rotation and translation between the coordinate systems of the color camera and the depth sensor can be computed.

In another exemplary approach, rather than fitting a plane into the depth image, a set of points in the depth image can be randomly sampled. A relatively large number of points in the depth image can be sampled, and at least some of such points will correspond to points of the planar object in the color image by way of a desirably computed rotation and translation between coordinate systems of the color camera and the depth sensor. If a sufficient number of points are sampled, a likelihood function can be learned and evaluated to compute the rotation and translation mentioned above.

Other aspects will be appreciated upon reading and understanding the attached Figs. and description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of an exemplary system that facilitates jointly calibrating a color camera and a depth sensor.

FIG. 2 illustrates coordinate systems of the color camera and the depth sensor.

FIG. 3 is a functional block diagram of an exemplary system that facilitates overlaying a color image onto a depth image based at least in part upon a computed rotation and translation between a color camera and a depth sensor.

FIG. 4 is a flow diagram that illustrates an exemplary methodology for automatically jointly calibrating a color camera and a depth sensor.

FIG. 5 is an exemplary computing system.

DETAILED DESCRIPTION

Various technologies pertaining to jointly calibrating a color camera and a depth sensor will now be described with reference to the drawings, where like reference numerals represent like elements throughout. In addition, several functional block diagrams of exemplary systems are illustrated and described herein for purposes of explanation; however, it is to be understood that functionality that is described as being carried out by certain system components may be performed by multiple components. Similarly, for instance, a component may be configured to perform functionality that is described as being carried out by multiple components. Additionally, as used herein, the term “exemplary” is intended to mean serving as an illustration or example of something, and is not intended to indicate a preference.

As used herein, the terms “component” and “system” are intended to encompass computer-readable data storage that is configured with computer-executable instructions that cause certain functionality to be performed when executed by a processor. The computer-executable instructions may include a routine, a function, or the like. It is also to be understood that a component or system may be localized on a single device or distributed across several devices.

With reference now to FIG. 1, an exemplary system 100 that facilitates jointly calibrating a color camera and depth sensor is illustrated. A combination of a color camera and a depth sensor will be referred to herein as a depth camera. As will be described in greater detail below, jointly calibrating a color camera and a depth sensor may comprise learning a rotation and translation between coordinate systems of the color camera and depth sensor, respectively. The system 100 comprises a receiver component 102 that receives a first digital image from a color camera 104 and a second digital image from a depth sensor 106. In an exemplary embodiment, the first digital image output by the color camera 104 may have a resolution that is the same as the resolution of the second digital image output by the depth sensor 106. Furthermore, the depth sensor 106 may be or include any suitable type of depth sensor system including, but not limited to, a stereo vision or structured light system, a depth from focus system, a depth from shape system, a depth from motion system, a time of flight system, or the like. A clock 108 can be in communication with the color camera 104 and the depth sensor 106, and can assign timestamps to images generated by the color camera 104 and the depth sensor 106, such that images from the color camera 104 and depth sensor 106 that correspond to one another in time can be determined.

In an exemplary embodiment, a housing 110 may comprise the color camera 104, the depth sensor 106, and the clock 108. The housing 110 may be a portion of a sensor that is utilized in connection with a video game console to detect position and motion of a game player. In another exemplary embodiment, the housing 110 may be a portion of a computing system that includes the color camera 104 and the depth sensor 106 for purposes of video-based communications. In still yet another exemplary embodiment, the housing 110 may be for a video camera that is configured to generate three-dimensional video. These embodiments are presented for purposes of explanation and are not intended to limit the scope of the claims. For example, the combination of the color camera 104 and the depth sensor 106 can be utilized in connection with a variety of different types of applications, including three-dimensional shape scanning, foreground-background segmentation, facial expression tracking, three-dimensional image or video generation, amongst others.

Pursuant to an example, the color camera 104 and the depth sensor 106 may be directed at a user 112 that is holding or supporting a planar object 114. In an example, the planar object 114 may be a patterned object such as a game board. For instance, the planar object 114 may be a checkerboard. Moreover, the user 112 can be instructed to move the planar object 114 to a plurality of different locations, and the color camera 104 and the depth sensor 106 can capture images that include the planar object 114 at these various locations.

A calibrator component 116 is in communication with the receiver component 102 and jointly calibrates the color camera 104 and the depth sensor 106 based at least in part upon the first digital image generated by the color camera 104 and the second digital image generated by the depth sensor 106. Pursuant to an example, jointly calibrating the color camera 104 and the depth sensor 106 may comprise computing a rotation and translation between a coordinate system of the color camera 104 and a coordinate system of the depth sensor 106. In other words, the calibrator component 116 can output values that indicate how the color camera 104 is aligned and rotated with respect to the depth sensor 106.

A data store 118 can be accessible to the calibrator component 116, and the calibrator component 116 can cause the rotation and translation to be retained in the data store 118. The data store 118 may be any suitable hardware data store, including a hard drive, memory, or the like. The calibrator component 116 may utilize any suitable technique for jointly calibrating the color camera 104 and the depth sensor 106. In an exemplary embodiment, the calibrator component 116 can have knowledge of the three-dimensional orientation and position of the planar object 114 in the first digital image generated by the color camera 104 based at least in part upon a priori knowledge of the pattern of the planar object 114. As the depth sensor 106 is also directed to capture an image of the planar object 114, the calibrator component 116 can leverage the knowledge of the existence of the planar object 114 in the second digital image generated by the depth sensor 106 to compute the rotation and translation between the coordinate systems of the color camera 104 and the depth sensor 106, respectively. Specifically, the calibrator component 116 can fit a plane that corresponds to the planar object 114 in the image generated by the color camera 104 onto the second digital image generated by the depth sensor 106. Such plane can be fit based at least in part upon three-dimensional points in the second digital image generated by the depth sensor 106. The plane fit onto the image generated by the depth sensor 106 and the plane corresponding to the planar object 114 observed in the first digital image generated by the color camera 104 correspond to one another by the rotation and translation that is desirably computed. The calibrator component 116 can compute such rotation and translation and cause these values to be retained in the data store 118.

In another exemplary embodiment, the calibrator component 116 can randomly sample points in the second digital image generated by the depth sensor 106 that are known to correspond to the planar object 114 in the second digital image. Each randomly sampled point in the image generated by the depth sensor 106 will correspond to a point in the color image that corresponds to the planar object 114. Each point in the image generated by the depth sensor 106 that corresponds to the planar object 114 is related to a point in the image generated by the color camera 104 that corresponds to the planar object 114 by the desirably computed rotation and translation values. If a sufficient number of points are sampled, the calibrator component 116 can compute the values for rotation and translation. Still further, a combination of these approaches can be employed.

Moreover, while the examples provided above have referred to a single image pair (a color image and a depth image), it is to be understood that the calibrator component 116 can consider multiple image pairs with the planar object 114 placed at various different locations and orientations relative to the color camera 104 and the depth sensor 106. For instance, a minimum number of image pairs used by the calibrator component 116 to determine a rotation matrix can be 2, while a minimum number of image pairs used by the calibrator component 116 to determine a translation can be 3. The rotation and translation between the color camera 104 and the depth sensor 106 may then be computed based upon correspondence of the planar object 114 across various color image/depth image pairs.

Further, while the calibrator component 116 has been described above as jointly calibrating the color camera 104 and the depth sensor 106 through analysis of images generated thereby that include the planar object 114, in other exemplary embodiments an object captured in the images need not be entirely planar. For instance, a planar board that includes a plurality of apertures in a pattern can be utilized such that the pattern can be recognized in the first digital image generated by the color camera 104 and the pattern can also be recognized in the second digital image generated by the depth sensor 106. A correspondence between the located patterns in the first digital image and the second digital image may then be employed by the calibrator component 116 to compute the rotation and translation between respective coordinate systems of the color camera 104 and the depth sensor 106.

In yet another exemplary embodiment, the calibrator component 116 can consider point correspondences between the first digital image generated by the color camera 104 and the second digital image generated by the depth sensor 106 in connection with jointly calibrating the color camera 104 and the depth sensor 106. For instance, a user may manually indicate a point in the color image and a point in the depth image, wherein these two points correspond to one another across the images. Additionally or alternatively, image analysis techniques can be employed to automatically locate corresponding points across images generated by the color camera 104 and the depth sensor 106. For instance, the calibrator component 116 can learn a likelihood function that minimizes projected distance between corresponding point pairs across images generated by the color camera 104 and images generated by the depth sensor 106.

In yet another exemplary embodiment, the calibrator component 116 may consider distortion in the depth sensor 106 when jointly calibrating the color camera 104 with the depth sensor 106. For example, depth values generated by the depth sensor 106 may have some distortion associated therewith. A model of such distortion is contemplated and can be utilized by the calibrator component 116 when jointly calibrating the color camera 104 and the depth sensor 106.

With reference now to FIG. 2, an exemplary illustration 200 of existence of the planar object 114 across a plurality of images and notations used to describe a calibration procedure is shown. For purposes of explanation, a three-dimensional coordinate system 202 of the color camera 104 may coincide with a world coordinate system. In a homogeneous representation, a three-dimensional point in the world coordinate system can be denoted by M=[X, Y, Z, 1]T, and its corresponding two-dimensional projection on a model X, Y plane 204 can be denoted m=[u, v, 1]T. The color camera 104 can be modeled by the following pinhole model:
sm=A[I 0]M  (1)
where I is the identity matrix, 0 is the zero vector, and s can be a scale factor. In an exemplary embodiment, s=Z. A is the intrinsic matrix of the color camera 104, which can be given as follows:

A = [ α γ u 0 0 β v 0 0 0 1 ] ( 2 )
where α and β are the scale factors in the image coordinate system, (u0, v0) are the coordinates of the principal point and γ is the skewness of the two image axes.

The depth sensor 106 has a second coordinate system 204 that is different from the coordinate system 202 of the color camera 104. The depth sensor 106 generally outputs an image with depth values denoted by x=[u, v, z]T, where (u, v) are the pixel coordinates, and z is the depth value. The mapping from x to the point in the three-dimensional coordinate system 204 of the depth sensor 106, Md=[Xd, Yd, Zd, 1]1, is usually known, and is denoted as Md=f(x). The rotation and translation between the color camera 104 and the depth camera or depth sensor 106 is denoted by R and t:

M = [ R t 0 T 1 ] M d ( 3 )

As mentioned above, the planar object 114 can be moved in front of the color camera 104 and the depth sensor 106. This can create n image pairs (color and depth) captured by the depth camera (the color camera 104 and the depth sensor 106). As shown, the position of the planar object 114 in the n images will be different. The model plane 204 thus has different positions and orientations relative to the position of the color camera 104. Three-dimensional coordinate systems 203 a-203 b (Xi, Yi, Zi) can be set up for each position of the model plane 204 a and 204 b across the images such that the Zi=0 plane coincides with the model plane 204. Additionally, it can be assumed that the model plane 204 has a set of M feature points. In an example, the feature points can be corners of a known pattern in the planar object 114, such as a checkerboard pattern. The feature points can be denoted as Pj, j=1, . . . , m. It can be noted that the three-dimensional coordinates of such feature points in each model plane's local coordinate system are identical. Each feature point's local three-dimensional coordinate is associated with a corresponding world coordinate as follows:

M i , j = [ R i t i 0 T 1 ] P j , ( 4 )
where Mij is the jth feature point of the ith image in the world coordinate system 202, Ri and ti are the rotation and translation from the ith model plane's local coordinate system 203 a to the world coordinate system 202. The feature points are observed in the color image as mi,j, which are associated with Mi,j through Eq. (1).

Given the set of feature points Pj and their projections mi,j, it is desirable to recover the intrinsic matrix A, the rotations and translations between the models planes 204 a and 204 b and the model plane 204 Ri and ti, and the transform between the color camera 104 and the depth sensor 106 R and t. The intrinsic matrix A and the model plane positions Ri and ti (relative to the global coordinate system 202) can be computed through conventional techniques. Images generated by the depth sensor 106 can be used to compute R and t automatically.

As mentioned previously, the calibration solution for only the color camera 104 is known. Due to the use of the pinhole camera model, the following can be acquired:
sijmij=A[Ri,ti]Pj.  (5)
In practice, feature points on images generated by the color camera 104 are typically extracted automatically through utilization of computer-executable algorithms, and therefore may have errors associated therewith. Accordingly, if it is assumed that Mi,j follows a Gaussian distribution with the ground truth position as its mean, e.g.,
m ij ˜N( m ij, Φij),  (6)
then the log likelihood function can be written as follows:

L 1 = - 1 2 nm i = 1 n j = 1 m ε ij T Φ ij - 1 ε ij , where ( 7 ) ε ij = m ij - 1 s ij A [ R i t i ] P j . ( 8 )

Terms related to images generated by the depth sensor 106 are now discussed. There are a set of points in the image generated by the depth sensor 106 that correspond to the model plane 204. Ki points within the quadrilateral in the depth image can be randomly sampled and denoted by Mik i d, i=1, . . . , n; ki=1, . . . , Ki. If the image generated by the depth sensor 106 (the depth image) is free of noise, the following is obtained:

[ 0 0 1 0 ] [ R i t i 0 T 1 ] - 1 [ R t 0 T 1 ] M ik i d = 0 , ( 9 )
which indicates that if these points are transformed to the local coordinate system of each model plane 204 a-204 b, the coordinate shall be zero.

Since images generated by the depth sensor 106 tend to be noisy, Mik i d can follow a Gaussian distribution as:
m ik i d ˜N( M ik i d, Φik i d).   (10)

The log likelihood function can thus be written as follows:

L 2 = - 1 2 i = 1 n K i i = 1 n k i = 1 K i ɛ ik i 2 σ ik i 2 , where ( 11 ) ɛ ik i = a i T M ik i d , where ( 12 ) a i = [ R T 0 t T 1 ] [ R i 0 - t i T R i 1 ] [ 0 0 1 0 ] , and ( 13 ) σ ik i 2 = a i T Φ ik i d a i . ( 14 )

As mentioned above, it may be helpful to have a plurality of corresponding point pairs in images generated by the color camera 104 and images generated by the depth sensor 106. Such point pairs can be denoted as (mip i , Mip i d), i=1, . . . , n; pi=1, . . . , Pi. Such point pairs shall satisfy the following:
sip i mip i =A[R t]Mip i d.  (15)
Further, whether the point correspondences are manually labeled or automatically established, such point correspondences may not be accurate. According, the following can be assumed:
mip i ˜N( m ip i , Φip i ); Mip i d˜N( M ip i d, Φip i d),   (16)
where Φip i models the inaccuracy of the point in the image generated by the color camera 104, and Φip i d models the uncertainty of the three-dimensional point in the image generated by the depth sensor 106. The log likelihood function can then be written as follows:

L 3 = - 1 2 i = 1 n P i i = 1 n p i = 1 p i ξ ip i T Φ ~ ip i - 1 ξ ip i , where ( 17 ) ξ ip i = m ip i - B ip i M ip i d , where ( 18 ) B ip i = 1 s ip i A [ R t ] , and ( 19 ) Φ ~ ip i = Φ ip i + B ip i Φ ip i d B ip i T . ( 20 )
Combining the above information together, the overall log likelihood can be maximized as follows:
maxA,R i ,t i R,tρ1L12L23L3,  (21)
where ρi, i=1,2,3 are weighting parameters. This objective function can be classified as a nonlinear least squares problem, which can be solved by the calibrator component 116 using the Levenberg-Marquardt method. The result is the computation of the parameters A, Ri, tiR, t.

The above algorithms describe calibration of the color camera 104 and the depth sensor 106 with an assumption of no distortions or noise in either of the color camera 104 or the depth sensor 106. A few other parameters, however, may be desirably estimated during calibration by the calibrator component 116. These parameters can include focus, camera center, and depth mapping function for both the color camera 104 and the depth sensor 106. For instance, the color camera 104 may exhibit lens distortions and thus it may be desirable to estimate such distortions based upon the observed model planes 204 a-204 b in images generated by the color camera 104. Another set of unknown parameters may be in a depth mapping function. For example, an exemplary structured light-based depth camera may have a depth mapping function as follows:

f ( x ) = [ ( μ z + v ) ( A d ) - 1 [ u , v , 1 ] T 1 ] , ( 22 )
where μ and υ are the scale and bias of the z value, and Ad is the intrinsic matrix of the depth sensor 106, which is typically predetermined. The other two parameters μ and υ can be used to model the calibration of the depth sensor 106 due to temperature variation or mechanical vibration, and can be estimated within the same maximum likelihood framework by the calibrator component 116.

The exemplary solution described above pertains to randomly sampling points in the image generated by the depth sensor 106. As discussed, however, the calibrator component 116 can use other approaches as alternatives to the techniques described above or in combination with such techniques. For instance, fitting the model plane 204 a-204 b onto the corresponding image generated by the depth sensor 106 can be undertaken by the calibrator component 116 in connection with calibrating the color camera 104 with the depth sensor 106. In an exemplary embodiment, this plane fitting can be undertaken during initialization to have a first estimate of unknown parameters. For instance, for the parameters related to the color camera 104, e.g., A, Ri, ti, a known initialization scheme can be adapted. Below, methods that can be utilized by the calibrator component 116 to provide an initial estimation of R and t between the color camera 104 and the depth sensor 106 are discussed. During the discussion below, it is assumed that A, Ri and ti of the color camera 104 are known.

For most commodity depth cameras, the color camera 104 and the depth sensor 106 are positioned relatively proximate to one another. Accordingly, it is relatively simple to automatically identify a set of points in each image generated by the depth sensor 106 that lies on the corresponding model plane 204 a-204 b. These points can be referred to as Mik i d, i=1, . . . , n; ki=1, . . . , Ki. For a given image i generated by the depth sensor 106, if Ki≧3, it is possible to fit a plane to the points in that image. In other words, given the following:

H i [ n i d b i d ] = [ ( M i 1 d ) T ( M i 2 d ) T ( M ik i d ) T ] [ n i d b i d ] = 0 , ( 23 )
where ni d is the normal of the model plane in the three-dimensional coordinate system of the depth sensor 106, ∥ni d2=1, and bi d is the bias from the origin. ∥ni d∥ and bi d can be found by the calibrator component 116 through least squares fitting.

In the coordinate system of the color camera 104 (the global coordinate system 202), the model plane can also be described by the following plane equation:

[ 0 0 1 0 ] [ R i t i 0 T 1 ] - 1 M = 0. ( 24 )
Since Ri and ti are known, the plane's normal can be represented as ni, ∥ni2=1, and bias from the origin bi.

The rotation matrix R may first be solved. For instance, R can be denoted as follows:

R = [ r 1 T r 2 T r 3 T ] . ( 25 )
The following objective function may then be minimized with constraint:
J(R)=Σi=1 n ∥n i −Rn i d∥+Σj=1 3λj(r j T r j−1)+2λ4 r 1 T r 2+2λ5 r 1 T r 3+2λ6r2Tr3.  (26)
Such objective function can be solved in closed form as follows:
C=Σi=1 nni dni T  (27)
The singular value decomposition of C can be written as:
C=UDVT,  (28)
where U and V are orthogonal matrices and D is a diagonal matrix. The rotation matrix is as follows:
R=VUT.   (29)
The minimum number of images to determine the rotation matrix R is n=2, provided that the two model planes are not parallel to one another.

For translation, the following relationship can exist:
(n i d)T t+b i d =b i.  (30)
Accordingly, three non-parallel model planes can determine a unique t. If n>3, t may be solved through least squares fitting.

Another exemplary method that can be used by the calibrator component 116 to estimate the initial rotation R and translation t is through knowledge of a set of point correspondences between images generated by the color camera 104 and images generated by the depth sensor 106. Such point pairs can be denoted as (mip i , Mip i d), i=1, . . . , n; pi=1, . . . , Pi. The following relationship exists:
sip i mip i =A[R t]Mip i d.  (31)
It can be noted that the intrinsic matrix A is known. In conventional methods, it has been shown that given three point pairs, there are in general four solutions to the rotation and translation. When one has four or more non-co-planar point pairs, the so-called POSIT algorithm can be used to find initial values of R and t.

With reference now to FIG. 3, an exemplary system 300 that facilitates applying the computed rotation and translation (computed by the calibrator component 116) to subsequently captured images from the color camera 104 and the depth sensor 106 is illustrated. The system 300 comprises the data store 118, which includes the computed rotation and translation matrices R and t. The system 300 further comprises a mapper component 302 that receives an image pair from the color camera 104 and the depth sensor 106. The mapper component 302 can apply the R and t to the images received from the color camera 104 and/or the depth sensor 106, thereby, for instance, overlaying the color image on the depth image to generate a three-dimensional image. Pursuant to an example, this can be undertaken to generate a three-dimensional video stream.

With reference now to FIG. 4, an exemplary methodology 400 is illustrated and described. While the methodology is described as being a series of acts that are performed in a sequence, it is to be understood that the methodology is not limited by the order of the sequence. For instance, some acts may occur in a different order than what is described herein. In addition, an act may occur concurrently with another act. Furthermore, in some instances, not all acts may be required to implement the methodology described herein.

Moreover, the acts described herein may be computer-executable instructions that can be implemented by one or more processors and/or stored on a computer-readable medium or media. The computer-executable instructions may include a routine, a sub-routine, programs, a thread of execution, and/or the like. Still further, results of acts of the methodologies may be stored in a computer-readable medium, displayed on a display device, and/or the like. The computer-readable medium may be any suitable computer-readable storage device, such as memory, hard drive, CD, DVD, flash drive, or the like. As used herein, the term “computer-readable medium” is not intended to encompass a propagated signal.

The exemplary methodology 400 facilitates jointly calibrating a color camera and depth sensor is illustrated. The methodology 400 starts at 402, and at 404 an image generated by a color camera that includes a planar object is received. Prior to receiving the image, an instruction can be output to a user with respect to placement of the planar object relative to the color camera and depth sensor. At 406, a depth image generated by a depth sensor is received, wherein the depth image additionally comprises the planar object. The image generated by the color camera and the image generated by the depth sensor may coincide with one another in time.

At 408, the color camera and the depth sensor are automatically jointly calibrated based at least in part upon the image that comprises the planar object generated by the color camera and the depth image that comprises the planar object generated by the depth sensor. Exemplary techniques for automatically jointly calibrating the color camera in the depth sensor have been described above. Further, while the above has indicated that a single image pair is used, it is to be understood that several image pairs (color images and depth images) can be utilized to jointly calibrate the color camera and depth sensor. The methodology 400 completes at 410.

Now referring to FIG. 5, a high-level illustration of an exemplary computing device 500 that can be used in accordance with the systems and methodologies disclosed herein is illustrated. For instance, the computing device 500 may be used in a system that supports jointly calibrating a color camera and a depth sensor in a depth camera. In another example, at least a portion of the computing device 500 may be used in a system that supports modeling noise/distortion of a color camera and/or depth sensor. The computing device 500 includes at least one processor 502 that executes instructions that are stored in a memory 504. The memory 504 may be or include RAM, ROM, EEPROM, Flash memory, or other suitable memory. The instructions may be, for instance, instructions for implementing functionality described as being carried out by one or more components discussed above or instructions for implementing one or more of the methods described above. The processor 502 may access the memory 504 by way of a system bus 506. In addition to storing executable instructions, the memory 504 may also store images (depth and/or color), computed rotation and translation values, etc.

The computing device 500 additionally includes a data store 508 that is accessible by the processor 502 by way of the system bus 506. The data store may be or include any suitable computer-readable storage, including a hard disk, memory, etc. The data store 508 may include executable instructions, images, etc. The computing device 500 also includes an input interface 510 that allows external devices to communicate with the computing device 500. For instance, the input interface 510 may be used to receive instructions from an external computer device, from a user, etc. The computing device 500 also includes an output interface 512 that interfaces the computing device 500 with one or more external devices. For example, the computing device 500 may display text, images, etc. by way of the output interface 512.

Additionally, while illustrated as a single system, it is to be understood that the computing device 500 may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by the computing device 500.

It is noted that several examples have been provided for purposes of explanation. These examples are not to be construed as limiting the hereto-appended claims. Additionally, it may be recognized that the examples provided herein may be permutated while still falling under the scope of the claims.

Claims (20)

What is claimed is:
1. A method, comprising:
receiving an image of a scene generated by a color camera, the scene comprising a planar object, the planar object having a known pattern;
locating, in the image, the pattern in the planar object based upon the pattern in the planar object being known;
computing a three-dimensional position and orientation of the planar object in the scene responsive to the pattern being located in the image, the three-dimensional position and orientation of the planar object derived from the image of the scene generated by the color camera;
receiving a depth image of the scene generated by a depth sensor, the depth image comprises pixels having depth values; and
jointly calibrating the color camera and the depth sensor based upon:
the computed three-dimensional position and orientation of the planar object; and
the depth values of the pixels in the depth image generated by the depth sensor.
2. The method of claim 1, wherein the color camera has a first coordinate system and the depth sensor has a second coordinate system, and wherein jointly calibrating the color camera and the depth sensor comprises determining a rotation and translation between the first coordinate system and the second coordinate system.
3. The method of claim 2, wherein jointly calibrating the color camera and the depth sensor comprises calculating a plurality of values of intrinsic parameters of the color camera and the depth sensor, the plurality of intrinsic parameters comprising a focus, a camera center, and a depth mapping function.
4. The method of claim 1, further comprising:
receiving a first plurality of images of the scene that are generated by the color camera over time;
receiving a second plurality of images of the scene that are generated by the depth sensor over time, wherein the planar object is at different locations in the scene relative to the color camera and the depth sensor in each of the images in the first plurality of images and the second plurality of images; and
jointly calibrating the color camera and the depth sensor based upon the first plurality of images and the second plurality of images.
5. The method of claim 1, wherein the color camera is a video camera and the depth sensor comprises an infrared camera.
6. The method of claim 1, wherein the depth sensor is one of a time of flight sensor or a structured light sensor.
7. The method of claim 1, wherein the planar object is a checkerboard, the known pattern being a checkerboard pattern.
8. The method of claim 1, wherein jointly calibrating the color camera and the depth sensor further comprises
fitting a plane on the depth image based upon the computed three-dimensional position and orientation of the planar object; and
learning a translation and rotation between a coordinate system of the depth sensor and a coordinate system of the color camera based upon an estimated correspondence between the computed three-dimensional position and orientation of the planar object and the plane fitted on the depth image.
9. The method of claim 1, wherein jointly calibrating the color camera and the depth sensor comprises:
sampling the values of the pixels in the depth image; and
learning a likelihood function that is configured to output a likelihood that a particular pixel in the depth image corresponds to the planar object.
10. The method of claim 9, wherein jointly calibrating the color camera and the depth sensor further comprises learning a translation and rotation between a coordinate system of the depth sensor and a coordinate system of the color camera based upon an evaluation of the likelihood function.
11. The method of claim 1, further comprising:
subsequent to jointly calibrating the color camera and the depth sensor, receiving a first image from the color camera;
subsequent to jointly calibrating the color camera and the depth sensor, receiving a second image from the depth sensor; and
overlaying at least a portion of the first image onto the second image to generate a three- dimensional image based upon the calibrating of the color camera and the depth sensor.
12. A system comprising:
a receiver component that receives:
a first digital image of a scene from a color camera, wherein the scene comprises a planar object that has a known pattern; and
a second digital image of the scene from a depth sensor, the first digital image and the second digital image being coincident in time; and
a calibrator component that jointly calibrates the color camera and the depth sensor based upon:
a computed position and orientation of the planar object in a coordinate system of the first digital image; and
values of pixels in the second digital image that correspond to the planar object.
13. The system of claim 12 comprised by a gaming console.
14. The system of claim 12, wherein the color camera and the depth sensor are included together in a housing.
15. The system of claim 12, wherein the planar object is a checkerboard.
16. The system of claim 12, wherein the calibrator component outputs a rotation and translation between the coordinate system of the color camera and a coordinate system of the depth sensor.
17. The system of claim 16, further comprising:
a mapper component that maps pixels of an image generated by the color camera to pixels of an image generated by the depth sensor.
18. The system of claim 17, wherein the mapper component generates a three-dimensional image based upon the pixels of the image generated by the color camera being mapped to the pixels of the image generated by the depth sensor.
19. A computer-readable data storage medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:
outputting at least one instruction to a user with respect to placement of a planar object having a known pattern relative to a color camera and a depth sensor;
subsequent to outputting the at least one instruction, causing the color camera to capture a first image of a scene that includes the planar object;
causing the depth sensor to capture a second image of the scene, the first image and the second image being coincident in time;
computing a position and orientation of the planar object in a coordinate system of the color camera;
identifying a plane that includes the planar object in the coordinate system of the color camera, the plane identified based upon the computed position and orientation of the planar object in the coordinate system of the color camera;
identifying pixels in the second image that correspond to the planar object based upon values of the pixels;
fitting the plane in the second image based upon the values of the pixels; and
computing an estimated translation and rotation between the coordinate system of the color camera and a coordinate system of the depth sensor based upon the fitting of the plane in the second image.
20. The computer-readable data storage medium of claim 19, the acts further comprising:
subsequent to computing the estimated translation and rotation between the coordinate system of the color camera and the coordinate system of the depth sensor, generating a three-dimensional color image based upon the estimated translation and rotation.
US13/178,494 2011-07-08 2011-07-08 Calibration between depth and color sensors for depth cameras Active 2032-08-15 US9270974B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US13/178,494 US9270974B2 (en) 2011-07-08 2011-07-08 Calibration between depth and color sensors for depth cameras

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US13/178,494 US9270974B2 (en) 2011-07-08 2011-07-08 Calibration between depth and color sensors for depth cameras
PCT/US2012/045879 WO2013009662A2 (en) 2011-07-08 2012-07-08 Calibration between depth and color sensors for depth cameras

Publications (2)

Publication Number Publication Date
US20130010079A1 US20130010079A1 (en) 2013-01-10
US9270974B2 true US9270974B2 (en) 2016-02-23

Family

ID=47438425

Family Applications (1)

Application Number Title Priority Date Filing Date
US13/178,494 Active 2032-08-15 US9270974B2 (en) 2011-07-08 2011-07-08 Calibration between depth and color sensors for depth cameras

Country Status (2)

Country Link
US (1) US9270974B2 (en)
WO (1) WO2013009662A2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160073080A1 (en) * 2014-09-05 2016-03-10 Qualcomm Incorporated Method and apparatus for efficient depth image transformation

Families Citing this family (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2334089A1 (en) * 2009-12-04 2011-06-15 Alcatel Lucent A method and systems for obtaining an improved stereo image of an object
US9384585B2 (en) * 2012-10-23 2016-07-05 Electronics And Telecommunications Research Institute 3-dimensional shape reconstruction device using depth image and color image and the method
KR101428866B1 (en) * 2012-10-26 2014-08-12 한국과학기술원 Apparatus and method for depth manipulation of streoscopic 3d
US9519968B2 (en) * 2012-12-13 2016-12-13 Hewlett-Packard Development Company, L.P. Calibrating visual sensors using homography operators
WO2014145279A1 (en) 2013-03-15 2014-09-18 Leap Motion, Inc. Determining the relative locations of multiple motion-tracking devices
US20140300702A1 (en) * 2013-03-15 2014-10-09 Tagir Saydkhuzhin Systems and Methods for 3D Photorealistic Automated Modeling
US20140267617A1 (en) * 2013-03-15 2014-09-18 Scott A. Krig Adaptive depth sensing
DE112013007165A5 (en) * 2013-06-13 2016-03-10 Leica Camera Ag Camera with opto-electronic rangefinder
WO2015008874A1 (en) * 2013-07-15 2015-01-22 엘지전자 주식회사 Robot cleaner and method for auto-correcting 3d sensor of the robot cleaner
US8917327B1 (en) 2013-10-04 2014-12-23 icClarity, Inc. Method to use array sensors to measure multiple types of data at full resolution of the sensor
CN104677911B (en) * 2013-11-27 2017-10-03 财团法人工业技术研究院 Inspection apparatus and method for machine vision inspection
US9747680B2 (en) * 2013-11-27 2017-08-29 Industrial Technology Research Institute Inspection apparatus, method, and computer program product for machine vision inspection
US10241616B2 (en) 2014-02-28 2019-03-26 Hewlett-Packard Development Company, L.P. Calibration of sensors and projector
KR20150112362A (en) * 2014-03-27 2015-10-07 한국전자통신연구원 Imaging processing method and apparatus for calibrating depth of depth sensor
GB201407270D0 (en) * 2014-04-24 2014-06-11 Cathx Res Ltd 3D data in underwater surveys
US20170223338A1 (en) * 2014-07-31 2017-08-03 Hewlett-Packard Development Company , LP Three dimensional scanning system and framework
US10033992B1 (en) * 2014-09-09 2018-07-24 Google Llc Generating a 3D video of an event using crowd sourced data
US20160360185A1 (en) * 2015-06-03 2016-12-08 Empire Technology Development Llc Three-dimensional imaging sensor calibration
US9609242B2 (en) * 2015-06-25 2017-03-28 Intel Corporation Auto-correction of depth-sensing camera data for planar target surfaces
US10129530B2 (en) * 2015-09-25 2018-11-13 Intel Corporation Video feature tagging
US10003783B2 (en) * 2016-02-26 2018-06-19 Infineon Technologies Ag Apparatus for generating a three-dimensional color image and a method for producing a three-dimensional color image
US20170270654A1 (en) * 2016-03-18 2017-09-21 Intel Corporation Camera calibration using depth data
CN106296789B (en) * 2016-08-05 2019-08-06 深圳迪乐普数码科技有限公司 It is a kind of to be virtually implanted the method and terminal that object shuttles in outdoor scene

Citations (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6373518B1 (en) * 1998-05-14 2002-04-16 Fuji Jukogyo Kabushiki Kaisha Image correction apparatus for stereo camera
US6633664B1 (en) 1999-05-11 2003-10-14 Nippon Telegraph And Telephone Corporation Three-dimensional structure acquisition method, apparatus and computer readable medium
US6768509B1 (en) 2000-06-12 2004-07-27 Intel Corporation Method and apparatus for determining points of interest on an image of a camera calibration object
US6816187B1 (en) 1999-06-08 2004-11-09 Sony Corporation Camera calibration apparatus and method, image processing apparatus and method, program providing medium, and camera
US6858826B2 (en) 1996-10-25 2005-02-22 Waveworx Inc. Method and apparatus for scanning three-dimensional objects
US20050231476A1 (en) * 1996-07-05 2005-10-20 Armstrong Brad A Image controller
US20060128087A1 (en) * 2000-11-09 2006-06-15 Canesta, Inc. Methods and devices for improved charge management for three-dimensional and color sensing
US20070115484A1 (en) 2005-10-24 2007-05-24 Peisen Huang 3d shape measurement system and method including fast three-step phase shifting, error compensation and calibration
US20090201384A1 (en) 2008-02-13 2009-08-13 Samsung Electronics Co., Ltd. Method and apparatus for matching color image and depth image
US20090213240A1 (en) 2008-02-25 2009-08-27 Samsung Electronics Co., Ltd. Method and apparatus for processing three-dimensional (3D) images
US20090231425A1 (en) 2008-03-17 2009-09-17 Sony Computer Entertainment America Controller with an integrated camera and methods for interfacing with an interactive application
US20100207938A1 (en) 2009-02-18 2010-08-19 International Press Of Boston, Inc. Simultaneous three-dimensional geometry and color texture acquisition using single color camera
US20100225743A1 (en) 2009-03-05 2010-09-09 Microsoft Corporation Three-Dimensional (3D) Imaging Based on MotionParallax
US20100235129A1 (en) * 2009-03-10 2010-09-16 Honeywell International Inc. Calibration of multi-sensor system
US20100303341A1 (en) * 2009-06-01 2010-12-02 Haeusler Gerd Method and device for three-dimensional surface detection with a dynamic reference frame
US20110018973A1 (en) * 2008-03-26 2011-01-27 Konica Minolta Holdings, Inc. Three-dimensional imaging device and method for calibrating three-dimensional imaging device
US20110054295A1 (en) * 2009-08-25 2011-03-03 Fujifilm Corporation Medical image diagnostic apparatus and method using a liver function angiographic image, and computer readable recording medium on which is recorded a program therefor
US7912252B2 (en) 2009-02-06 2011-03-22 Robert Bosch Gmbh Time-of-flight sensor-assisted iris capture system and method
US20110069892A1 (en) 2009-09-24 2011-03-24 Chih-Hsiang Tsai Method of comparing similarity of 3d visual objects
US20110150101A1 (en) * 2008-09-02 2011-06-23 Yuan Liu 3d video communication method, sending device and system, image reconstruction method and system
US8090194B2 (en) * 2006-11-21 2012-01-03 Mantis Vision Ltd. 3D geometric modeling and motion capture using both single and dual imaging
US20120026296A1 (en) * 2010-07-29 2012-02-02 Samsung Electronics Co., Ltd. Image processing apparatus and method

Patent Citations (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050231476A1 (en) * 1996-07-05 2005-10-20 Armstrong Brad A Image controller
US6858826B2 (en) 1996-10-25 2005-02-22 Waveworx Inc. Method and apparatus for scanning three-dimensional objects
US6373518B1 (en) * 1998-05-14 2002-04-16 Fuji Jukogyo Kabushiki Kaisha Image correction apparatus for stereo camera
US6633664B1 (en) 1999-05-11 2003-10-14 Nippon Telegraph And Telephone Corporation Three-dimensional structure acquisition method, apparatus and computer readable medium
US6816187B1 (en) 1999-06-08 2004-11-09 Sony Corporation Camera calibration apparatus and method, image processing apparatus and method, program providing medium, and camera
US6768509B1 (en) 2000-06-12 2004-07-27 Intel Corporation Method and apparatus for determining points of interest on an image of a camera calibration object
US20060128087A1 (en) * 2000-11-09 2006-06-15 Canesta, Inc. Methods and devices for improved charge management for three-dimensional and color sensing
US20070115484A1 (en) 2005-10-24 2007-05-24 Peisen Huang 3d shape measurement system and method including fast three-step phase shifting, error compensation and calibration
US8090194B2 (en) * 2006-11-21 2012-01-03 Mantis Vision Ltd. 3D geometric modeling and motion capture using both single and dual imaging
US20090201384A1 (en) 2008-02-13 2009-08-13 Samsung Electronics Co., Ltd. Method and apparatus for matching color image and depth image
US20090213240A1 (en) 2008-02-25 2009-08-27 Samsung Electronics Co., Ltd. Method and apparatus for processing three-dimensional (3D) images
US20090231425A1 (en) 2008-03-17 2009-09-17 Sony Computer Entertainment America Controller with an integrated camera and methods for interfacing with an interactive application
US20110018973A1 (en) * 2008-03-26 2011-01-27 Konica Minolta Holdings, Inc. Three-dimensional imaging device and method for calibrating three-dimensional imaging device
US20110150101A1 (en) * 2008-09-02 2011-06-23 Yuan Liu 3d video communication method, sending device and system, image reconstruction method and system
US7912252B2 (en) 2009-02-06 2011-03-22 Robert Bosch Gmbh Time-of-flight sensor-assisted iris capture system and method
US20100207938A1 (en) 2009-02-18 2010-08-19 International Press Of Boston, Inc. Simultaneous three-dimensional geometry and color texture acquisition using single color camera
US20100225743A1 (en) 2009-03-05 2010-09-09 Microsoft Corporation Three-Dimensional (3D) Imaging Based on MotionParallax
US20100235129A1 (en) * 2009-03-10 2010-09-16 Honeywell International Inc. Calibration of multi-sensor system
US20100303341A1 (en) * 2009-06-01 2010-12-02 Haeusler Gerd Method and device for three-dimensional surface detection with a dynamic reference frame
US20110054295A1 (en) * 2009-08-25 2011-03-03 Fujifilm Corporation Medical image diagnostic apparatus and method using a liver function angiographic image, and computer readable recording medium on which is recorded a program therefor
US20110069892A1 (en) 2009-09-24 2011-03-24 Chih-Hsiang Tsai Method of comparing similarity of 3d visual objects
US20120026296A1 (en) * 2010-07-29 2012-02-02 Samsung Electronics Co., Ltd. Image processing apparatus and method

Non-Patent Citations (26)

* Cited by examiner, † Cited by third party
Title
"International Search Report", Mailed Date: Dec. 21, 2012, Application No. PCT/US2012/045879, Filed Date: Dec. 21, 2012, pp. 1-9.
Arun, et al., "Least-Squares Fitting of Two 3-D Point Sets", Retrieved at <<http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4767965>>, IEEE Transactions on Pattern Analysis and Machine Intelligenc, vol. PAMI-9, No. 5, Sep. 1987, pp. 698-700.
Arun, et al., "Least-Squares Fitting of Two 3-D Point Sets", Retrieved at >, IEEE Transactions on Pattern Analysis and Machine Intelligenc, vol. PAMI-9, No. 5, Sep. 1987, pp. 698-700.
C. Daniel Herrera, J. Kannala, & J. Heikkila, "Accurate and Practical Calibration of a Calibration of a Depth and Color Camera Pair", 6855 Lecture Notes in Computer Sci. 437-445 (Aug. 2011). *
C. Daniel Herrera, J. Kannala, & J. Heikkila, "Joint Depth and Color Camera Calibration with Distortion Correction", 34 IEEE Transactions on Patern Analysis & Machine Intelligence 2058-2064 (May 2012). *
C. Raposo, J.P. Barreto, & U. Nunes, Fast and Accurate Calibration of a Kinect Sensor, 2013 Int'l Conf. on 3D Vision 342-349 (2013). *
Cai, et al., "3D Deformable Face Tracking with a Commodity Depth Camera", Retrieved at <<http://research.microsoft.com/en-us/um/people/zhang/papers/eccv2010-facetrackingwithdepthcamera.pdf>>, Proceedings of the 11th European conference on computer vision: Part III, 2010, pp. 229-242.
Crabb, et al., "Real-Time Foreground Segmentation via Range and Color Imaging", Retrieved at <<http://mplab.ucsd.edu/wp-content/uploads/CVPR2008/WorkShops/data/papers/221.pdf>>, IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Jun. 23-28, 2008, pp. 1-5.
Crabb, et al., "Real-Time Foreground Segmentation via Range and Color Imaging", Retrieved at >, IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Jun. 23-28, 2008, pp. 1-5.
Cui, et al., "3D Shape Scanning with a Time-of-Flight Camera", Retrieved at <<http://ai.stanford.edu/˜schuon/sr/cvpr10-scanning.pdf>>, IEEE Conference on Computer Vision and Pattern Recognition, Jun. 13-18, 2010, pp. 1-8.
Cui, et al., "3D Shape Scanning with a Time-of-Flight Camera", Retrieved at >, IEEE Conference on Computer Vision and Pattern Recognition, Jun. 13-18, 2010, pp. 1-8.
Dementhon, et al., "Model-Based Object Pose in 25 Lines of Code", Retrieved at <<http://www.cfar.umd.edu/˜daniel/daniel-papersfordownload/Pose25Lines.ps.gz>>, Computer Vision-ECCV'95, 1995, pp. 1-30.
Dementhon, et al., "Model-Based Object Pose in 25 Lines of Code", Retrieved at >, Computer Vision-ECCV'95, 1995, pp. 1-30.
Fischler, et al., "Random Sample Consensus : A Paradigm for Model Fitting with Apphcatlons to Image Analysis and Automated Cartography", Retrieved at <<http://www.ai.sri.com/pubs/files/836.pdf>>, Communications of the ACM, vol. 24, No. 6, 1981, pp. 381-395.
Fischler, et al., "Random Sample Consensus : A Paradigm for Model Fitting with Apphcatlons to Image Analysis and Automated Cartography", Retrieved at >, Communications of the ACM, vol. 24, No. 6, 1981, pp. 381-395.
Frick, et al., "Generation of 3D-TV LDV-Content with Time of Flight Camera", Retrieved at <<http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.155.5198&rep=rep1&type=pdf>>, 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video, May 4-6, 2009, pp. 1-4.
Frick, et al., "Generation of 3D-TV LDV-Content with Time of Flight Camera", Retrieved at >, 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video, May 4-6, 2009, pp. 1-4.
Guan, et al., "3D Object Reconstruction with Heterogeneous Sensor Data", Retrieved at <<http://www.cc.gatech.edu/conferences/3DPVT08/Program/Papers/paper108.pdf>>, The Fourth International Symposium on 3D Data Processing, Visualization and Transmission, 2008, pp. 1-8.
Guan, et al., "3D Object Reconstruction with Heterogeneous Sensor Data", Retrieved at >, The Fourth International Symposium on 3D Data Processing, Visualization and Transmission, 2008, pp. 1-8.
J. Smisek, J. Jancosek, & T. Pajdla, "3D with Kinect", 2011 IEEE Int'l conf. on Computer Vision Workshops 1154-1160 (Nov. 2011). *
Tsai, Roger Y., "A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology using Off-the-Shelf TV Cameras and Lenses", Retrieved at <<http://www.vision.caltech.edu/bouguetj/calib-doc/papers/Tsai.pdf>>, IEEE Journal of Robotics and Automation, vol. 3, No. 4, Aug. 1987, pp. 323-344.
Tsai, Roger Y., "A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology using Off-the-Shelf TV Cameras and Lenses", Retrieved at >, IEEE Journal of Robotics and Automation, vol. 3, No. 4, Aug. 1987, pp. 323-344.
Yuan, Joseph S. C., "A General Photogrammetric Method for Determining Object Position and Orientation", Retrieved at <<http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=88034>>, IEEE Transactions on Robotics and Automation, vol. 5, No. 2, Apr. 1989, pp. 129-142.
Yuan, Joseph S. C., "A General Photogrammetric Method for Determining Object Position and Orientation", Retrieved at >, IEEE Transactions on Robotics and Automation, vol. 5, No. 2, Apr. 1989, pp. 129-142.
Zhang, et al., "A Flexible New Technique for Camera Calibration", Retrieved at <<http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=888718>>, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, No. 11, Nov. 2000, pp. 1330-1334.
Zhang, et al., "A Flexible New Technique for Camera Calibration", Retrieved at >, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, No. 11, Nov. 2000, pp. 1330-1334.

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160073080A1 (en) * 2014-09-05 2016-03-10 Qualcomm Incorporated Method and apparatus for efficient depth image transformation
US9948911B2 (en) * 2014-09-05 2018-04-17 Qualcomm Incorporated Method and apparatus for efficient depth image transformation

Also Published As

Publication number Publication date
WO2013009662A3 (en) 2013-03-07
WO2013009662A2 (en) 2013-01-17
US20130010079A1 (en) 2013-01-10

Similar Documents

Publication Publication Date Title
Svoboda et al. A convenient multicamera self-calibration for virtual environments
US8630460B2 (en) Incorporating video meta-data in 3D models
Stoykova et al. 3-D time-varying scene capture technologies—A survey
US9177381B2 (en) Depth estimate determination, systems and methods
Zhang et al. Calibration between depth and color sensors for commodity depth cameras
Cyganek et al. An introduction to 3D computer vision techniques and algorithms
Zhu et al. Reliability fusion of time-of-flight depth and stereo geometry for high quality depth maps
Zhang Camera calibration with one-dimensional objects
Ke et al. Quasiconvex optimization for robust geometric reconstruction
US20140253679A1 (en) Depth measurement quality enhancement
Herrera et al. Joint depth and color camera calibration with distortion correction
Wang et al. Estimation of multiple directional light sources for synthesis of augmented reality images
Jiang et al. A global linear method for camera pose registration
US20070127101A1 (en) Method for automatic stereo measurement of a point of interest in a scene
US9872010B2 (en) Lidar stereo fusion live action 3D model video reconstruction for six degrees of freedom 360° volumetric virtual reality video
JP2011530706A (en) 3D-TOF camera device and position / orientation calibration method therefor
US20130121559A1 (en) Mobile device with three dimensional augmented reality
Zhang Camera calibration
Sturm et al. A generic concept for camera calibration
US10109107B2 (en) Method and system for representing a virtual object in a view of a real environment
Soatto et al. Recursive 3-d visual motion estimation using subspace constraints
US20170337739A1 (en) Mobile augmented reality system
Bonfort et al. General specular surface triangulation
KR20150013709A (en) A system for mixing or compositing in real-time, computer generated 3d objects and a video feed from a film camera
Ramalingam et al. Towards complete generic camera calibration

Legal Events

Date Code Title Description
AS Assignment

Owner name: MICROSOFT CORPORATION, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHANG, CHA;ZHANG, ZHENGYOU;REEL/FRAME:026558/0949

Effective date: 20110706

AS Assignment

Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034544/0001

Effective date: 20141014

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4