EP3040941A1 - Method for calibrating a depth camera - Google Patents
Method for calibrating a depth camera Download PDFInfo
- Publication number
- EP3040941A1 EP3040941A1 EP14307197.5A EP14307197A EP3040941A1 EP 3040941 A1 EP3040941 A1 EP 3040941A1 EP 14307197 A EP14307197 A EP 14307197A EP 3040941 A1 EP3040941 A1 EP 3040941A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- depth camera
- depth
- computer
- reference object
- calibration parameters
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three-dimensional [3D] modelling for computer graphics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/327—Calibration thereof
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/593—Depth or shape recovery from multiple images from stereo images
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T7/85—Stereo camera calibration
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/128—Adjusting depth or disparity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0081—Depth or disparity estimation from stereoscopic image signals
Definitions
- the invention relates to a computer-implemented method for calibrating a depth camera.
- the invention belongs to the technical field of three-dimensional (3D) reconstruction, which consists in building digital models of physical three-dimensional objects from data acquired by sensors such as conventional cameras and/or depth cameras. 3D reconstruction lends itself to applications such as computer vision, computer graphics and augmented reality.
- the invention relates to the calibration - and more particularly to the "intrinsic" calibration - of depth cameras.
- Depth cameras also known as “range cameras” are devices which produce two-dimensional (2D) images - known as “depth maps” - showing the distance of points in a scene from a specific point of the depth camera.
- 2D images known as “depth maps” - showing the distance of points in a scene from a specific point of the depth camera.
- time-of-flight cameras based on a principle similar to radar, and structured light cameras, which project an infrared pattern onto the scene, acquire an image thereof and deduce depth information using image processing.
- Depth cameras can be combined with conventional color (RGB - Red, Green and Blue) cameras to produce two-dimensional images containing both color and depth information, which can be referred to as "RGB-Depth", or simply "RGBD” images. Combining a depth camera with a black-and-white camera is also possible, albeit less useful.
- a depth camera DC can be used to acquire multiple depth maps DM1, DM2, DM3 ("pure” depth maps or RGBD images) of a "physical" three-dimensional object PO from different positions.
- a reconstruction algorithm RA executed by a computer, produces a three-dimensional reconstruction 3DR of the physical object taking the depth maps as inputs.
- suitable reconstruction algorithms are known in the art, see e.g. R.A. Newcombe et al. "KinectFusion: Real-Time Dense Surface Mapping and Tracking", Symposium ISMAR 2011 .
- the reconstruction algorithm does not necessarily need to receive, at its inputs, the positions and orientations ("poses") taken by the depth cameras to acquire the different depth maps, because this information - also called “extrinsic” calibration - can be deduced using the depth maps themselves.
- "intrinsic" calibration parameters of the depth camera - such as its focal length(s), principal point and possibly distortion parameters - do have to be provided to the reconstruction algorithm.
- figure 2 shows different reconstructions of two three-dimensional objects - a figurine representing a rabbit on the top line, a cube on the bottom line - obtained by providing to the reconstruction algorithm different values of the focal length of the depth camera used to acquire the depth maps used as inputs.
- the central image in a frame
- images to the left and to the right correspond to increasingly incorrect values of this parameter. It can be seen that an incorrect calibration of the depth camera can lead to a completely meaningless reconstruction.
- calibration parameters provided by the manufacturer of a depth camera can be used for 3D reconstruction (this is the case of the above-referenced paper by R.A. Newcombe et al.), but if more accurate results are required, a dedicated calibration step has to be carried out.
- the problem of calibrating a depth (or a RGB-Depth) camera has been addressed e.g. in the following documents:
- the first of these documents concerns specifically the case of a RGB-Depth camera, but can also be applied to other kinds of depth camera which also provides an intensity image of the scene; the other two concern the calibration of conventional cameras allowing no depth measurements, but their teaching also applies to RGB-Depth cameras, and to depth cameras acquiring a grayscale image of the scene over which an infrared pattern has been projected.
- the invention aims at overcoming the drawbacks of the prior art mentioned above, and more specifically at providing a calibration method for a depth camera which is better suited for 3D reconstruction.
- An object of the present invention is a method for calibrating a depth camera comprising the steps of:
- Another object of the invention is a computer program product, stored on a computer-readable data-storage medium, comprising computer-executable instructions to cause a computer system to carry out steps S2) to S4) of such a method.
- Another object of the invention is a computer-readable data-storage medium containing computer-executable instructions to cause a computer system to carry out steps S2) to S4) of such a method.
- Another object of the invention is a computer system comprising a processor coupled to a memory and a graphical user interface, the memory storing computer-executable instructions to cause the computer system to carry out steps S2) to S4) of such a method.
- Depth camera designates any device producing images containing depth information.
- Depth map designates any image containing depth information such as a “pure” depth map (only containing depth information) or a RGBD one.
- Calibration parameters of a depth camera will be used specifically to designate “intrinsic” parameters, i.e. all the optical, mechanical, electronics etc. parameter of the depth camera having an influence on the way a scene is converted to an image containing depth information.
- Calibration parameters usually include:
- a depth camera can be characterized by a set of calibration parameters, which form a vector, or point of a suitable "parameter space", and a pose - i.e. its position and orientation.
- the parameters expressing the pose are also called “extrinsic parameters”.
- a physical object i.e. an object existing in the physical word
- a "digital model" of a physical object is a representation of such an object (or simply of its outer surface) in a computer memory, in the form of a file, plurality of files or data structure.
- a digital model can represent an object as a grid corresponding to its outer surface, or as a point cloud.
- a “3D digital model” is a digital model preserving the three-dimensional spatial structure of the object;
- Reconstruction refers to the process of building a digital model of an object from depth maps thereof, and also to the digital model obtained by this process.
- a “distance field” is a particular data structure in a computer memory, representing a portion of the physical space containing a physical object. This portion of physical space is represented by a finite “reconstruction space", discretized in voxels ("voxelized”). A numerical value is attributed to each voxel, said value expressing the distance (typically, a signed and truncated distance) between the voxel and a surface of the physical object.
- An “isosurface” of the reconstruction space is a surface defined by voxels of a same value.
- Regular transformation is the operation of determining the rigid transformation (rotation-translation) which links, exactly or at least approximately, two digital models of a physical object.
- the depth camera DC to be calibrated is used to take a set DMS of depth maps of a reference three-dimensional physical object PO, an accurate three-dimensional digital model RM of which (reference model) is available.
- the reconstruction algorithm requires a set of calibration parameters of the depth camera - focal length(s), principal point, possibly also distortion parameters - which are unknown, and whose determination is the aim of the inventive method.
- a sampling of a parameter space is performed, which provides a finite number of sets - or vectors - of parameters, SCP i , each characterized by a particular value of each calibration parameter. For example, if astigmatism and distortion are ignored, the depth camera is characterized by three intrinsic calibration parameters: its focal length f and the coordinates u 0 , v 0 of its principal point.
- N parameter vectors SCP i [f i , u 0,i , v 0,i ] - with f i ⁇ [f min , f max ], u 0,i ⁇ [u 0,min , u 0,max ], v 0,i ⁇ [v 0,min , v 0,max ], i ⁇ [1, N] - are obtained by sampling the three-dimensional parameter space.
- a first sample of the parameter space SCP 1 is chosen for performing the first reconstruction of the reference object. Other samples are used during successive iterations of the inventive method.
- This method uses a truncated signed distance field for iteratively fusing a plurality of depth maps into a unified representation of a scene including the reference object.
- the distance field is obtained by defining a finite reconstruction volume, voxelizing it, and assigning to each voxel a signed and truncated distance d ( x,y,z ) from the 3D surface of the reference object, x,y,z being the coordinates of the reconstruction volume.
- d x,y,z
- a first depth map is arbitrary positioned relative to the distance field, and then fused into it according to the process described further.
- the position and orientation (i.e. the pose) of the depth camera when acquiring the (n-1)th depth map are supposed known.
- the pose of the camera corresponding to this image i.e. the rotation and translation defining its position and orientation with respect to a reference frame. This is performed using the so-called point-to-plane iterative closest point algorithm.
- the virtual view is less noisy than the real view from the (n-1)th pose; moreover using a virtual view instead of a real one avoids a drift of the pose.
- the normal to the surface at this point is determined by computing the gradient of the distance field at that point.
- refPC a point cloud
- Each pixel of the current (n-th) depth map is then back-projected to create another point cloud, and the normal to the surface corresponding to each point of the cloud is also determined by computing the cross product between points corresponding to neighboring pixels.
- This point cloud is called currPC.
- said current depth map is fused into the distance field.
- the new information given by the current depth map is simply averaged with the previous information stored in each voxel of the distance field.
- Each voxel of the distance field is projected onto the depth map, and the distance of this voxel to the sensor is compared with the depth given by the depth map at the projection of the voxel. This gives the distance d' that should have the voxel according to the current depth map.
- the weight w can be initialized to 1 and then increases according to the second equation, while being limited to a preset maximum value M.
- the weight w' can either take a constant value (e.g. 1) or a value depending on d' (to account for the fact that precision often decreases with distance) or on the incidence angle of the ray connecting the centre of the camera with the voxel.
- the three-dimensional reconstruction 3DR i of the reference object obtained this way is then aligned (registered) with the reference model RM to be compared to it in order to validate the estimation of the intrinsic parameters. Registration and comparison proper are jointly represented, on figure 3 , by block "CMP".
- the second stage consists in describing those points.
- the description is often based on their neighborhood. In order to match the points representing the same geometry and differentiate those of different geometries, the description must be fully detailed.
- a 3D-to-3D distance between the models is computed to estimate an error ERR i between them.
- the error can be e.g. a mean square distance or a (possibly symmetrical) Hausdorff distance.
- the error ERR i is then compared to a fixed threshold TH. If it is lower than, or lower or equal to, this threshold, the iteration stops and SCP i gives the optimal calibration parameters of the depth camera. Otherwise, "i" is increased (i.e. different trial values for the calibration parameters are chosen) and a new iteration is performed. If all the samples of the parameter space have been used (i.e. "i" has attained its maximum value), then the set of parameters corresponding to the lowest error is chosen. Alternatively, comparison to a threshold can be omitted, and full exploration of the parameter space is performed in any case.
- the inventive method (or, more precisely, its data processing part, following the acquisition of the depth maps) can be performed by a suitably-programmed general-purpose computer or computer system, possibly including a computer network, storing a suitable program in non-volatile form on a computer-readable medium such as a hard disk, a solid state disk or a CD-ROM and executing said program using its microprocessor(s) and memory.
- a suitably-programmed general-purpose computer or computer system possibly including a computer network, storing a suitable program in non-volatile form on a computer-readable medium such as a hard disk, a solid state disk or a CD-ROM and executing said program using its microprocessor(s) and memory.
- FIG. 4 A computer suitable for carrying out a method according to an exemplary embodiment of the present invention is described with reference to figure 4 .
- the computer includes a Central Processing Unit (CPU) P which performs the processes described above.
- the process can be stored as an executable program, i.e. a set of computer-readable instructions in memory, such as RAM M1 or ROM M2, or on hard disk drive (HDD) M3, DVD/CD drive M4, or can be stored remotely.
- At least one digital model of a reference object is also stored on one or more of memory devices M1 to M4, or remotely.
- the claimed invention is not limited by the form of the computer-readable media on which the computer-readable instructions and/or the digital model(s) of the inventive process are stored.
- the instructions, the trial sets of calibration parameters and the digital model(s) can be stored on CDs, DVDs, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk or any other information processing device with which the computer aided design station communicates, such as a server or computer.
- the program, the trial sets of calibration parameters and the digital model(s) can be stored on a same memory device or on different memory devices.
- a computer program suitable for carrying out the inventive method can be provided as a utility application, background daemon, or component of an operating system, or combination thereof, executing in conjunction with CPU 800 and an operating system such as Microsoft VISTA, Microsoft Windows 7, UNIX, Solaris, LINUX, Apple MAC-OS and other systems known to those skilled in the art.
- an operating system such as Microsoft VISTA, Microsoft Windows 7, UNIX, Solaris, LINUX, Apple MAC-OS and other systems known to those skilled in the art.
- CPU P can be a Xenon processor from Intel of America or an Opteron processor from AMD of America, or can be other processor types, such as a Freescale ColdFire, IMX, or ARM processor from Freescale Corporation of America.
- the CPU can be a processor such as a Core2 Duo from Intel Corporation of America, or can be implemented on an FPGA, ASIC, PLD or using discrete logic circuits, as one of ordinary skill in the art would recognize.
- the CPU can be implemented as multiple processors cooperatively working to perform the computer-readable instructions of the inventive processes described above.
- the computer aided design station in figure 4 also includes a network interface NI, such as an Intel Ethernet PRO network interface card from Intel Corporation of America, for interfacing with a network, such as a local area network (LAN), wide area network (WAN), the Internet and the like.
- the computer aided design station further includes a display controller DC, such as a NVIDIA GeForce GTX graphics adaptor from NVIDIA Corporation of America for interfacing with display DY, such as a Hewlett Packard HPL2445w LCD monitor.
- a general purpose I/O interface IF interfaces with a keyboard KB and pointing device PD, such as a roller ball, mouse, touchpad and the like.
- Disk controller DKC connects HDD M3 and DVD/CD M4 with communication bus CBS, which can be an ISA, EISA, VESA, PCI, or similar, for interconnecting all of the components of the computer aided design station.
- CBS can be an ISA, EISA, VESA, PCI, or similar, for interconnecting all of the components of the computer aided design station.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
S1) using a depth camera (DC) to be calibrated for acquiring a plurality of depth maps (DMS) of a physical three-dimensional reference object (PO), corresponding to different positions of the depth camera relative to the reference object;
S2) computing a three-dimensional reconstruction (3DRi) of the reference object from said depth maps, depending on values assigned to a set of calibration parameters (SCPi) of the depth camera;
S3) computing an error (ERRi) between the three-dimensional reconstruction computed at step S2) and a predetermined digital model (RM) of the reference object;
said steps S2) and S3) being iterated by assigning different values to said set of calibration parameters of the depth camera; and
S4) choosing values (SCP) of said set of calibration parameters which minimize the error computed at step S3);
said steps S2) to S4) being performed by a computer.
Description
- The invention relates to a computer-implemented method for calibrating a depth camera.
- The invention belongs to the technical field of three-dimensional (3D) reconstruction, which consists in building digital models of physical three-dimensional objects from data acquired by sensors such as conventional cameras and/or depth cameras. 3D reconstruction lends itself to applications such as computer vision, computer graphics and augmented reality.
- More precisely, the invention relates to the calibration - and more particularly to the "intrinsic" calibration - of depth cameras. Depth cameras (also known as "range cameras") are devices which produce two-dimensional (2D) images - known as "depth maps" - showing the distance of points in a scene from a specific point of the depth camera. Several different technologies can be used to implement depth cameras; among the most widely used one might mention time-of-flight cameras, based on a principle similar to radar, and structured light cameras, which project an infrared pattern onto the scene, acquire an image thereof and deduce depth information using image processing. Depth cameras can be combined with conventional color (RGB - Red, Green and Blue) cameras to produce two-dimensional images containing both color and depth information, which can be referred to as "RGB-Depth", or simply "RGBD" images. Combining a depth camera with a black-and-white camera is also possible, albeit less useful.
- As illustrated on
figure 1 , a depth camera DC can be used to acquire multiple depth maps DM1, DM2, DM3 ("pure" depth maps or RGBD images) of a "physical" three-dimensional object PO from different positions. A reconstruction algorithm RA, executed by a computer, produces a three-dimensional reconstruction 3DR of the physical object taking the depth maps as inputs. Several suitable reconstruction algorithms are known in the art, see e.g. R.A. Newcombe et al. "KinectFusion: Real-Time Dense Surface Mapping and Tracking", Symposium ISMAR 2011. - The reconstruction algorithm does not necessarily need to receive, at its inputs, the positions and orientations ("poses") taken by the depth cameras to acquire the different depth maps, because this information - also called "extrinsic" calibration - can be deduced using the depth maps themselves. On the contrary, "intrinsic" calibration parameters of the depth camera - such as its focal length(s), principal point and possibly distortion parameters - do have to be provided to the reconstruction algorithm. The importance of the intrinsic calibration parameters is highlighted by
figure 2 , which shows different reconstructions of two three-dimensional objects - a figurine representing a rabbit on the top line, a cube on the bottom line - obtained by providing to the reconstruction algorithm different values of the focal length of the depth camera used to acquire the depth maps used as inputs. On both lines, the central image (in a frame) corresponds to the correct values of the focal length; images to the left and to the right correspond to increasingly incorrect values of this parameter. It can be seen that an incorrect calibration of the depth camera can lead to a completely meaningless reconstruction. - In some cases, calibration parameters provided by the manufacturer of a depth camera can be used for 3D reconstruction (this is the case of the above-referenced paper by R.A. Newcombe et al.), but if more accurate results are required, a dedicated calibration step has to be carried out. The problem of calibrating a depth (or a RGB-Depth) camera has been addressed e.g. in the following documents:
- Herrera C., D., Kannala, J., Heikkilä, J., "Joint depth and color camera calibration with distortion correction", TPAMI, 2012;
- R.Y. Tsai, "An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision", Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, pp. 364-374, 1986;
- Zhang, Zhengyou "Flexible Camera Calibration By Viewing a Plane From Unknown Orientations", ICCV, 1999.
- The first of these documents concerns specifically the case of a RGB-Depth camera, but can also be applied to other kinds of depth camera which also provides an intensity image of the scene; the other two concern the calibration of conventional cameras allowing no depth measurements, but their teaching also applies to RGB-Depth cameras, and to depth cameras acquiring a grayscale image of the scene over which an infrared pattern has been projected.
- These approaches use a planar pattern of known 2D geometry, such as a chessboard with known distance between corners, several images of which are acquired using the RGB or RGB-Depth camera to be calibrated. These images allow estimating a projection matrix of the camera, from which calibration parameters are extracted.
- These approaches are more robust than the use of constructor datasheets, but there is no guarantee that they provide calibration parameters sufficiently accurate for 3D reconstruction; for instance, an error on focal lengths can be compensated by a drift on extrinsic parameters, i.e. parameters expressing the position and orientation of the camera. Moreover, the obtained parameters are extremely dependent on the protocol used for image acquisition.
- The invention aims at overcoming the drawbacks of the prior art mentioned above, and more specifically at providing a calibration method for a depth camera which is better suited for 3D reconstruction.
- An object of the present invention is a method for calibrating a depth camera comprising the steps of:
- S1) using a depth camera to be calibrated for acquiring a plurality of depth maps of a physical three-dimensional reference object, corresponding to different positions of the depth camera relative to the reference object;
- S2) computing a three-dimensional reconstruction of the reference object from said depth maps, depending on values assigned to a set of calibration parameters of the depth camera;
- S3) computing an error between the three-dimensional reconstruction computed at step S2) and a predetermined digital model of the reference object;
said steps S2) and S3) being iterated by assigning different values to said set of calibration parameters of the depth camera; and - S4) choosing values of said set of calibration parameters which minimize the error computed at step S3);
said steps S2) to S4) being performed by a computer. - According to different embodiments of such a method:
- Said steps S2) and S3) may be iterated a predetermined number of times by assigning respective predetermined values to said set of calibration parameters of the depth camera, said predetermined values sampling a parameter space.
- Alternatively, said steps S2) and S3) may be iterated until the error computed at step S3) becomes lower, or lower of equal, to a predetermined threshold.
- Said set of calibration parameters may comprise at least a focal length and two coordinates of a principal point of the depth camera. Said set of calibration parameters may also comprise at least one distortion parameter.
- Said step S2) may comprise jointly determining the position of the depth camera, relative to a reference position, corresponding to each of said depth maps, and fusing said depth maps into a single three-dimensional signed and truncated distance field; and extracting an isosurface of said distance field into a three-dimensional mesh constituting said reconstruction of the reference object.
- Said step S3) may comprise registering the three-dimensional reconstruction computed at step S2) and said predetermined digital model of the reference object prior to computing said error.
- Said step S3) may comprise computing an exact or approximate Hausdorff distance or symmetrical Hausdorff distance between the three-dimensional reconstruction computed at step S2) and said predetermined digital model of the reference object.
- Another object of the invention is a computer program product, stored on a computer-readable data-storage medium, comprising computer-executable instructions to cause a computer system to carry out steps S2) to S4) of such a method.
- Another object of the invention is a computer-readable data-storage medium containing computer-executable instructions to cause a computer system to carry out steps S2) to S4) of such a method.
- Another object of the invention is a computer system comprising a processor coupled to a memory and a graphical user interface, the memory storing computer-executable instructions to cause the computer system to carry out steps S2) to S4) of such a method.
- "Depth camera" designates any device producing images containing depth information.
- "Depth map" designates any image containing depth information such as a "pure" depth map (only containing depth information) or a RGBD one.
- "Calibration parameters" of a depth camera will be used specifically to designate "intrinsic" parameters, i.e. all the optical, mechanical, electronics etc. parameter of the depth camera having an influence on the way a scene is converted to an image containing depth information. Calibration parameters usually include:
- In optics, the "physical" focal length F is the distance at which initially collimated rays are brought to a focus by a lens. When modeling cameras, however, focal length "f" is most often defined as the product of the physical focal length F of the lens (measured in mm) and the pixel density of the imaging sensor (pixels/mm), and is therefore measured in pixels. If the pixels are rectangular, there are two different values of pixel density along the x and y directions: sx and sy. As a consequence, two focal length are defined: fx=sx·F and fy=sy·F. Only fx and fy can be derived without actually dismantling the camera and measuring its components directly.
- Coordinates of the principal point in the image plane. The principal point is defined by the intersection of the optical axis of the camera with its image plane.
- Distortion parameters express radial and/or tangential distortion of the camera lens. Usually, they are the parameters of a polynomial function linking undistorted and distorted image points.
- A depth camera can be characterized by a set of calibration parameters, which form a vector, or point of a suitable "parameter space", and a pose - i.e. its position and orientation. The parameters expressing the pose are also called "extrinsic parameters".
- A physical object (i.e. an object existing in the physical word) can be considered "three-dimensional" if its smallest dimension is greater or equal than one-hundredth, and preferably one-tenth, of its greatest dimension.
- A "digital model" of a physical object is a representation of such an object (or simply of its outer surface) in a computer memory, in the form of a file, plurality of files or data structure. For instance, a digital model can represent an object as a grid corresponding to its outer surface, or as a point cloud. A "3D digital model" is a digital model preserving the three-dimensional spatial structure of the object;
- "Reconstruction" refers to the process of building a digital model of an object from depth maps thereof, and also to the digital model obtained by this process.
- A "distance field" is a particular data structure in a computer memory, representing a portion of the physical space containing a physical object. This portion of physical space is represented by a finite "reconstruction space", discretized in voxels ("voxelized"). A numerical value is attributed to each voxel, said value expressing the distance (typically, a signed and truncated distance) between the voxel and a surface of the physical object. An "isosurface" of the reconstruction space is a surface defined by voxels of a same value.
- "Registering" is the operation of determining the rigid transformation (rotation-translation) which links, exactly or at least approximately, two digital models of a physical object.
- Additional features and advantages of the present invention will become apparent from the subsequent description, taken in conjunction with the accompanying drawings, wherein:
-
Figure 1 , described above, is a schematic representation of the process of performing three-dimensional reconstruction of a physical object using a depth camera; -
Figure 2 , described above, illustrates the importance of calibration of the depth camera for three-dimensional reconstruction; -
Figure 3 is a flow-chart of a calibration method according to an embodiment of the invention; and -
Figures 4 is a block diagram of a computer system suitable for carrying out a method according to an embodiment of the invention. - As illustrated on the flow chart of
figure 3 , the depth camera DC to be calibrated is used to take a set DMS of depth maps of a reference three-dimensional physical object PO, an accurate three-dimensional digital model RM of which (reference model) is available. - Then, three-dimensional reconstruction of the reference object is performed by applying a suitable reconstruction algorithm RA to the depth maps. The reconstruction algorithm requires a set of calibration parameters of the depth camera - focal length(s), principal point, possibly also distortion parameters - which are unknown, and whose determination is the aim of the inventive method. To this aim, a sampling of a parameter space is performed, which provides a finite number of sets - or vectors - of parameters, SCPi, each characterized by a particular value of each calibration parameter. For example, if astigmatism and distortion are ignored, the depth camera is characterized by three intrinsic calibration parameters: its focal length f and the coordinates u0, v0 of its principal point. N parameter vectors SCPi=[fi, u0,i, v0,i] - with fi∈[fmin, fmax], u0,i∈[u0,min, u0,max], v0,i∈[v0,min, v0,max], i ∈[1, N] - are obtained by sampling the three-dimensional parameter space.
- A first sample of the parameter space SCP1 is chosen for performing the first reconstruction of the reference object. Other samples are used during successive iterations of the inventive method.
- Several reconstruction algorithms are known from the prior art and may be used to implement the inventive method, such as the one described in the above-reference paper by R.A. Newcombe et al. "KinectFusion: Real-Time Dense Surface Mapping and Tracking", Symposium ISMAR 2011, which is incorporated by reference. See also Richard Szeliski "Computer Vision: Algorithms and Applications". Springer, New York, 2010.
- This method uses a truncated signed distance field for iteratively fusing a plurality of depth maps into a unified representation of a scene including the reference object. As mentioned above, the distance field is obtained by defining a finite reconstruction volume, voxelizing it, and assigning to each voxel a signed and truncated distance d(x,y,z) from the 3D surface of the reference object, x,y,z being the coordinates of the reconstruction volume. The fact that the distance is signed allows differentiating the inside from the outside of the reference object. Each voxel is also characterized by a weight w which balances the relative importance of the information given by a current depth map for this voxel compared to all the other depth maps of the set DMS. It is worth noting that d(x,y,z) also defines an implicit function, i.e. the surface represented by the distance field is given by d(x,y,z) = 0, and thus the normal to surface at (x,y,z) is given by the gradient of d at (x,y,z).
- The reconstruction algorithm is carried out iteratively. A first depth map is arbitrary positioned relative to the distance field, and then fused into it according to the process described further.
- To perform the n-th iteration, the position and orientation (i.e. the pose) of the depth camera when acquiring the (n-1)th depth map are supposed known. Before being able to fuse the n-th depth map into the depth field, it is necessary to estimate the pose of the camera corresponding to this image, i.e. the rotation and translation defining its position and orientation with respect to a reference frame. This is performed using the so-called point-to-plane iterative closest point algorithm.
- According to this algorithm, ray-casting is used to compute an estimation (or "virtual view") of the current surface stored in the distance field - i.e. defined by d(x,y,z)=0 - viewed from the (n-1)th position of the depth camera. The virtual view is less noisy than the real view from the (n-1)th pose; moreover using a virtual view instead of a real one avoids a drift of the pose.
- A ray is traced from the position of the depth camera at time n-1, towards the reconstruction volume, passing through each pixel of this virtual view, and when it intersects the surface (i.e. when it goes from a voxel with positive distance to a next voxel with negative distance) an associated point is created at the surface d(x,y,z)=0. The normal to the surface at this point is determined by computing the gradient of the distance field at that point. Thus, a point cloud, called refPC, is obtained.
- Each pixel of the current (n-th) depth map is then back-projected to create another point cloud, and the normal to the surface corresponding to each point of the cloud is also determined by computing the cross product between points corresponding to neighboring pixels. This point cloud is called currPC.
- Then, registration between refPC and currPC is performed. First, the pose of the depth camera corresponding to the n-th depth map is initialized to coincide to the (n-1)th pose. Then, it is iteratively optimized until convergence is reached. Iterative optimization comprises the following steps:
- For each point in the currPC point cloud, the closest point in the refPC point cloud is found. To do so, each point of currPC is projected onto the virtual view, using the current n-th estimated pose, in order to associate it to a point of refPC. Outliers are rejected by removing correspondences where the angle between the two normals (i.e. the normal to the point of refPC and the normal to the point of currPC) is greater than a threshold, or the distance between the two corresponding points is greater than a threshold
- The combination of rotation and translation that best aligns each source point to its match found in the previous step is estimated. The Gauss-Newton algorithm is used to minimize a mean squared distance cost function, based on a point-to-plane distance defined by the distance of each point from currPC to the plane defined by its correspondent point in refPC and the normal at that point. Robust M-estimators can also be used to improve convergence. This point-to-plane distance is also more robustto inaccurate correspondences than a point-to-point distance.
- The source points (i.e. the points of refPC)are transformed using the obtained transformation.
- The steps above are iterated until the cost function becomes lower than, or lower or equal to, a defined threshold, or a maximum number of iterations is performed.
- Once the translation and orientation of the current depth map with respect to the frame of reference is known, said current depth map is fused into the distance field. To do so, the new information given by the current depth map is simply averaged with the previous information stored in each voxel of the distance field. Each voxel of the distance field is projected onto the depth map, and the distance of this voxel to the sensor is compared with the depth given by the depth map at the projection of the voxel. This gives the distance d' that should have the voxel according to the current depth map. Then averaging is performed (different schemes are possible):
where the index "n" or "n-1" refer the number of the iteration, and therefore of the pose. The weight w can be initialized to 1 and then increases according to the second equation, while being limited to a preset maximum value M. The weight w' can either take a constant value (e.g. 1) or a value depending on d' (to account for the fact that precision often decreases with distance) or on the incidence angle of the ray connecting the centre of the camera with the voxel. - Finally, the isosurface d(x,y,z)=0 of the distance field is extracted into a discrete triangular mesh representation of this surface. This can be done using the well-known Marching Cubes algorithm which proceeds through the distance field, taking eight neighbor voxels at a time, then determining the polygon(s) needed to represent the part of the isosurface that passes through these eight voxels.
- The three-dimensional reconstruction 3DRi of the reference object obtained this way is then aligned (registered) with the reference model RM to be compared to it in order to validate the estimation of the intrinsic parameters. Registration and comparison proper are jointly represented, on
figure 3 , by block "CMP". - Classical methods for 3D model registration are based on a strong description of a reduced subset of points, leading to robust correspondences between points belonging to different point clouds. The pipeline of this kind of approach is divided in three stages:
- The first stage consists in determining the subset of points that will be described. Those points must represent the most remarkable geometries of the point cloud, such as corners or edges. Therefore they are called "Keypoints" or "Points of Interest".
- The second stage consists in describing those points. The description is often based on their neighborhood. In order to match the points representing the same geometry and differentiate those of different geometries, the description must be fully detailed.
- Finally (third stage), the points having similar description are matched together. Those correspondences are not always right; in order to distinguish outliers (bad correspondences) from inliers (good correspondences), statistical methods, such as RANSAC (RANdom Sample Consensus) are often used. The final rigid transformation between the point clouds is worked out from the inliers. This approach is known from the paper by M. A. Fischler et al."Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography" Commun. ACM 24 (1981) 381 395, which is incorporated by reference.
- Alternative approaches to solve the registration problem can also be applied to the present invention. An approach allowing real-time computation is described in the paper by B. Drost et al.: "Model Globally, Match Locally: Efficient and Robust 3D Object Recognition", IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, California (USA), June 2010, which is also incorporated by reference. This approach is based on a simple and fast method for point pairs feature descriptor computation. In consequence, many point pairs will have a similar description resulting in a huge number of correspondences. Therefore, many of them will be wrong. A voting scheme is then carried on to discern the inliers from the outliers.
- Once the reconstructed model 3DRi is registered to the reference model RM, a 3D-to-3D distance between the models is computed to estimate an error ERRi between them. The error can be e.g. a mean square distance or a (possibly symmetrical) Hausdorff distance. The Hausdorff distance between a first surface S1 and a second surface S2, dH(S1, S2) is given by:
-
- Where p' is a point of S2 and ∥ ∥2 is the Euclidian norm.
-
- The paper by N. Aspert et al. "Measurement errors between surfaces using the Haussdorff distance", ICME 2002, which is incorporated by reference, describes an efficient algorithm for estimating an approximate Hausdorff distance between two surfaces.
- The error ERRi is then compared to a fixed threshold TH. If it is lower than, or lower or equal to, this threshold, the iteration stops and SCPi gives the optimal calibration parameters of the depth camera. Otherwise, "i" is increased (i.e. different trial values for the calibration parameters are chosen) and a new iteration is performed. If all the samples of the parameter space have been used (i.e. "i" has attained its maximum value), then the set of parameters corresponding to the lowest error is chosen. Alternatively, comparison to a threshold can be omitted, and full exploration of the parameter space is performed in any case.
- It can be easily understood that the calibration parameters found this way are optimal for 3D reconstruction.
- The inventive method (or, more precisely, its data processing part, following the acquisition of the depth maps) can be performed by a suitably-programmed general-purpose computer or computer system, possibly including a computer network, storing a suitable program in non-volatile form on a computer-readable medium such as a hard disk, a solid state disk or a CD-ROM and executing said program using its microprocessor(s) and memory.
- A computer suitable for carrying out a method according to an exemplary embodiment of the present invention is described with reference to
figure 4 . Infigure 4 , the computer includes a Central Processing Unit (CPU) P which performs the processes described above. The process can be stored as an executable program, i.e. a set of computer-readable instructions in memory, such as RAM M1 or ROM M2, or on hard disk drive (HDD) M3, DVD/CD drive M4, or can be stored remotely. At least one digital model of a reference object is also stored on one or more of memory devices M1 to M4, or remotely. - The claimed invention is not limited by the form of the computer-readable media on which the computer-readable instructions and/or the digital model(s) of the inventive process are stored. For example, the instructions, the trial sets of calibration parameters and the digital model(s) can be stored on CDs, DVDs, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk or any other information processing device with which the computer aided design station communicates, such as a server or computer. The program, the trial sets of calibration parameters and the digital model(s) can be stored on a same memory device or on different memory devices.
- Further, a computer program suitable for carrying out the inventive method can be provided as a utility application, background daemon, or component of an operating system, or combination thereof, executing in conjunction with CPU 800 and an operating system such as Microsoft VISTA, Microsoft Windows 7, UNIX, Solaris, LINUX, Apple MAC-OS and other systems known to those skilled in the art.
- CPU P can be a Xenon processor from Intel of America or an Opteron processor from AMD of America, or can be other processor types, such as a Freescale ColdFire, IMX, or ARM processor from Freescale Corporation of America. Alternatively, the CPU can be a processor such as a Core2 Duo from Intel Corporation of America, or can be implemented on an FPGA, ASIC, PLD or using discrete logic circuits, as one of ordinary skill in the art would recognize. Further, the CPU can be implemented as multiple processors cooperatively working to perform the computer-readable instructions of the inventive processes described above.
- The computer aided design station in
figure 4 also includes a network interface NI, such as an Intel Ethernet PRO network interface card from Intel Corporation of America, for interfacing with a network, such as a local area network (LAN), wide area network (WAN), the Internet and the like. The computer aided design station further includes a display controller DC, such as a NVIDIA GeForce GTX graphics adaptor from NVIDIA Corporation of America for interfacing with display DY, such as a Hewlett Packard HPL2445w LCD monitor. A general purpose I/O interface IF interfaces with a keyboard KB and pointing device PD, such as a roller ball, mouse, touchpad and the like. The display, the keyboard and the pointing device, together with the display controller and the I/O interfaces, form a graphical user interface. - Disk controller DKC connects HDD M3 and DVD/CD M4 with communication bus CBS, which can be an ISA, EISA, VESA, PCI, or similar, for interconnecting all of the components of the computer aided design station.
- A description of the general features and functionality of the display, keyboard, pointing device, as well as the display controller, disk controller, network interface and I/O interface is omitted herein for brevity as these features are known.
Claims (11)
- A method for calibrating a depth camera comprising the steps of:S1) using a depth camera (DC) to be calibrated for acquiring a plurality of depth maps (DMS) of a physical three-dimensional reference object (PO), corresponding to different positions of the depth camera relative to the reference object;S2) computing a three-dimensional reconstruction (3DRi) of the reference object from said depth maps, depending on values assigned to a set of calibration parameters (SCPi) of the depth camera;S3) computing an error (ERRi) between the three-dimensional reconstruction computed at step S2) and a predetermined digital model (RM) of the reference object;
said steps S2) and S3) being iterated by assigning different values to said set of calibration parameters of the depth camera; andS4) choosing values (SCP) of said set of calibration parameters which minimize the error computed at step S3);
said steps S2) to S4) being performed by a computer. - The method of claim 1, wherein said steps S2) and S3) are iterated a predetermined number of times by assigning respective predetermined values to said set of calibration parameters of the depth camera, said predetermined values sampling a parameter space.
- The method of claim 1, wherein said steps S2) and S3) are iterated until the error computed at step S3) becomes lower, or lower of equal, to a predetermined threshold (TH).
- The method of any of the preceding claims wherein said set of calibration parameters comprises at least a focal length and two coordinates of a principal point of the depth camera.
- The method of claim 4 wherein said set of calibration parameters also comprises at least one distortion parameter.
- The method of any of the preceding claims wherein said step S2) comprises jointly determining the position of the depth camera, relative to a reference position, corresponding to each of said depth maps, and fusing said depth maps into a single three-dimensional signed and truncated distance field; and extracting an isosurface of said distance field into a three-dimensional mesh constituting said reconstruction of the reference object.
- The method of any of the preceding claims wherein said step S3) comprises registering the three-dimensional reconstruction computed at step S2) and said predetermined digital model of the reference object prior to computing said error.
- The method of any of the preceding claims wherein said step S3) comprises computing an exact or approximate Hausdorff distance or symmetrical Hausdorff distance between the three-dimensional reconstruction computed at step S2) and said predetermined digital model of the reference object.
- A computer program product, stored on a computer-readable data-storage medium (M1, M2, M3, M4), comprising computer-executable instructions to cause a computer system to carry out steps S2) to S4) of a method according to any of the preceding claims.
- A computer-readable data-storage medium (M1, M2, M3, M4) containing computer-executable instructions to cause a computer system to carry out steps S2) to S4) of a method according to any of claims 1 to 8.
- A computer system comprising a processor (P) coupled to a memory (M1, M2, M3, M4) and a graphical user interface (IF, KB, PD, DC, DY), the memory storing computer-executable instructions to cause the computer system to carry out steps S2) to S4) of a method according to any of claims 1 to 8.
Priority Applications (6)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EP14307197.5A EP3040941B1 (en) | 2014-12-29 | 2014-12-29 | Method for calibrating a depth camera |
| CA2915214A CA2915214A1 (en) | 2014-12-29 | 2015-12-15 | Method for calibrating a depth camera |
| JP2015254978A JP6807639B2 (en) | 2014-12-29 | 2015-12-25 | How to calibrate the depth camera |
| CN201511000373.XA CN105741346B (en) | 2014-12-29 | 2015-12-28 | Method for calibrating a depth camera |
| KR1020150187819A KR20160082931A (en) | 2014-12-29 | 2015-12-28 | Method for calibrating a depth camera |
| US14/982,591 US10070121B2 (en) | 2014-12-29 | 2015-12-29 | Method for calibrating a depth camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EP14307197.5A EP3040941B1 (en) | 2014-12-29 | 2014-12-29 | Method for calibrating a depth camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| EP3040941A1 true EP3040941A1 (en) | 2016-07-06 |
| EP3040941B1 EP3040941B1 (en) | 2017-08-02 |
Family
ID=52946220
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP14307197.5A Active EP3040941B1 (en) | 2014-12-29 | 2014-12-29 | Method for calibrating a depth camera |
Country Status (6)
| Country | Link |
|---|---|
| US (1) | US10070121B2 (en) |
| EP (1) | EP3040941B1 (en) |
| JP (1) | JP6807639B2 (en) |
| KR (1) | KR20160082931A (en) |
| CN (1) | CN105741346B (en) |
| CA (1) | CA2915214A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018185363A1 (en) * | 2017-04-05 | 2018-10-11 | Oy Mapvision Ltd | Machine vision system |
| WO2021204865A1 (en) * | 2020-04-07 | 2021-10-14 | 3Shape A/S | Scanning system and calibration thereof |
Families Citing this family (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9878447B2 (en) * | 2015-04-10 | 2018-01-30 | Microsoft Technology Licensing, Llc | Automated collection and labeling of object data |
| US9609242B2 (en) * | 2015-06-25 | 2017-03-28 | Intel Corporation | Auto-correction of depth-sensing camera data for planar target surfaces |
| CN108022264B (en) * | 2016-11-01 | 2023-06-13 | 北京墨土科技有限公司 | Method and equipment for determining camera pose |
| KR102468897B1 (en) * | 2017-10-16 | 2022-11-21 | 삼성전자주식회사 | Method and apparatus of estimating depth value |
| US11568568B1 (en) * | 2017-10-31 | 2023-01-31 | Edge 3 Technologies | Calibration for multi-camera and multisensory systems |
| FI129042B (en) * | 2017-12-15 | 2021-05-31 | Oy Mapvision Ltd | Machine vision system with a computer generated virtual reference object |
| CN108234998A (en) * | 2018-01-09 | 2018-06-29 | 南京华捷艾米软件科技有限公司 | The accuracy measurement method of body-sensing camera and the accuracy measuring device of body-sensing camera |
| WO2019175629A1 (en) * | 2018-03-14 | 2019-09-19 | Nokia Technologies Oy | Camera calibration and/or use of a calibrated camera |
| KR102472156B1 (en) * | 2018-04-19 | 2022-11-30 | 삼성전자주식회사 | Electronic Device and the Method for Generating Depth Information thereof |
| CN109003294A (en) * | 2018-06-21 | 2018-12-14 | 航天科工仿真技术有限责任公司 | A kind of unreal & real space location registration and accurate matching process |
| CN108898630B (en) * | 2018-06-27 | 2020-12-15 | 清华-伯克利深圳学院筹备办公室 | A three-dimensional reconstruction method, apparatus, device and storage medium |
| CN110291771B (en) * | 2018-07-23 | 2021-11-16 | 深圳市大疆创新科技有限公司 | Depth information acquisition method of target object and movable platform |
| US10951809B2 (en) * | 2018-10-09 | 2021-03-16 | The Boeing Company | Adaptive camera control and calibration for dynamic focus |
| US10628968B1 (en) | 2018-12-05 | 2020-04-21 | Toyota Research Institute, Inc. | Systems and methods of calibrating a depth-IR image offset |
| US11436756B2 (en) * | 2018-12-20 | 2022-09-06 | Microsoft Technology Licensing, Llc | Calibrating a machine vision camera |
| CN109848073A (en) * | 2019-02-22 | 2019-06-07 | 浙江大学滨海产业技术研究院 | A kind of equipment and method for sorting coal and coal gangue |
| KR102297683B1 (en) * | 2019-07-01 | 2021-09-07 | (주)베이다스 | Method and apparatus for calibrating a plurality of cameras |
| CN111862139B (en) * | 2019-08-16 | 2023-08-18 | 中山大学 | A Parametric Modeling Method for Dynamic Objects Based on Color-Depth Camera |
| CN112907455B (en) * | 2019-12-04 | 2024-06-21 | 财团法人成大研究发展基金会 | Geometric camera correction system and method |
| CN113393528B (en) * | 2020-03-11 | 2024-12-17 | 阿里巴巴集团控股有限公司 | Method, device and system for calibrating camera external parameters in vehicle-mounted mobile measurement system |
| CN112422848B (en) * | 2020-11-17 | 2024-03-29 | 深圳市歌华智能科技有限公司 | Video stitching method based on depth map and color map |
| CN112381952B (en) * | 2020-11-25 | 2024-03-15 | 华南理工大学 | A multi-camera based facial contour point cloud model reconstruction method and device |
| KR102533909B1 (en) * | 2020-12-15 | 2023-05-26 | 광운대학교 산학협력단 | A multi-view camera-based iterative calibration method for the generation of a 3D volume model |
| CN112509064B (en) * | 2020-12-22 | 2024-06-04 | 阿波罗智联(北京)科技有限公司 | Method, device, equipment and storage medium for displaying camera calibration progress |
| GB2605621A (en) | 2021-04-07 | 2022-10-12 | Robok Ltd | Monocular depth estimation |
| CN113112551B (en) * | 2021-04-21 | 2023-12-19 | 阿波罗智联(北京)科技有限公司 | Methods, devices, roadside equipment and cloud control platform for determining camera parameters |
| CN116091610B (en) * | 2021-11-08 | 2023-11-10 | 南京理工大学 | Combined calibration method of radar and camera based on three-dimensional tower type checkerboard |
| DE102022103940A1 (en) * | 2022-02-18 | 2023-08-24 | Dspace Gmbh | Test setup and method for testing a stereo camera |
| CN116342802A (en) * | 2023-02-22 | 2023-06-27 | 中电海康集团有限公司 | Three-dimensional reconstruction method and device |
| CN117437290B (en) * | 2023-12-20 | 2024-02-23 | 深圳市森歌数据技术有限公司 | Multi-sensor fusion type three-dimensional space positioning method for unmanned aerial vehicle in natural protection area |
Family Cites Families (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5845288A (en) * | 1995-12-11 | 1998-12-01 | Xerox Corporation | Automated system for indexing graphical documents having associated text labels |
| US6868191B2 (en) * | 2000-06-28 | 2005-03-15 | Telefonaktiebolaget Lm Ericsson (Publ) | System and method for median fusion of depth maps |
| JP2004340753A (en) * | 2003-05-15 | 2004-12-02 | Topcon Corp | Calibration chart image display device |
| US8126253B2 (en) * | 2005-11-12 | 2012-02-28 | Cognex Technology And Investment Corporation | Automatically determining machine vision tool parameters |
| EP2132707A2 (en) * | 2006-12-18 | 2009-12-16 | Koninklijke Philips Electronics N.V. | Calibrating a camera system |
| JP5027746B2 (en) * | 2008-07-01 | 2012-09-19 | 株式会社トプコン | POSITION MEASUREMENT METHOD, POSITION MEASUREMENT DEVICE, AND PROGRAM |
| US8248476B2 (en) * | 2008-09-03 | 2012-08-21 | University Of South Carolina | Robust stereo calibration system and method for accurate digital image correlation measurements |
| US8866889B2 (en) * | 2010-11-03 | 2014-10-21 | Microsoft Corporation | In-home depth camera calibration |
| US8711206B2 (en) * | 2011-01-31 | 2014-04-29 | Microsoft Corporation | Mobile camera localization using depth maps |
| US8401242B2 (en) * | 2011-01-31 | 2013-03-19 | Microsoft Corporation | Real-time camera tracking using depth maps |
| KR101913321B1 (en) * | 2012-05-10 | 2018-10-30 | 삼성전자주식회사 | Method of geometry acquisition for specular object based on depth sensor and the device thereof |
| US20150178988A1 (en) * | 2012-05-22 | 2015-06-25 | Telefonica, S.A. | Method and a system for generating a realistic 3d reconstruction model for an object or being |
| EP2685403B1 (en) * | 2012-07-09 | 2025-04-23 | Deep Learning Robotics Ltd. | Natural machine interface system |
| JP6323993B2 (en) * | 2012-08-28 | 2018-05-16 | キヤノン株式会社 | Information processing apparatus, information processing method, and computer program |
| US9552668B2 (en) * | 2012-12-12 | 2017-01-24 | Microsoft Technology Licensing, Llc | Generation of a three-dimensional representation of a user |
| US9519968B2 (en) * | 2012-12-13 | 2016-12-13 | Hewlett-Packard Development Company, L.P. | Calibrating visual sensors using homography operators |
| JP6071522B2 (en) * | 2012-12-18 | 2017-02-01 | キヤノン株式会社 | Information processing apparatus and information processing method |
| US10268885B2 (en) * | 2013-04-15 | 2019-04-23 | Microsoft Technology Licensing, Llc | Extracting true color from a color and infrared sensor |
| CN103400409B (en) * | 2013-08-27 | 2016-08-10 | 华中师范大学 | A kind of coverage 3D method for visualizing based on photographic head attitude Fast estimation |
| US9864758B2 (en) * | 2013-12-12 | 2018-01-09 | Nant Holdings Ip, Llc | Image recognition verification |
| US9641830B2 (en) * | 2014-04-08 | 2017-05-02 | Lucasfilm Entertainment Company Ltd. | Automated camera calibration methods and systems |
-
2014
- 2014-12-29 EP EP14307197.5A patent/EP3040941B1/en active Active
-
2015
- 2015-12-15 CA CA2915214A patent/CA2915214A1/en not_active Abandoned
- 2015-12-25 JP JP2015254978A patent/JP6807639B2/en active Active
- 2015-12-28 CN CN201511000373.XA patent/CN105741346B/en active Active
- 2015-12-28 KR KR1020150187819A patent/KR20160082931A/en not_active Withdrawn
- 2015-12-29 US US14/982,591 patent/US10070121B2/en active Active
Non-Patent Citations (12)
| Title |
|---|
| B. DROST ET AL.: "Model Globally, Match Locally: Efficient and Robust 3D Object Recognition", IEEE COMPUTER SOCIETY CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR, June 2010 (2010-06-01) |
| GUI POPO ET AL: "Accurately calibrate kinect sensor using indoor control field", 2014 THIRD INTERNATIONAL WORKSHOP ON EARTH OBSERVATION AND REMOTE SENSING APPLICATIONS (EORSA), IEEE, 11 June 2014 (2014-06-11), pages 9 - 13, XP032664472, DOI: 10.1109/EORSA.2014.6927839 * |
| HERRERA C., D.; KANNALA, J.; HEIKKI A, J.: "Joint depth and color camera calibration with distortion correction", TPAMI, 2012 |
| KAI BERGER ET AL: "Markerless Motion Capture using multiple Color-Depth Sensors", 1 January 2011 (2011-01-01), XP055197584, Retrieved from the Internet <URL:http://dx.doi.org/10.2312/PE/VMV/VMV11/317-324> [retrieved on 20150625], DOI: 10.2312/PE/VMV/VMV11/317-324 * |
| KOICHIRO YAMAUCHI ET AL: "3D Reconstruction of a Human Body from Multiple Viewpoints", 17 December 2007, ADVANCES IN IMAGE AND VIDEO TECHNOLOGY; [LECTURE NOTES IN COMPUTER SCIENCE], SPRINGER BERLIN HEIDELBERG, BERLIN, HEIDELBERG, PAGE(S) 439 - 448, ISBN: 978-3-540-77128-9, XP019085152 * |
| M. A. FISCHLER ET AL.: "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography", COMMUN. ACM, vol. 24, 1981, pages 381 395 |
| N. ASPERT ET AL.: "Measurement errors between surfaces using the Haussdorff distance", ICME, 2002 |
| R.A. NEWCOMBE ET AL.: "KinectFusion: Real-Time Dense Surface Mapping and Tracking", SYMPOSIUM ISMAR, 2011 |
| R.Y. TSAI: "An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision", PROCEEDINGS OF IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 1986, pages 364 - 374 |
| RICHARD A NEWCOMBE ET AL: "KinectFusion: Real-time dense surface mapping and tracking", MIXED AND AUGMENTED REALITY (ISMAR), 2011 10TH IEEE INTERNATIONAL SYMPOSIUM ON, IEEE, 26 October 2011 (2011-10-26), pages 127 - 136, XP032201443, ISBN: 978-1-4577-2183-0, DOI: 10.1109/ISMAR.2011.6092378 * |
| RICHARD SZELISKI: "Computer Vision: Algorithms and Applications", 2010, SPRINGER |
| ZHANG, ZHENGYOU: "Flexible Camera Calibration By Viewing a Plane From Unknown Orientations", ICCV, 1999 |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018185363A1 (en) * | 2017-04-05 | 2018-10-11 | Oy Mapvision Ltd | Machine vision system |
| US11087455B2 (en) | 2017-04-05 | 2021-08-10 | Oy Mapvision Ltd | Machine vision system |
| WO2021204865A1 (en) * | 2020-04-07 | 2021-10-14 | 3Shape A/S | Scanning system and calibration thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| US20160189358A1 (en) | 2016-06-30 |
| CN105741346B (en) | 2021-05-04 |
| EP3040941B1 (en) | 2017-08-02 |
| JP6807639B2 (en) | 2021-01-06 |
| CN105741346A (en) | 2016-07-06 |
| CA2915214A1 (en) | 2016-06-29 |
| KR20160082931A (en) | 2016-07-11 |
| JP2016128810A (en) | 2016-07-14 |
| US10070121B2 (en) | 2018-09-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3040941B1 (en) | Method for calibrating a depth camera | |
| US10334168B2 (en) | Threshold determination in a RANSAC algorithm | |
| US9858670B2 (en) | Information processing apparatus and method thereof | |
| CN110176075B (en) | System and method for simultaneous consideration of edges and normals in image features through a vision system | |
| TW202034215A (en) | Mapping object instances using video data | |
| CN110462685B (en) | Three-dimensional model reconstruction method and system | |
| WO2014024579A1 (en) | Optical data processing device, optical data processing system, optical data processing method, and optical data processing-use program | |
| EP3633606B1 (en) | Information processing device, information processing method, and program | |
| CN108648194A (en) | Based on the segmentation of CAD model Three-dimensional target recognition and pose measuring method and device | |
| EP3300025B1 (en) | Image processing device and image processing method | |
| CN115601430A (en) | Texture-free high-reflection object pose estimation method and system based on key point mapping | |
| Cui et al. | ACLC: Automatic calibration for nonrepetitive scanning LiDAR-camera system based on point cloud noise optimization | |
| Siddique et al. | 3d object localization using 2d estimates for computer vision applications | |
| Bracci et al. | Challenges in fusion of heterogeneous point clouds | |
| Tylecek et al. | Depth map fusion with camera position refinement | |
| Mkhitaryan et al. | RGB-D sensor data correction and enhancement by introduction of an additional RGB view | |
| Afzal et al. | Kinect Deform: enhanced 3d reconstruction of non-rigidly deforming objects | |
| P Duarte et al. | Automatic camera calibration using a single image to extract intrinsic and extrinsic parameters | |
| Kovacs et al. | Edge detection in discretized range images | |
| Charron et al. | A target-based extrinsic calibration framework for non-overlapping camera-lidar systems using a motion capture system | |
| Antipin et al. | Improving the Quality of the Disparity Map | |
| Cheng et al. | An iterative refinement method for pose estimation from planar target | |
| Lari et al. | Quality analysis of 3D surface reconstruction using multi-platform photogrammetric systems | |
| Sølund et al. | Teach it yourself-fast modeling of industrial objects for 6D pose estimation | |
| KR20250081383A (en) | Method and appartus for reconstructing high-precision 3d spatial data using lidar sensor point cloud and camera image data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
| AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
| AX | Request for extension of the european patent |
Extension state: BA ME |
|
| 17P | Request for examination filed |
Effective date: 20161227 |
|
| RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
| REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602014012489 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: G06T0007000000 Ipc: G06T0007800000 |
|
| GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
| RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06T 7/80 20170101AFI20170209BHEP |
|
| INTG | Intention to grant announced |
Effective date: 20170313 |
|
| GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
| GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
| AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
| REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP Ref country code: AT Ref legal event code: REF Ref document number: 915258 Country of ref document: AT Kind code of ref document: T Effective date: 20170815 |
|
| REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
| REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602014012489 Country of ref document: DE |
|
| REG | Reference to a national code |
Ref country code: SE Ref legal event code: TRGR |
|
| REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 4 |
|
| REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20170802 |
|
| REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 915258 Country of ref document: AT Kind code of ref document: T Effective date: 20170802 |
|
| REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171102 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171103 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171102 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171202 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 |
|
| REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602014012489 Country of ref document: DE |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 |
|
| PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
| 26N | No opposition filed |
Effective date: 20180503 |
|
| REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 |
|
| REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171229 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171229 |
|
| REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20171231 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171229 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171231 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171231 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20171231 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20141229 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170802 |
|
| P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230529 |
|
| PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20251119 Year of fee payment: 12 |
|
| PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20251113 Year of fee payment: 12 |
|
| PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: IT Payment date: 20251126 Year of fee payment: 12 |
|
| PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20251124 Year of fee payment: 12 |
|
| PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: SE Payment date: 20251126 Year of fee payment: 12 |