KR20140052824A - Apparatus and method for calibrating depth image based on depth sensor-color camera relations - Google Patents

Abstract

Disclosed is a method for calibrating a depth image based on the relationship between a depth sensor and a photographing camera. A depth image calibrating apparatus comprises a 3D point determiner for determining a 3D point of a camera image and a 3D point of a depth image simultaneously captured with the camera image; a calibration information determiner for determining calibration information in order to calibrate an error of a depth sensor photographing the depth image and the geometric relationship between the depth sensor and a photographing camera, by using the 3D point of the camera image and the 3D point of the depth image; and a depth image calibrator for calibrating the depth image based on the calibration information and the 3D point of the depth image.

Description

TECHNICAL FIELD [0001] The present invention relates to a depth image correcting apparatus and method based on a relationship between a depth sensor and a photographing camera,

The present invention relates to an apparatus and method for correcting a depth image, and more particularly, to an apparatus and method for correcting a depth image based on a relationship between a depth sensor and a photographing camera.

Depth sensors that triangulate using structured light, such as Microsoft's Kinect, can result in errors when they are used by the end user due to assembly tolerances in manufacturing and errors in the optics.

In addition, when extracting the texture of the object simultaneously using the depth sensor and the photographing camera, the geometric relationship between the depth sensor and the camera must be obtained. If the geometric relationship is not precisely determined, an error occurs in the texture information about the object to be reconstructed three-dimensionally.

Since the conventional depth image correction apparatus obtains the geometrical relationship with the photographing camera based on the depth image measured by the depth sensor, the depth image corrected by the conventional depth image correcting apparatus includes the error according to the error included in the depth sensor can do.

Therefore, there is a demand for a method of correcting an error of a depth sensor and correcting a depth image by obtaining a geometric relationship between a depth sensor and a photographing camera.

The present invention improves the accuracy of the depth sensor when the depth image is corrected using the depth sensor for measuring the distance to the object and the photographing camera for photographing the object simultaneously and corrects the geometric relationship between the depth sensor and the photographing camera Apparatus and method.

The depth image correction apparatus according to an embodiment of the present invention includes a three-dimensional point determiner for determining a three-dimensional point of a camera image and a three-dimensional point of a depth image photographed simultaneously with the camera image; Determining a correction information for correcting an error of the depth sensor that captures the depth image and a geometric geometric relationship between the depth sensor and the photographing camera using a point on the three-dimensional image of the camera image and a point on the three- A correction information determination unit for determining a correction information; And a depth image correction unit for shifting the three-dimensional point of the depth image to a camera coordinate system based on the correction information.

The three-dimensional point determination unit of the depth image correction apparatus according to an embodiment of the present invention self-corrects the camera image, and a point on the three-dimensional image of the camera image feature point used for self-correction of the camera image is divided into three It can be decided by the dimension point.

The three-dimensional point determination unit of the depth image correction apparatus according to an exemplary embodiment of the present invention extracts a depth image feature point corresponding to the camera image feature point from the depth image, Dimensional point of the depth image.

The correction information determination unit of the depth image correction apparatus according to an exemplary embodiment of the present invention generates Equation 1, Equation 2, and Equation 3 using the three-dimensional point of the camera image and the three-dimensional point of the depth image .

The depth image correction unit of the depth image correction apparatus according to an exemplary embodiment of the present invention defines a cost function having an initial value of the correction information and performs a nonlinear optimization process to minimize a defined cost function to calculate an error of a depth sensor, It is possible to correct the geometric relationship between the photographing camera and the photographing camera.

The depth image correcting apparatus according to an embodiment of the present invention corrects the depth image of the depth image by using the equation of the pattern surface restored by the photographing camera and the equation of the pattern surface restored by the depth sensor, A correction information determination unit for determining correction information for correcting the geometrical relationship between the photographing cameras; And a depth image correction unit for shifting the three-dimensional point of the depth image to a camera coordinate system based on the correction information.

The correction information determination unit of the depth image correction apparatus according to an embodiment of the present invention may be configured such that the equation of the pattern surface restored by the photographing camera is Equation 5 and the equation of the pattern surface restored by the depth sensor is Equation 6, (7), (8), and (9) based on Equation (5) and Equation (6).

The correction information determination unit of the depth image correction apparatus according to an exemplary embodiment of the present invention calculates a factor included in the matrix L, which is the inverse matrix of the matrix H including the correction information, according to Equation 1, Equation 2, and Equation 3 The relationship information between the coordinates of the depth image and the camera coordinates can be determined by replacing the correction information included in the depth image.

According to an embodiment of the present invention, there is provided a method of correcting an image, comprising: determining a three-dimensional point of a camera image and a three-dimensional point of a depth image photographed simultaneously with the camera image; Determining a correction information for correcting an error of the depth sensor that captures the depth image and a geometric geometric relationship between the depth sensor and the photographing camera using a point on the three-dimensional image of the camera image and a point on the three- ; And moving the three-dimensional point of the depth image to the camera coordinate system based on the correction information.

The depth image correcting apparatus according to an embodiment of the present invention corrects the depth image of the depth image by using the equation of the pattern surface restored by the photographing camera and the equation of the pattern surface restored by the depth sensor, Determining correction information for correcting the geometric relationship between the photographing cameras; And moving the three-dimensional point of the depth image to the camera coordinate system based on the correction information.

According to an embodiment of the present invention, an error of a depth sensor that captures the depth image using a point on a three-dimensional image of the camera image and a three-dimensional point of the depth image captured simultaneously with the camera image, It is possible to acquire a precise depth image by correcting the geometrical relationship between

1 is a diagram illustrating a relationship between a depth image correction apparatus and other apparatuses according to an exemplary embodiment of the present invention.
2 is an example of a correction pattern according to an embodiment of the present invention.
3 is an example of a depth image and a camera image according to an embodiment of the present invention.
4 is a diagram illustrating a depth image correction apparatus according to an embodiment of the present invention.
5 is an example of a process of matching a feature point of a depth image and a feature point of a camera image according to an embodiment of the present invention.
6 is a flowchart illustrating an image correction method according to an embodiment of the present invention.
7 is a flowchart showing another example of the image correction method according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. An image correction method according to an embodiment of the present invention may be performed by a depth image correction apparatus.

1 is a diagram illustrating a relationship between a depth image correction apparatus and other apparatuses according to an exemplary embodiment of the present invention.

The photographing camera 120 may photograph the correction pattern 110 to generate a camera image including the correction pattern. At this time, the correction pattern 110 may be a three-dimensional object having a plurality of marks formed on the surface thereof so that the geometric relationship between the photographing camera 120 and the depth sensor 130 can be easily identified.

The shape of the correction pattern 110 will be described in detail below with reference to FIG.

In addition, the depth sensor 130 can simultaneously generate the depth image by photographing the correction pattern 110 at the same time as the photographing camera 120. At this time, the depth sensor 130 can triangulate the distance between the correction pattern 110 and the depth sensor 130 using the structured light, and generate the depth image based on the measured distance.

The photographing camera 120 and the depth sensor 130 may generate a plurality of camera images and depth images, respectively, according to the image correction method, and transmit the camera images and the depth images to the depth image correction device 100.

The depth image correction apparatus 100 identifies a geometric relationship between the photographing camera 120 and the depth sensor 130 by using the depth image received from the depth sensor 130 and the camera image received from the photographing camera 120, The depth image can be corrected based on the identified relationship.

2 is an example of a correction pattern according to an embodiment of the present invention.

The correction pattern 110 captured by the photographic camera 120 and the depth sensor 130 in the embodiment of the present invention is a polyhedron 200 having square surfaces on each side as shown in FIG. The circle 210 may be drawn.

At this time, the depth image correction apparatus 100 corrects the correction pattern 110 included in the depth image and the correction pattern 110 included in the camera image based on the position of the circle 210 on each surface of the correction pattern 110 And can identify the geometric relationship between the photographing camera 120 and the depth sensor 130 based on the matching result.

3 is an example of a depth image and a camera image according to an embodiment of the present invention.

The depth sensor 130 and the photographing camera 120 may generate at least one depth image 310 and a camera image 320 according to the image correction method and transmit the generated depth image 310 and the camera image 320 to the depth image correction apparatus 100, respectively.

The depth image 310 and the camera image 320 are generated by photographing the same correction pattern 110 by the depth sensor 130 and the photographing camera 120, respectively. 3, the correction pattern 110 included in the depth image 310 and the correction pattern 110 included in the camera image 320 (see FIG. 3), which correspond to the geometric relationship between the camera 120 and the depth sensor 130, 110 may have different sizes and angles.

4 is a diagram illustrating a depth image correction apparatus according to an embodiment of the present invention.

4, the depth image correction apparatus 100 according to an exemplary embodiment of the present invention includes a 3D point determination unit 410, a correction information determination unit 420, and a depth image correction unit 430 .

The three-dimensional point determination unit 410 can determine a three-dimensional point of the camera image and a three-dimensional point of the depth image photographed simultaneously with the camera image.

At this time, the three-dimensional point determining unit 410 can self-correct the camera image and determine the three-dimensional point of the camera image feature point used for the self-correction of the camera image as the three-dimensional point of the camera image. At this time, the three-dimensional point determination unit 410 can select one of the methods used in the field of camera correction to correct the camera image itself. The three-dimensional point determination unit 410 may define a three-dimensional point on the camera image feature point as (X, Y, Z).

The three-dimensional point determination unit 410 extracts a depth image feature point corresponding to the camera image feature point from the depth image, and determines a three-dimensional point corresponding to the extracted depth image feature point as a three-dimensional point of the depth image . At this time, the three-dimensional point determining unit 410 may define a three-dimensional point on the depth image characteristic point as (X ', Y', Z ').

The correction information determination unit 420 determines the error of the depth sensor 130 and the geometric geometry between the depth sensor 130 and the photographing camera 120 using the point on the 3D image of the camera image and the point on the 3D image of the depth image, The correction information for correcting the relationship can be determined.

For example, the correction information determination unit 420 determines the correction information by using Equation 1, Equation 2, and Equation 3 using the three-dimensional point of the camera image determined by the three-dimensional point determination unit 410 and the three- Can be created at the same time.

Then, the correction information determination unit 420 simultaneously generates Equation 1, Equation 2, and Equation 3 for all corresponding points in the camera image and the depth image, and outputs the generated Equations 1, 2, Equation (3) can be expressed in the form of a matrix to generate a homogeneous equation as shown in Equation (4).

Here, h is a variable (H ₁₁ H ₁₂ H ₁₃ H ₁₄ H ₂₁ H ₂₂ H ₂₃ H ₂₄ H ₃₁ H ₃₂ H ₃₃ H ₃₄ H ₄₁ H ₄₂ H ₄₃ H ₄₄ ).

Also, A may be a 3m X 16 matrix composed of points (X, Y, Z) on the three-dimensional plane of the camera image feature point and points (X ', Y', Z ') on the three-dimensional plane of the depth image feature point .

Here, m is the number of points on the three-dimensional image included in the camera image or the depth image. Specifically, since the three-dimensional point of the depth image feature point is determined according to the depth image feature point corresponding to the camera image feature point, the number of points on the three-dimensional image of the camera image feature point included in the camera image, The number of points on the three-dimensional image of the included depth image feature point may be the same. Therefore, the correction information determination unit 420 can select one of the number of points on the three-dimensional image included in the camera image, or the number of points on the three-dimensional image included in the depth image, and set it as m.

The correction information determination unit 420 may calculate the values of the variables included in Equation (4) and determine the values of the calculated variables as correction information.

The correction information determination unit 420 determines the depth information of the depth sensor 130 that captures the depth image using the equation of the pattern surface restored by the photographing camera 120 and the equation of the pattern surface restored by the depth sensor 130 The correction information for correcting the geometrical relationship between the error and the depth sensor 130 and the photographing camera 120 may be determined.

For example, when the equation of the pattern surface of the correction pattern 110 obtained by the camera 120 is restored from the camera image is Equation 5 and the equation of the pattern surface of the correction pattern 110 restored by the depth sensor 130 is ≪ / RTI >

At this time, the correction information determination unit 420 may generate Equations (7), (8), and (9) based on Equations (1), (2), and (3). Specifically, the correction information determination unit 420 changes (X, Y, Z) to (a, b, c) in Equations 1, 2, (A ', b', c ') to generate Equations (7), (8), and (9).

The correction information determination unit 420 may generate a homogeneous equation expressed by Equation (10) by expressing Equation (7), Equation (8), and Equation (9) in the form of a matrix.

Here, h is a variable (H ₁₁ H ₁₂ H ₁₃ H ₁₄ H ₂₁ H ₂₂ H ₂₃ H ₂₄ H ₃₁ H ₃₂ H ₃₃ H ₃₄ H ₄₁ H ₄₂ H ₄₃ H ₄₄ ).

Further, B may be a 3m X 16 matrix composed of (a ', b', c ') and (a', b ', c') of Equation (5).

Here, m is the number of pattern surfaces reconstructed from the camera image or the depth image. Specifically, since the restored pattern surface in the depth image is determined according to the restored pattern surface in the depth image corresponding to the pattern surface restored in the camera image, the number of restored pattern surfaces included in the camera image is included in the depth image The number of reconstructed pattern planes may be the same. Accordingly, the correction information determination unit 420 may select one of the number of restored pattern planes included in the camera image, or the number of restored pattern planes included in the depth image, and set the selected number as m.

Further, the correction information determination unit 420 can obtain the matrix H of the equation (11) using the equation (10) of equation (10)

Finally, the correction information determination unit 420 may calculate the values of the variables included in the matrix H using Equation (10), and determine the values of the calculated parameters as correction information. At this time, the variable H ₁₁ H ₁₂ H ₁₃ H ₁₄ H ₂₁ H ₂₂ H ₂₃ H ₂₄ H ₃₁ H ₃₂ H ₃₃ H ₃₄ H ₄₁ H ₄₂ H ₄₃ H ₄₄ ) may be a variable included in h of Equation (10). That is, the correction information determination unit 420 may convert the column vector h into the matrix H, and determine the converted H as the correction information.

At this time, the correction information determination unit 420 replaces the factor included in the matrix L, which is the inverse matrix of the matrix H, with the correction information included in Equations (1), (2) and (3) Can be determined. For example, the correction information determination unit 420 can determine the matrix L using Equation (12).

Specifically, the correction information determination unit 420 determines the factor L _ij of the matrix L as a camera image and H _ij It is possible to determine the relationship information indicating the conversion system for the point (X ', Y', Z ') and the point (X, Y, Z)

The depth image correcting unit 430 can correct the depth image based on the three-dimensional point included in the depth image and the correction information determined by the correction information determining unit 420. [

For example, when the point on the three-dimensional image included in the depth image is (X ', Y', Z '), the depth image correction unit 430 calculates (X' Y ', Z') and correction information (H ₁₁ H ₁₂ H ₁₃ H ₁₄ H ₂₁ H ₂₂ H ₂₃ H ₂₄ H ₃₁ H ₃₂ H ₃₃ H ₃₄ H ₄₁ H ₄₂ H ₄₃ H ₄₄ ) can be substituted to determine the point (X, Y, Z) on the three-dimensional plane.

At this time, the three-dimensional point (X, Y, Z) determined by the depth image correcting unit 430 corrects the error according to the geometrical relationship between the photographing camera 120 and the depth sensor 130, It can be a value that moves the point on the dimension to the camera coordinate system.

That is, the depth image correcting unit 430 can determine the three-dimensional point (X, Y, Z) and correct it according to the geometrical relationship between the photographing camera 120 and the depth sensor 130.

In addition, the depth image correction unit 430 defines a cost function that sets the correction information h as an initial value, performs a nonlinear optimization process to minimize the defined cost function, and calculates a geometric error between the depth sensor and the depth sensor Can be corrected more precisely.

For example, the depth image correction unit 430 may define a cost function as shown in Equation (13).

At this time, P = (X, Y, Z) and P "= (X", Y ", Z"). Further, (X ", Y", Z ") may be a result of substituting (X ', Y', Z ') into Equations (1), (2) and (3).

5 is an example of a process of matching a feature point of a depth image and a feature point of a camera image according to an embodiment of the present invention.

The three-dimensional point determining unit 410 may correct the camera image 520 and determine a three-dimensional point of the camera image feature point 521 used for self-correction of the camera image as a three-dimensional point of the camera image .

The 3D point determining unit 410 extracts a depth image feature point 511 corresponding to the camera image feature point 521 in the depth image 510 and extracts the depth image feature point 511 corresponding to the extracted depth image feature point 511 The point on the dimension can be determined as the three-dimensional point of the depth image.

6 is a flowchart illustrating an image correction method according to an embodiment of the present invention.

6 is an image correction method using three-dimensional points.

In step 610, the three-dimensional point determination unit 410 may receive the camera image and the depth image captured by the imaging camera 120 and the depth sensor 130 at the same time.

In step 620, the 3D point determining unit 410 may determine a three-dimensional point on the camera image received in step 610 and a 3D point on the depth image received in step 610.

Specifically, the three-dimensional point determining unit 410 can self-correct the camera image and determine the three-dimensional point of the camera image feature point used for self-correction of the camera image as the three-dimensional point of the camera image. Next, the three-dimensional point determination unit 410 extracts a depth image feature point corresponding to the camera image feature point from the depth image, and calculates a three-dimensional point corresponding to the extracted depth image feature point as a three- You can decide.

In step 630, the correction information determination unit 420 determines an error of the depth sensor 130, a depth sensor, and a photographing camera using the three-dimensional point on the camera image and the three- The correction information for correcting the geometric relationship between the reference point and the reference point can be determined.

For example, the correction information determination unit 420 determines the correction information by using the three-dimensional point of the camera image and the three-dimensional point of the depth image determined by the three-dimensional point determination unit 410, Equation 2 and Equation 3 can be generated. Then, the correction information determination unit 420 generates Equation 1, Equation 2, and Equation 3 for all corresponding points in the camera image and the depth image, and outputs the generated Equations 1, 2, 3 can be expressed in the form of a matrix to generate a homogeneous equation as shown in Equation (4). Finally, the correction information determination unit 420 may calculate the values of the variables included in Equation (4) and determine the values of the calculated variables as correction information.

In addition, the correction information determination unit 420 replaces the factor included in the matrix L, which is the inverse displacement matrix of the matrix H including the correction information, with the correction information included in Equations (1), (2) and (3) And the camera coordinate can be determined.

In step 640, the depth image correction unit 430 may define a cost function having h as an initial value and perform a nonlinear optimization process to minimize the defined cost function. Step 650 is performed to accurately correct the camera image, and step 650 may be omitted depending on the degree of correction of the camera image.

The depth image correction unit 430 may correct the depth image based on the three-dimensional point included in the depth image and the correction information determined by the correction information determination unit 420 in step 630 in step 650.

For example, when the point on the three-dimensional image included in the depth image is (X ', Y', Z '), the depth image correcting unit 430 calculates (X', Y ', Z ') And correction information (H ₁₁ H ₁₂ H ₁₃ H ₁₄ H ₂₁ H ₂₂ H ₂₃ H ₂₄ H ₃₁ H ₃₂ H ₃₃ H ₃₄ H ₄₁ H ₄₂ H ₄₃ H ₄₄ ) can be substituted to determine the point (X, Y, Z) on the three-dimensional plane.

In addition, the depth image correcting unit 430 may correct the depth image based on the three-dimensional image point included in the depth image and the cost function minimized in step 650.

7 is a flowchart showing another example of the image correction method according to an embodiment of the present invention.

7 is an image correction method using an equation of a pattern surface.

In step 710, the three-dimensional point determination unit 410 may receive the camera image and the depth image captured by the imaging camera 120 and the depth sensor 130 at the same time.

In step 720, the 3D point determining unit 410 may restore the equation of the pattern surface of the correction pattern 110 from the camera image received in step 710. In addition, the three-dimensional point determination unit 410 may restore the equation of the pattern surface of the correction pattern 110 from the depth image received in step 710.

In step 730, the correction information determination unit 420 determines whether the depth sensor 130 captures the depth image using the equation of the pattern surface of the camera image restored in step 720 and the equation of the pattern surface of the depth image The correction information for correcting the geometrical geometry between the error and depth sensor 130 and the photographing camera 120 can be determined

At this time, the correction information determination unit 420 may generate Equations (7), (8), and (9) based on Equations (1), (2), and (3). Specifically, the correction information determination unit 420 changes (X, Y, Z) to (a, b, c) in Equations 1, 2, (A ', b', c ') to generate Equations (7), (8), and (9).

The correction information determination unit 420 may generate a homogeneous equation expressed by Equation (10) by expressing Equation (7), Equation (8), and Equation (9) in the form of a matrix.

Next, the correction information determination unit 420 may calculate the values of the variables included in equation (10) and determine the values of the calculated variables as correction information.

In addition, the correction information determination unit 420 replaces the factor included in the matrix L, which is the inverse displacement matrix of the matrix H including the correction information, with the correction information included in Equations (1), (2) and (3) And the camera coordinate can be determined.

In step 740, the depth image correcting unit 430 may define a cost function that sets the correction information h as an initial value, and perform a nonlinear optimization process to minimize the defined cost function. The step 750 is performed when the camera image is to be precisely corrected, and the execution of the step 750 may be omitted depending on the degree of correction of the camera image.

The depth image correction unit 430 may correct the depth image based on the three-dimensional point included in the depth image and the correction information determined by the correction information determination unit 420 in step 730 in step 750.

In addition, the depth image correcting unit 430 may correct the depth image based on the three-dimensional point included in the depth image and the cost function minimized in step 750.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

100: Depth image correction device
410: three-dimensional point determining unit
420: correction information determination unit
430: Depth image correction unit

Claims (18)

A three-dimensional point determiner for determining a three-dimensional point of the camera image and a three-dimensional point of the depth image photographed simultaneously with the camera image;
Wherein the correction information is used to correct an error of the depth sensor that captures the depth image and a geometric geometry relationship between the depth sensor and the photographing camera using a point on the three-dimensional image of the camera image and a point on the three- A correction information determination unit determining the correction information; And
And a depth image correcting unit for correcting the depth image based on the correction information and a three-dimensional point on the depth image,
And a depth image correction unit. The method according to claim 1,
Wherein the three-
Extracts a depth image characteristic point corresponding to a camera image characteristic point used for self-correcting the camera image in the depth image, and determines a three-dimensional point corresponding to the extracted depth image characteristic point as a three-dimensional point of the depth image A depth image correction device The method according to claim 1,
Wherein the correction information determination unit determines,
Wherein the depth image correction unit generates Equations (1), (2), and (3) using a three-dimensional point of the camera image and a three-dimensional point of the depth image.
[Equation 1]
X = (X '* H ₁₁ + Y' * H ₁₂ + Z '* H ₁₃ + H ₁₄ ) / (X '* H ₄₁ + Y' * H ₄₂ + Z '* H ₄₃ + H ₄₄ )
&Quot; (2) "
Y = (X '* H ₂₁ + Y' * H ₂₂ + Z '* H ₂₃ + H ₂₄ ) / (X '* H ₄₁ + Y' * H ₄₂ + Z '* H ₄₃ + H ₄₄ )
&Quot; (3) "
Z = (X '* H ₃₁ + Y' * H ₃₂ + Z '* H ₃₃ + H ₃₄ ) / (X '* H ₄₁ + Y' * H ₄₂ + Z '* H ₄₃ + H ₄₄ )
X ', Y', and Z 'are coordinates of a point on the three-dimensional image of the depth image, and (H ₁₁ H ₁₂ H ₁₃ H ₁₄ H ₂₁ H ₂₂ H ₂₃ H ₂₄ H ₃₁ H ₃₂ H ₃₃ H ₃₄ H ₄₁ H ₄₂ H ₄₃ H ₄₄ ) may be correction information. The method according to claim 1,
Wherein the correction information determination unit determines,
A depth image correcting unit for generating Equation (4) by using a three-dimensional point on the camera image and a three-dimensional point on the depth image, and determining correction information using Equation (4).
&Quot; (4) "
Ah = 0
(Where A is a 3m * 16 matrix composed of three-dimensional points on the camera image and three-dimensional points on the depth image, h is correction information, m is a camera image, or 3 It may be the number of points on the dimension.) The method according to claim 1,
Wherein the depth image correction unit comprises:
And a nonlinear optimization process for minimizing the defined cost function to define a geometric geometric relationship between the depth sensor and the photographing camera by defining a cost function having the correction information as an initial value and correcting the geometric geometric relationship between the depth sensor and the photographing camera. An error of the depth sensor that captures the depth image and a correction for correcting the geometric geometry relationship between the depth sensor and the photographing camera by using the equation of the pattern surface restored by the photographing camera and the equation of the pattern surface restored by the depth sensor, A correction information determination unit for determining information; And
And a depth image correcting unit for correcting the depth image based on the correction information and a three-dimensional point on the depth image,
And a depth image correction unit. The method according to claim 6,
Wherein the correction information determination unit determines,
When the equation of the pattern surface restored by the photographing camera is Equation 5 and the equation of the pattern surface restored by the depth sensor is Equation 6, Equation 7, Equation 8, and Equation 7 are calculated based on Equation 5 and Equation 6, (9). ≪ / RTI >
&Quot; (5) "
aX + bY + cZ + 1 = 0
&Quot; (6) "
a'X + b'Y + c'Z + 1 = 0
&Quot; (7) "
a = (a '* H ₁₁ + b' * H ₁₂ + c '* H ₁₃ + H ₁₄ ) / (a '* H ₄₁ + b' * H ₄₂ + c '* H ₄₃ + H ₄₄ )
&Quot; (8) "
b = (a '* H ₂₁ + b' * H ₂₂ + c '* H ₂₃ + H ₂₄ ) / (a '* H ₄₁ + b' * H ₄₂ + c '* H ₄₃ + H ₄₄ )
&Quot; (9) "
c = (a '* H ₃₁ + b' * H ₃₂ + c '* H ₃₃ + H ₃₄ ) / (a '* H ₄₁ + b' * H ₄₂ + c '* H ₄₃ + H ₄₄ )
(Here, (H ₁₁ H ₁₂ H ₁₃ H ₁₄ H ₂₁ H ₂₂ H ₂₃ H ₂₄ H ₃₁ H ₃₂ H ₃₃ H ₃₄ H ₄₁ H ₄₂ H ₄₃ H ₄₄ ) may be correction information. 8. The method of claim 7,
Wherein the correction information determination unit determines,
A factor included in the matrix L, which is a matrix of inverse of the matrix H including the correction information, is replaced with the correction information included in the equations (1), (2) and (3) A depth image correction device for determining relationship information. The method according to claim 6,
Wherein the depth image correction unit comprises:
And a point on the three-dimensional image of the depth image is shifted to a camera coordinate system based on the correction information. Determining a three-dimensional point on the camera image and a three-dimensional point on the depth image captured simultaneously with the camera image;
Determining a correction information for correcting an error of the depth image capturing the depth image and a geometric geometry relationship between the depth sensor and the photographing camera using a point on the three-dimensional image of the camera image and a point on the three- step; And
Correcting the depth image based on the correction information and the three-dimensional point of the depth image,
And correcting the depth image. 11. The method of claim 10,
Wherein the step of determining a point on the three-
Extracts a depth image characteristic point corresponding to a camera image characteristic point used for self-correcting the camera image in the depth image, and determines a three-dimensional point corresponding to the extracted depth image characteristic point as a three-dimensional point of the depth image A depth image correction method. 11. The method of claim 10,
Wherein the step of determining the correction information comprises:
Wherein the depth image correction unit generates the mathematical equations (1), (2), and (3) using the three-dimensional point of the camera image and the three-dimensional point of the depth image.
[Equation 1]
X = (X '* H ₁₁ + Y' * H ₁₂ + Z '* H ₁₃ + H ₁₄ ) / (X '* H ₄₁ + Y' * H ₄₂ + Z '* H ₄₃ + H ₄₄ )
&Quot; (2) "
Y = (X '* H ₂₁ + Y' * H ₂₂ + Z '* H ₂₃ + H ₂₄ ) / (X '* H ₄₁ + Y' * H ₄₂ + Z '* H ₄₃ + H ₄₄ )
&Quot; (3) "
Z = (X '* H ₃₁ + Y' * H ₃₂ + Z '* H ₃₃ + H ₃₄ ) / (X '* H ₄₁ + Y' * H ₄₂ + Z '* H ₄₃ + H ₄₄ )
X ', Y', and Z 'are coordinates of a point on the three-dimensional image of the depth image, and (H ₁₁ H ₁₂ H ₁₃ H ₁₄ H ₂₁ H ₂₂ H ₂₃ H ₂₄ H ₃₁ H ₃₂ H ₃₃ H ₃₄ H ₄₁ H ₄₂ H ₄₃ H ₄₄ ) may be correction information. 11. The method of claim 10,
Wherein the step of determining the correction information comprises:
Wherein the correction information is determined by using Equation (4) using Equation (4) using a point on the 3D image of the camera image and a point on the 3D image of the depth image.
&Quot; (4) "
Ah = 0
(Where A is a 3m * 16 matrix composed of three-dimensional points on the camera image and three-dimensional points on the depth image, h is correction information, m is a camera image, or 3 It may be the number of points on the dimension.) 11. The method of claim 10,
Wherein the step of correcting the depth image comprises:
Defining a cost function having the correction information as an initial value, and performing a nonlinear optimization process for minimizing a defined cost function to correct the depth image. 11. The method of claim 10,
The step of correcting the depth comprises:
And a point on the three-dimensional image of the depth image is shifted to a camera coordinate system based on the correction information. An imaging camera capturing the camera image using an equation of a pattern surface restored by a photographing camera and an equation of a pattern surface restored by the depth sensor, an error of a depth sensor that captures the depth image, an error between a depth sensor and a photographing camera Determining correction information for correcting the geometric relationship of the first and second regions; And
Correcting the depth image based on the correction information and the three-dimensional point of the depth image,
And correcting the depth image. 17. The method of claim 16,
Wherein the step of determining the correction information comprises:
When the equation of the pattern surface restored by the photographing camera is Equation 5 and the equation of the pattern surface restored by the depth sensor is Equation 6, Equation 7, Equation 8, and Equation 7 are calculated based on Equation 5 and Equation 6, A depth image correction method for generating Equation (9).
&Quot; (5) "
aX + bY + cZ + 1 = 0
&Quot; (6) "
a'X + b'Y + c'Z + 1 = 0
&Quot; (7) "
a = (a '* H ₁₁ + b' * H ₁₂ + c '* H ₁₃ + H ₁₄ ) / (a '* H ₄₁ + b' * H ₄₂ + c '* H ₄₃ + H ₄₄ )
&Quot; (8) "
b = (a '* H ₂₁ + b' * H ₂₂ + c '* H ₂₃ + H ₂₄ ) / (a '* H ₄₁ + b' * H ₄₂ + c '* H ₄₃ + H ₄₄ )
&Quot; (9) "
c = (a '* H ₃₁ + b' * H ₃₂ + c '* H ₃₃ + H ₃₄ ) / (a '* H ₄₁ + b' * H ₄₂ + c '* H ₄₃ + H ₄₄ )
(Here, (H ₁₁ H ₁₂ H ₁₃ H ₁₄ H ₂₁ H ₂₂ H ₂₃ H ₂₄ H ₃₁ H ₃₂ H ₃₃ H ₃₄ H ₄₁ H ₄₂ H ₄₃ H ₄₄ ) may be correction information. 18. The method of claim 17,
Wherein the step of determining the correction information comprises:
A factor included in the matrix L, which is a matrix of inverse of the matrix H including the correction information, is replaced with the correction information included in the equations (1), (2) and (3) A depth image correction method for determining relationship information.

Images