KR101845253B1 - Apparatus and method for estimating motion using depth image - Google Patents

Abstract

An apparatus and method for motion estimation using input depth images are disclosed. The motion estimation apparatus includes a pose parameter determination unit for determining an initial pose parameter corresponding to an input depth image and a pose generation unit for applying the pose parameter to an operation model to generate a first result pose of the operation model, Can generate a second result pose of the motion model based on the difference between the first result pose and the input depth image, so that the motion of the object can be estimated more quickly and accurately.

Description

[0001] APPARATUS AND METHOD FOR ESTIMATING MOTION USING DEPTH IMAGE [0002]

Embodiments of the present invention relate to motion estimation apparatus and method for estimating motion using a depth image.

Recently, the motion of a target object (such as a human body) in a real space, such as a video game, a virtual world, a cinematographic computer graphic (CG) There is an increasing interest in technology to implement in space.

However, this technique, which can be called various names such as motion estimation, motion capture, etc., has a disadvantage in that the computation time from the detection of the motion to the derivation of the motion estimation result is long.

In the case of estimating the motion of a person, the result of motion estimation is expected to be more accurate as the number of cameras, markers, sensors, and the like increases. However, in the case of motion estimation using a camera, In the case of the motion estimation method using a marker, a sensor, etc., there is a problem that motion of the human body is limited due to the number and position of sensors that can be attached to a human body, and it is difficult to estimate the motion of the whole human body. As the number of cameras, markers, sensors, and the like increases, precise dynamic estimation can be performed. However, since the number of parameters to be calculated increases, the time required for motion estimation further increases.

Therefore, until now, motion estimation techniques based on camera or motion sensor input have been limited in estimating complex motion as the entire human body.

Therefore, there is a demand for a technique capable of accurately estimating complex operations.

There is provided an apparatus and method for motion estimation capable of precisely estimating motion in environments such as complicated lighting, background, and the like.

There is provided an apparatus and method for estimating motion with high calculation speed even when accurate motion is estimated.

An operation estimating apparatus using an input depth image includes a pose parameter determining unit for determining an initial pose parameter corresponding to an input depth image and a pause generating unit for generating a first result pose of the operation model by applying the pose parameter to the operation model And the pause generator may generate a second result pose of the motion model based on a difference between the first result pose and the input depth image.

According to one aspect, the pose parameter determination unit detects a first pose parameter corresponding to the input depth image among a plurality of pose parameters stored in a database, using a Locality Sensitive Hash (pseudo Hash) based pause detection technique, Parameter to the initial pose parameter.

According to another aspect, the pose parameter determination unit may calculate a second pose parameter that matches the operation model with the input depth image, and determine the second pose parameter as the initial pose parameter.

According to another aspect of the present invention, the pause generation unit generates a first depth image using the first result pose of the operation model, and generates a first depth image based on a difference between the first depth image and the input depth image, 2 result pose can be generated.

According to another aspect of the present invention, the pause generation unit may calculate a depth value corresponding to the pixel, a configuration index of the operation model, and a coordinate value for each pixel of the first depth image, A difference in depth value of a pixel in the corresponding input depth image may be calculated to generate a second result pose of the motion model.

According to another aspect, the pause generator may generate a second result pose of the motion model according to a result analyzed through principal component analysis on the database.

According to another aspect of the present invention, when the pixel is located in an overlapping area between the input depth image and the first depth image, the pose generating unit may calculate the depth value of the calculated pixel and the pixel value of the pixel Calculating a difference of depth values to generate a second result pose of the motion model; and if the pixel is located in a non-overlapping region of the input depth image and the first depth image, using an IPC (Iterative Closest Point) To generate a second result pose of the motion model by matching the depth value of the calculated pixel with the depth value of the pixel in the corresponding input depth image.

According to another aspect of the present invention, the method may further include a pose determining unit that determines the second result pose as a result pose corresponding to the input image when the difference between the second result pose and the input depth image is less than or equal to a predetermined threshold value .

According to another aspect, the pause generation unit may generate the second result pose using at least one of input color images, silhouettes, edges, and physical constraints of the motion model at the time of generating the second result pose.

Meanwhile, a method of estimating an operation using an input depth image received by an operation estimating apparatus includes a pose parameter determining step of determining an initial pose parameter corresponding to the input depth image, a pose parameter determining step of applying the pose parameter to an operation model, And a second result pose generation step of generating a second result pose of the motion model based on a difference between the first result pose and the input depth image .

By estimating the motion using the depth image, it is possible to accurately estimate the motion even in an environment of complicated lighting, background, and the like.

The motion can be estimated using the color image, the silhouette, the edge, and the physical constraints of the human body, so that the motion can be estimated more accurately.

Complex operations can be estimated in real time by reducing the number of parameters to be calculated in motion estimation through principal component analysis.

1 is a block diagram illustrating an operation estimating apparatus using an input depth image according to an embodiment of the present invention.
FIG. 2A is an exemplary view of an input depth image in an embodiment of the present invention, FIG. 2B is an exemplary view of an operation model in an embodiment of the present invention, and FIG. FIG. 8 is a diagram showing an example of a result pose generated using an image. FIG.
FIG. 3 is a diagram for explaining a process of calculating a difference between an input depth image and a first depth image, according to an embodiment of the present invention.
4 is a diagram for explaining a process of estimating an operation of an operation estimating apparatus according to an embodiment of the present invention.
5 is a flowchart illustrating an operation estimation method in an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an operation estimating apparatus using an input depth image according to an embodiment of the present invention. Referring to the drawings, the motion estimation apparatus 100 includes a pose parameter determination unit 120 and a pause generation unit 140.

When a depth image is input to the motion estimation apparatus 100 from a depth camera or the like, the pose parameter determination unit 120 determines an initial pose parameter corresponding to the input depth image 110.

The pose generation unit 140 applies the pose parameter determined by the pose parameter determination unit 120 to the operation model to generate a first result pose of the operation model. And the pose generator 140 generates a second result pose of the motion model based on the difference between the first result pose and the input depth image 110. [

At this time, the pose parameter determination unit 120 uses the Locality Sensitive Hash (LSH) -based pose detection technique to more precisely estimate the motion of the input depth image 110 among a plurality of pose parameters stored in the database 130, To determine the first pose parameter as the initial pose parameter.

Meanwhile, the pose parameter determination unit 120 may calculate a second pose parameter that matches the operation model with the input depth image, and determine the second pose parameter as the initial pose parameter.

More specifically, the database 130 stores pose parameters calculated by capturing various operations of an object such as a human body using a plurality of precise sensors, markers, cameras, and the like. Using these pose parameters, you can create almost any pose the body can produce.

For example, the pose parameter determination unit 120 extracts features from the input depth image 110 through an adaptive boost (Adaboost) technique, and then uses the LSH technique to calculate the parameters of the pixels corresponding to the feature in the database 130 And detects the closest plurality of parameters. An appropriate match is determined using an analysis-by-synthesis technique. The determined match can be used in a motion model as an initial pose parameter or in a model for restoring motion estimation failure.

The pose parameter determination unit 120 periodically performs such a process to thereby generate an initial result pose by causing the pose generation unit 140 to reflect the pose parameter when generating a result pose, So that accurate operation can be detected again even if an error occurs.

Hereinafter, a process of generating a result pose will be described in detail with reference to FIGS. 1 and 2. FIG. FIG. 2A is an exemplary view of an input depth image in an embodiment of the present invention, FIG. 2B is an exemplary view of an operation model in an embodiment of the present invention, and FIG. FIG. 8 is a diagram showing an example of a result pose generated using an image. FIG.

The pose generation unit 140 applies the pose parameter determined by the pose parameter determination unit 120 to the operation model to generate a first result pose of the operation model and outputs the first result pose to the first depth image . A second result pose of the motion model is then generated based on the difference between the first depth image and the input depth image.

The input depth image 110 is an image captured by a depth camera and having only a depth value as shown in FIG. 2A. In the drawings, the depth values of the human body are displayed constantly for convenience, but the actual depth values of the human body are differently displayed in the three-dimensional space according to the distance from the depth camera. Since the depth image has only the depth value according to the distance from the depth camera, the object such as the human body can be accurately extracted even in a change of surrounding environment such as illumination, background, and the like.

Therefore, the motion estimation apparatus 100 according to the present invention estimates the motion of an object such as a human body included in the input depth image using the input depth image 110, so that it can accurately detect the pose of the object, .

Meanwhile, the operation model is a model capable of generating an operation using a plurality of pose parameters stored in the database 130 as shown in FIG. 2B. Although the operation model is shown as a 3D model in the drawing, the operation model is not limited to a 3D model. Instead, the operation model is implemented as a 2D model having a depth value, and the pose generation unit 140 uses the input depth image 110 May be estimated based on the motion of the object.

The first result pose may be determined by applying an initial pose parameter determined by the pose parameter determination unit 120 to the motion model as shown in FIG. 2C so that the motion model is in a pose similar to the pose in the input depth image .

That is, the pose generation unit 140 applies the pose parameter determined by the pose parameter determination unit 120 to the operation model (FIG. 2B) to generate a pose of the operation model similar to the pose of the input depth image 110 To produce a first result pose (Figure 2c).

However, there is a certain difference between the first result pose and the pose of the input depth image, since the first result pose thus generated is a pose generated only by the measured values for the input depth image. Accordingly, the pose generator 140 generates the first depth image by rendering the first result pose reflecting the pose of the input depth image, such that the first result pose is in the same pose as the pose of the input depth image, Generating a second result pose of the motion model consistent with a pose of the input depth image based on a difference between the first depth image and the input depth image.

Specifically, the pose generator 140 calculates a depth value corresponding to the pixel, a composition index of the motion model, and a coordinate value for each pixel of the first depth image, A second result pose of the motion model may be generated by calculating a difference between a depth value and a depth value of a pixel in the input depth image 110 corresponding thereto.

Meanwhile, the pause generation unit 140 generates the pause generation unit 140 based on the analyzed result through Principal Components Analysis (PCA) on the database 130 so that the motion estimation apparatus 100 can estimate the operation in real time. A second result pose of the model can be generated.

For example, assuming that there are 17 parameters for determining the motion of the human body, it is assumed that 17x3, that is, 51 degrees of freedom (DOF: Degree of Freedom) is calculated by the three-dimensional spatial coordinates (x, . When the parameter is directly applied to the motion estimation, the amount of calculation is increased according to the number of parameters stored in the database, which is directly related to the calculation speed at the time of motion estimation.

therefore. In the present invention, by prioritizing the parameters to be calculated at the time of motion estimation through principal component analysis, it is possible to reduce the number of parameters to be calculated, so that complex operations can be estimated in real time.

In addition, the motion estimation apparatus 100 can reduce the number of parameters that must be calculated using the degree of association of object motion, such as physical constraints (e.g., it can not be bent backward in the case of an arm of the human body) It is possible to estimate the operation.

When the pixel is located in the overlapping region of the input depth image and the first depth image, the pose generating unit 140 generates a pose of the pixel in the input depth image corresponding to the depth value of the calculated pixel Calculating a difference of depth values to generate a second result pose of the motion model; and if the pixel is located in a non-overlapping region of the input depth image and the first depth image, using an IPC (Iterative Closest Point) To generate a second result pose of the motion model by matching the depth value of the calculated pixel with the depth value of the pixel in the corresponding input depth image.

This will be described in detail with reference to FIG. FIG. 3 is a diagram for explaining a process of calculating a difference between an input depth image and a first depth image, according to an embodiment of the present invention.

For example, an arm portion of a human body pose included in the input depth image 110 is referred to as a first region 310, and an arm portion in the corresponding first depth image is referred to as a second region 330. In this case, when a pixel to be calculated is located in the region 320 where the first region 310 and the region 320 overlap, the pixel of the input depth image 110 corresponding to the pixel exists, Value and the depth value of the pixel in the corresponding input depth image 110 and calculating the difference to generate a second result pose of the motion model.

However, when the pixel is located in the areas 315 and 335 where the first area 310 and the second area 330 do not overlap, there is no pixel to be compared with the pixel to be calculated. Therefore, the IPC (Iterative Closest Point) The depth value of the moved pixel is matched with the depth value of the corresponding pixel by moving the coordinate value of the pixel to be calculated in the direction of the overlapping area 320 and calculates the difference value thereof to calculate the second result You can create a pose.

The pose generation unit 140 may repeat the process until the second result pose generated through the above process coincides with the pose of the input depth image 110, thereby enhancing the performance of the motion estimation apparatus 100.

Although not shown in FIG. 1, the motion estimation apparatus 100 further includes a pause determination unit, and when the difference between the second result pose and the input depth image is equal to or less than a preset threshold value using the pause determination unit, As a result pose 150 corresponding to the input image. In this case, the pose determining unit is not included in the operation estimating apparatus 100 but is included in the pose generating unit 140, so that the pause generating unit 140 may calculate the difference between the second result pose and the input depth image The second result pose may be determined as a result pose 150 corresponding to the input image.

As described above, the motion estimation apparatus 100 generates a result pose 150 for each input depth image 110 input, so that it can quickly and accurately estimate the motion of the human body and the like.

The pose generator 140 may generate the second result pose by additionally using the input depth image, the input color image, the silhouette, the edge, and the physical constraints of the operation model at the time of generating the second result pose, The accuracy of motion estimation can be further improved.

Hereinafter, a process of estimating an operation of the motion estimation apparatus 100 according to an embodiment of the present invention will be described with reference to FIG.

When a pixel to be calculated in the three-dimensional spatial coordinates (x, y, z) moves from the first position 420 to the second position 430, the projection plane of the two- For example, the position of the pixel corresponding to the first position 430 in the first depth image is moved from the third position 440 to the fourth position 450.

D)가 발생한다.As the pixel moves from the first position 420 to the second position 430, the distance from the depth camera 410 becomes farther away,

D) occurs.

p)이 충분히 작다고 가정할 때, 다음과 같은 수학식이 성립한다.At this time, the motion of the pixel

p is sufficiently small, the following equation is established.

Here, t is time and p is the coordinates of the pixel.

When the Taylor series expansion method is applied to the right side of Equation (1), the following equation is established.

Here, the right side of Equation (2) can be calculated by a difference in depth value between the pixel of the input depth image 110 and the pixel of the first depth image.

p)은 키네마틱스 함수(kinematics function)를 이용하여 수학식 3과 같이 표현할 수 있다.In order to use Equation (2) to generate a pose of a human body or the like,

p) can be expressed as Equation (3) using a kinematics function.

Here, h is a forward kinematics function, p0 is a coordinate value of each constituent index, and k is a constituent index number.

By integrating equations (2) and (3) using chain rules, the following equations are established.

Accordingly, the pose generation unit 140 can generate the second depth image by calculating the depth difference difference between the pixels of the first depth image and the input depth image using Equation (4).

5 is a flowchart illustrating an operation estimation method in an embodiment of the present invention.

When the motion estimation apparatus 100 receives the input depth image (S510), the pose parameter determination unit 120 determines an initial parameter corresponding to the pose included in the input depth image 110 (S520).

The pose generation unit 140 generates a first result pose similar to the input depth image 110 by applying the initial parameters received by the pose parameter determination unit 120 to the motion model (S530).

Thereafter, the pose generator 140 generates a first depth image by rendering an operation model reflecting the first result pose (S540), and generates a first depth image based on the difference between the first depth image and the depth value of the input depth image A second result pose corresponding to the depth image 110 may be generated (S550).

The pose generation unit 140 or the pose determination unit determines whether the second result pose matches the input depth image (S560). If the difference between the generated second result pose and the input depth image 110 is less than or equal to a preset threshold value (S560), the second result pose is determined to coincide with the input depth image and the second result pose is determined as a result pose . However, if the difference between the second result pose and the input depth image 110 is equal to or greater than the threshold value, the depth difference between the second result pose and the input depth image is compared with the input depth image 110, The operation is estimated by repeating the process of generating the result pose.

The motion estimation apparatus according to an embodiment of the present invention can estimate a motion by generating a result pose for each depth image input through the process described above.

The method according to an embodiment of the present invention can be implemented in the form of a program command which can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

100: motion estimation device
110: input depth image
120: pose parameter determination unit
130: Database
140:
150: Results pose

Claims (19)

A pose parameter determination unit that determines an initial pose parameter corresponding to an input depth image; And
Generating a first result pose of the motion model by applying the pose parameter to the motion model;
Lt; / RTI >
Wherein the pause generator generates a second result pose of the motion model based on a difference between the first result pose and the input depth image. The method according to claim 1,
Wherein the pose parameter determination unit determines,
A first pose parameter matching the input depth image is detected among a plurality of pose parameters stored in a database by using a Locality Sensitive Hash (LSH) based pose detection technique, and the first pose parameter is determined as the initial pose parameter , Motion estimation device. The method according to claim 1,
Wherein the pose parameter determination unit determines,
Calculates a second pose parameter that matches the motion model to the input depth image, and determines the second pose parameter as the initial pose parameter. The method according to claim 1,
Wherein the pause generating unit comprises:
Generate a first depth image using the first result pose of the motion model and generate a second result pose of the motion model based on a difference between the first depth image and the input depth image, . 5. The method of claim 4,
Wherein the pause generating unit comprises:
Calculating a depth value corresponding to the pixel, a composition index of the operation model, and a coordinate value for each pixel of the first depth image, calculating a depth value of the calculated pixel and a pixel value of the pixel in the input depth image corresponding thereto And calculates a difference in depth value to generate a second result pose of the motion model. 6. The method of claim 5,
Wherein the pause generating unit comprises:
And generates a second result pose of the motion model according to the analyzed result through principal component analysis on the database. 6. The method of claim 5,
Wherein the pause generating unit comprises:
Calculating a difference between a depth value of the calculated pixel and a depth value of a pixel in the input depth image corresponding to the depth value of the calculated pixel when the pixel is located in an overlapping area of the input depth image and the first depth image, To produce a second result pose of < RTI ID = 0.0 >
When the pixel is located in a non-overlapping area between the input depth image and the first depth image, a depth value of the calculated pixel is calculated using an IPC (Iterative Closest Point) To generate a second result pose of the motion model. The method according to claim 1,
And a pose determining unit that determines the second result pose as a result pose corresponding to the input depth image, when the difference between the second result pose and the input depth image is less than or equal to a preset threshold value. The method according to claim 1,
Wherein the pause generating unit comprises:
And generating the second result pose using at least one of an input color image, a silhouette, an edge, and a physical constraint of the motion model at the time of generating the second result pose. A method for estimating an operation using an input depth image received by an operation estimating apparatus,
A pose parameter determining step of determining an initial pose parameter corresponding to the input depth image;
Applying a pose parameter to the action model to generate a first result pose of the action model; And
A second result pose generation step of generating a second result pose of the motion model based on a difference between the first result pose and the input depth image
/ RTI > 11. The method of claim 10,
Wherein the pose parameter determination step comprises:
Detecting a first pose parameter corresponding to the input depth image among a plurality of pose parameters stored in a database using a locality sensitive hash (LSH) based pose detection technique; And
Determining the first pose parameter as the initial pose parameter
/ RTI > 11. The method of claim 10,
Wherein the pose parameter determination step comprises:
Calculating a second pose parameter that matches the motion model to the input depth image; And
Determining the second pose parameter as the initial pose parameter
/ RTI > 11. The method of claim 10,
Wherein the second result pose generation step comprises:
Generating a first depth image using the first result pose of the motion model; And
Generating a second result pose of the motion model based on a difference between the first depth image and the input depth image
/ RTI > 14. The method of claim 13,
Wherein generating a second result pose of the action model comprises:
Calculating a depth value corresponding to the pixel, a configuration index of the operation model, and a coordinate value for each pixel of the first depth image; And
Calculating a difference between a depth value of the computed pixel and a depth value of a pixel in the input depth image corresponding thereto to generate a second result pose of the operation model
/ RTI > 15. The method of claim 14,
Wherein generating a second result pose of the action model comprises:
And generating a second result pose of the motion model according to a result analyzed through principal component analysis on the database. 15. The method of claim 14,
Wherein generating a second result pose of the action model comprises:
Calculating a difference between the calculated depth value and a depth value of a pixel in the input depth image corresponding to the input depth image if the pixel is located in an overlapping area of the input depth image and the first depth image, 2 result pose,
When the pixel is located in a non-overlapping area between the input depth image and the first depth image, the calculated depth value and the depth of the pixel in the input depth image corresponding to the calculated depth value using the IPC (Iterative Closest Point) Value to generate a second result pose of the motion model. 11. The method of claim 10,
After the second result pose generation step,
Further comprising determining the second result pose as a result pose corresponding to the input depth image if the difference between the second result pose and the input depth image is less than or equal to a predetermined threshold. 11. The method of claim 10,
Wherein the second result pose generation step comprises:
And generating the second result pose using at least one of an input color image, a silhouette, an edge, and a physical constraint of the motion model at the time of generating the second result pose. A computer-readable recording medium containing a program for performing the motion estimation method according to any one of claims 10 to 18.

Images