KR20140056992A - Method of tracking motion using depth image and device thereof - Google Patents

Abstract

Disclosed is a motion tracking method using a depth image. According to an embodiment, the present invention includes a step of obtaining continuous depth images of at least two frames; a step of calculating first parameters related to a depth flow between adjacent frames based on the depth images; a step of obtaining second parameters related to motion priors; and a step of estimating a pose change rate between the at least two continuous frames based on the first parameters and the second parameters.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a motion estimation method and an apparatus using depth images,

The following embodiments relate to motion estimation methods and apparatus using depth images. In particular, the following embodiments are directed to a method and apparatus for estimating a change in pose based on parameters associated with depth flows and parameters associated with motion fliers.

Recently, the motion of a target object (such as a human body) in a real space, such as a video game, a virtual world, a cinematographic computer graphic (CG) There is an increasing interest in technology to implement in space.

The depth image is an image that stores the depth from the depth image camera to the target object. In this case, the single depth image may include depth images in which the target object is imaged in a single direction by a single depth camera.

An operation estimation method using a depth image according to one side includes obtaining depth images of at least two consecutive frames; Calculating first parameters related to a depth flow between adjacent frames based on the depth images; Motion priors - acquiring second parameters associated with each of the motion fliers comprising information corresponding to a predetermined pose; And estimating a change amount of a pause between the at least two consecutive frames based on the first parameters and the second parameters.

At this time, the amount of change in the pose is modeled by a variation amount of each of the plurality of joint angles, and each of the plurality of joint angles may correspond to any one of a plurality of joints included in the skeleton model of the target object.

In addition, estimating the amount of change in the pose may include determining an amount of change in the pose such that a sum of an absolute value associated with the first parameters and an absolute value associated with the second parameters is minimized.

The step of estimating the amount of change of the pose may include determining a change amount of the pose such that the amount of change of the pose is minimized while taking into consideration the constraint associated with the first parameters and the constraint associated with the second parameters . ≪ / RTI >

Also, the motion fliers may be stored in advance in the database, and the second parameters may be generated by applying Principal Components Analysis (PCA) techniques to the motion fliers.

In addition, the motion fliers may be stored in advance in the database, and the second parameters may be generated by applying a Mixture of Factor Analyzers (MFA) technique to the motion fliers.

The obtaining of the second parameters may further include extracting K neighboring pauses similar to the pose in the tth frame of the motion ficers based on a pose in the tth frame; And generating the second parameters by applying a Local Principal Components Analysis (LPCA) technique to the K neighboring pauses, wherein the t < th > frame is included in the at least two consecutive frames , The value of K may depend on a predetermined similarity determination threshold.

The step of extracting the K neighboring pauses may further comprise the steps of: calculating in parallel the similarity values between the pose in the t < th > frame and at least a portion of the motion fryers using a processor that parallelizes the plurality of threads; And selecting the K neighboring pauses based on the parallelly calculated similarity values and the predetermined similarity determination threshold.

The step of extracting the K neighboring pauses may include generating first quantization results and a second quantization result by applying a product quantization technique to each of the pauses in the motion fliers and the t frame, ; And calculating a similarity value between the motion fliers and a pose in the t-th frame based on the first quantization results and the second quantization result.

The motion estimating method according to the one aspect further includes obtaining a reference depth image of a target object that takes a predetermined reference pose; Classifying the plurality of pixels included in the reference depth image into one of a plurality of joints included in the skeleton model of the target object based on the information related to the predetermined reference pose; And determining a radius parameter of each of the plurality of joints such that a maximum number of depth pixels corresponding to each of the plurality of joints is maximum based on a random sampling technique.

According to another aspect of the present invention, there is provided a method for estimating an operation according to one side, comprising: recognizing a joint pair symmetric to each other among the plurality of joints based on a symmetric nature of the skeleton model; Calculating an average of a radius parameter of a first joint included in the joint pair and a radius parameter of a second joint included in the joint pair; And modifying each of the radius parameter of the first joint and the radius parameter of the second joint to the average.

A motion estimation apparatus using a depth image according to another aspect includes: a processor; And a depth image acquisition unit for acquiring depth images of at least two consecutive frames, wherein the processor calculates parameter calculations for calculating first parameters related to a depth flow between adjacent frames based on the depth images, part; Motion priors - a parameter acquiring unit for acquiring second parameters associated with each of the motion fliers, wherein each of the motion fiers includes information corresponding to a predetermined pose; And a pose change amount estimating unit that estimates a change amount of a pause between the at least two consecutive frames based on the first parameters and the second parameters.

At this time, the amount of change in the pose is modeled by a variation amount of each of the plurality of joint angles, and each of the plurality of joint angles may correspond to any one of a plurality of joints included in the skeleton model of the target object

The pose change amount estimating unit may determine the amount of change of the pose such that a sum of an absolute value associated with the first parameters and an absolute value associated with the second parameters is minimized.

The pose change amount estimating unit may determine the amount of change of the pose such that the change amount of the pose is minimized while considering the restriction condition related to the first parameters and the restriction condition related to the second parameters.

Also, the motion fliers may be stored in advance in the database, and the second parameters may be generated by applying Principal Components Analysis (PCA) techniques to the motion fliers.

In addition, the motion fliers may be stored in advance in the database, and the second parameters may be generated by applying a Mixture of Factor Analyzers (MFA) technique to the motion fliers.

The parameter obtaining unit may further include: a neighbor pose extracting unit for extracting K neighboring pose similar to the pose in the t-th frame of the motion ficers based on the pose in the tth frame; And a parameter generator for generating the second parameters by applying a local principal component analysis (LPCA) technique to the K neighboring pauses, wherein the t < th > frame is included in at least two consecutive frames , And the value of K may depend on a predetermined similarity determination threshold.

The neighbor pose extracting unit may include a parallel processing unit for calculating the similarity values between the pose of the t-th frame and at least a part of the motion fingers in parallel; And a neighbor pose selector for selecting the K neighbor pose based on the parallel calculated similarity values and the predetermined similarity determination threshold.

The neighboring pose extracting unit may include a product quantization unit that generates first quantization results and a second quantization result by applying a product quantization technique to the motion fliers and the pose in the t frame, respectively; And a similarity calculation unit for calculating a pose in the t-th frame and similarity values between the motion fliers based on the first quantization results and the second quantization result.

The depth image acquisition unit may acquire a reference depth image of a target object that takes a predetermined reference pose, and the processor may calculate a plurality of pixels included in the reference depth image based on the information associated with the predetermined reference pose, A classifier for classifying the object into any one of a plurality of joints included in the skeleton model of the object; And a radius parameter determination unit for determining a radius parameter of each of the plurality of joints so that the number of depth pixels corresponding to each of the plurality of joints becomes the maximum based on a random sampling technique.

The processor is further configured to recognize a symmetric symmetry pair of the plurality of joints based on the symmetric nature of the skeleton model and determine a radius parameter of the first joint included in the joint pair, And the radius parameter of the first joint and the radius parameter of the second joint may be modified to the average of the radius parameter of the first joint and the radius parameter of the second joint.

1 is a flowchart illustrating an operation estimation method using a depth image according to an exemplary embodiment;
2 is a diagram for explaining a depth flow function according to an embodiment;
3 is a view for explaining a human skeleton model according to an embodiment.
4 is a diagram for describing automatic skeleton calibration according to one embodiment;
5 is a block diagram illustrating an operation estimating apparatus using a depth image according to an embodiment.

Work In the embodiment  Depth image ( depth image ) ≪ / RTI >

An operation estimation method using a depth image according to an exemplary embodiment includes a step 110 of acquiring depth images, a step 120 of calculating first parameters, a step of acquiring second parameters 130, (Step 140).

At this time, acquiring depth images 110 may acquire depth images of at least two consecutive frames. Here, the depth image is an image storing information related to the distance to the target object, for example, an image storing the distance information from the two-dimensional projection plane (project plane) of the depth image photographing apparatus to the target object as each pixel value .

In addition, calculating 120 the first parameters may calculate first parameters related to a depth flow between adjacent frames based on the depth images. More specifically, the first parameters may include various partial differential parameters included in a depth flow equation. More details related to the first parameters will be described below with reference to FIG.

In addition, acquiring (130) the second parameters may acquire second parameters associated with motion priors. Here, each of the motion fliers may include information corresponding to a predetermined pose. For example, if the target object is a person, the pose that a person can take depending on the joint structure of a person can be limited. Thus, motion fliers may be limited to those that contain information corresponding to the poses that the target object may take.

In addition, estimating the amount of change in the pose 140 may estimate the amount of change in the pose between at least two consecutive frames based on the first and second parameters.

Here, the first parameters are parameters related to the depth flow between adjacent frames, and the second parameters are parameters related to motion fliers. Thus, step 140 of estimating the amount of change in the pose can estimate the amount of change in the pose that satisfies both the constraints associated with the depth flow and the constraints associated with the motion fliers.

For example, estimating a variation amount of a pose (140) may include expressing a constraint associated with the depth flow with an absolute value associated with the first parameters, expressing the constraint associated with the motion fliers with an absolute value associated with the second parameters . Estimating the amount of change in the pose 140 may determine the amount of change in the pose such that the sum of the absolute values associated with the first parameters and the absolute values associated with the second parameters is at a minimum.

Accordingly, even when there is a large amount of noise in the depth images, the motion estimation method according to an exemplary embodiment can estimate a variation amount of a pose in which a natural result pose is derived by satisfying a constraint related to motion fliers.

In addition, the motion estimation method according to an exemplary embodiment of the present invention includes a pause in which a natural result pose is derived by satisfying a constraint associated with motion fliers even when self-occluded pixels are included in depth images Can be estimated.

For example, suppose a case where the motion of moving both arms from the posture of crossing to the posture of posture is estimated. In this case, the part of the arm that is located on the back side of the crossed arms is inevitably covered.

Therefore, it is difficult to accurately estimate the motion of the dorsal arm on the back side when using only the constraints related to the depth flow. However, by further considering the constraints associated with motion fliers, it is possible to estimate the amount of change in pose that results in a natural result pose, even if the pixels that are hidden by the depth images are included.

In addition, the step 140 of estimating the amount of change in pose according to another embodiment may include calculating a variation amount of the pose such that the variation amount of the pose is minimized while taking into consideration the restriction condition related to the first parameters and the restriction condition related to the second parameters Can be determined.

Accordingly, the motion estimation method according to an exemplary embodiment can derive smoother motion.

Meanwhile, a pose according to one embodiment may be modeled as a plurality of joint angles. In this case, each of the plurality of joint angles may correspond to any one of a plurality of joints included in a skeleton model of the target object.

For example, referring to FIG. 2, if the target object according to one embodiment is a person, the skeleton model of the target object may include fifteen joints 210, 220, 230, 235, 240, 245, 250, 255, , 275, 280, 285, 290, 295). Each joint may correspond to a particular bone segment of the target object.

Each joint can be represented by a joint coordinate and a joint angle. The joint coordinates may include the position coordinates of the joint, which is the origin of the rotation of the bone segment corresponding to the joint, and the joint angle may include an angle vector in which the bone segment of the target object is rotated about the corresponding joint coordinate.

Thus, the pose according to one embodiment may be modeled by the joint angles of the joints included in the skeleton model. Each joint can be represented by one joint angle, and in some cases can be represented by a plurality of joint angles. For example, joints between the thigh bone and the shin bone can only rotate in one direction. At this time, the joints 290 and 295 corresponding to the shin bone can be modeled by one joint angle. On the other hand, the skull can be rotated in three directions (tilting the head to the left and right, tilting the head back and forth, and shaking the head to the left and right). Thus, the joint 210 corresponding to the skull can be modeled by three joint angles.

According to another embodiment, a pose according to an embodiment may be modeled by joint angles and joint coordinates of the joints included in the skeleton model. In this case, each joint may be modeled by one joint coordinate and at least one joint angle. For example, since the vertebrae become the center of the human body and the vertebrae can also be rotated in three directions, the joint 260 corresponding to the vertebrae will have a root position coordinate and three joint angles Lt; / RTI > In addition, since the shin bone is rotated with reference to the joint which is in contact with the thigh bone, the joints 290 and 295 corresponding to the shin bone can be modeled by the position coordinates of the knee joint and the angle of one joint.

Further, the amount of change in the pose according to one embodiment can be modeled by the amount of change of each of the plurality of joint angles. Of course, according to another embodiment, the amount of change in the pose can be modeled by the amount of change of each of the plurality of joint angles and the amount of change of each of the plurality of joint coordinates.

Hereinafter, the first parameters and the second parameters will be described in detail.

(1) Depth flow function

Hereinafter, a process of calculating the first parameters by the motion estimation method according to an embodiment will be described with reference to FIG.

When the pixel to be calculated in the three-dimensional spatial coordinates (x, y, z) moves from the first position 320 to the second position 330, the projection plane of the two- For example, the position of the pixel corresponding to the first position 330 at the first depth image is moved from the third position 340 to the fourth position 350.

As the pixel moves from the first position 320 to the second position 330, the distance from the depth camera 310 increases, so that a change in depth value DELTA D occurs in the pixel.

At this time, assuming that the motion (? Q) of the pixel is sufficiently small, the following equation holds.

Here, t is a frame, and q is a coordinate of a pixel.

When the Taylor series expansion method is applied to the right side of Equation (1), the following equation is established.

Here, the right side of Equation (2) can be calculated by a difference in depth value between the pixel of the input depth image 110 and the pixel of the first depth image.

In order to use Equation (2) to generate a pose of a human body or the like, the movement (? Q) of the pixel in the three-dimensional space can be expressed as Equation (3) using a kinematics function.

Where f is the forward kinematics function, q0 is the coordinate value of each configuration index, and k is the respective configuration index number.

By integrating equations (2) and (3) using chain rules, the following equations are established.

In each of the two adjacent frames, each of the depth pixels must satisfy Equation (4). Thus, a system of linear functions can be constructed using a set including a depth flow of a plurality of depth pixels. The system of such a linear function can be simplified as shown in Equation (5).

The amount of change ([Delta] p) of the pose between two adjacent frames can be derived by solving Equation (5). Therefore, the operation of the target object can be sequentially estimated by repeatedly solving Equation (5).

As described above, the motion estimation method according to an exemplary embodiment can estimate a variation amount of a pose that satisfies both constraints related to the depth flow and constraints associated with motion fliers. Hereinafter, three statistical models related to motion fliers are described.

(2-1) Principle Component Analysis (PCA)

The pose of the target object (for example, the pose of the human body) generally has a large degree of freedom. For example, the human skeleton model can be modeled by 38 joint angles. However, the pose of a particular human body can be highly correlated with other degrees of freedom. Here, the other degrees of freedom may include a degree of freedom that is less than the greater degrees of freedom of a particular human body pose. In conclusion, Principal Component Analysis is a linear statistical model for modeling these high associations.

Mathematically, principal component analysis models a high dimensional pose (e. G., A pose of the human body with a large degree of freedom) as shown in equation (6).

는 평균 포즈이며, C는 저차원 공간으로부터 고차원 공간으로의 변환(transformation) 매트릭스이다. 이 때, p_low는 제로-평균 가우스 분포된 N(0;1)으로 가정될 수 있다. Where p _low is a low dimensional vector (e.g., a vector with a lower degree of freedom relative to the pose of the human body)

Is the average pose, and C is the transformation matrix from low dimensional space to high dimensional space. At this time, p _low can be assumed to be a zero-mean gaussian distributed N (0; 1).

Also, p _high must be converted to p _low and the existing values should not change even after being restored to p _high again.

을 대입하면, 수학식 8이 도출된다.In Equation 7,

, Equation (8) is derived.

Equation (8) is a linear function related to the amount of change (? P) of the pose, and can be simplified as shown in Equation (9).

On the other hand, as another constraint condition, p _low is assumed to be a zero-mean gaussian distributed N (0; 1), the equation (11) can be derived from the equation (10).

Equation (11) is also a linear function related to the change amount (p) of the pose, and can be simplified as shown in Equation (12).

In addition, the motion estimation method according to an exemplary embodiment may determine a variation amount of a pose such that a variation amount of the pose is minimized while taking into consideration the constraints related to the first parameters and the constraints related to the second parameters for smoothness of operation have. The constraint condition that minimizes the amount of change of the pose can be considered by Equation (13).

Consequently, the motion estimation method can estimate the amount of change? P of the pose based on Equations (5), (9), (12), and (13). Equation (5), Equation (9), Equation (12) and Equation (13) are summarized as Equation (14).

In this case, equation (14) may correspond to a linear system that is larger than the linear system corresponding to each of equations (5), (9) and (12). The operation estimating method according to an embodiment can estimate the amount of change? P of the pose by solving Equation (14) using Equation (15).

Here, a ₁ , a ₂ , a ₃ , a ₄ are coefficients that can be adjusted by the motion estimation method.

The motion estimation method is based on a pause (p) in a new frame (e.g., the (t + 1) frame) based on a pose (p _t ) in an existing frame p _{t + 1} ). For example, the process of generating a pose in a new frame can be simply expressed by Equation (16).

(2-2) Mixture of Factor Analyzer (MFA)

The Factor Analyzer (FA) analyzes the high-dimensional real-valued data vector p∈R ^D (for example, the pose of the human body) with a low-dimensional vector s∈R ^d and a multivariate Gaussian Lt; RTI ^{ID = 0.0} > u < / RTI >

Mathematically, the factor analyzer models the high dimensional real value data vector as shown in equation (17).

은 요인 로딩 매트릭스(factor loading matrix)이고, 저차원 벡터 s는 제로-평균 가우스 분포된 N(0;1)로 가정될 수 있다. 또한, 랜덤 벡터 u는 다변수 가우스 분포된

로 가정될 수 있다. 이 때,

는 평균 벡터이고

는 대각 공분산 매트릭스(diagonal covariance matrix)이다.here,

Is a factor loading matrix, and the low dimensional vector s can be assumed to be a zero-mean gaussian distributed N (0; 1). Also, the random vector u is a multivariate Gaussian distributed

. ≪ / RTI > At this time,

Is the mean vector

Is a diagonal covariance matrix.

The poses of most human bodies are hardly represented by a system of linear functions. Instead, these human body poses can be easily represented by a nonlinear system.

Factor analyzer mixing can mix factor analyzers to model the distribution of nonlinear pose. More specifically, the factor analyzer mixer may partition the high dimensional real value data vector p < RTI ID = 0.0 > R ^D < / RTI > into a plurality of local spaces probabilistically. Further, the factor analyzer mixing can model each of the plurality of partitioned local spaces using a different factor analyzer.

Mathematically, the factor analyzer mix can be assumed to be a mixture of normal distributions for input data p.

Where K is the number of factor analyzers, the vector μ _k is the mean vector of the kth factor analyzer, and the matrix L _k is the factor loading matrix of the kth factor analyzer. ψ∈R ^DxD is a diagonal covariance matrix for an independent noise u.

At this time, the motion estimation method according to the embodiment can estimate the amount of change? P of the pose based on Equations (5), (18), and (13). More specifically, the motion estimation method can estimate the amount of change? P of the pose by solving Equation (19).

Here, b ₁ , b ₂ , and b ₃ are coefficients that can be adjusted by the motion estimation method. The motion estimation method according to an exemplary embodiment can solve Equation (15) using the Levenberg-Marquart algorithm.

(2-3) Local Principle Component Analysis (LPCA)

Previously, we have mentioned that most human poses are difficult to represent as a system of linear functions and can instead be easily represented by nonlinear systems. Local principal component analysis can also model the distribution of nonlinear pose.

More specifically, local principal component analysis can generate a local principal component analysis model in each frame instead of generating a global principal component analysis model.

For example, suppose that a local principal component analysis model is generated in the t-th frame. The motion estimation method may extract K neighbor pose similar to the pose in the t frame from the motion fliers stored in the database. Furthermore, the motion estimation method may generate a local principal component analysis model in the t-th frame based on K neighbor poses in real time.

와 제t 프레임에서의 변환 매트릭스

를 이용할 수 있다. 나아가, 지역 주성분 분석은 수학식 14와 유사한 선형 함수의 시스템을 풀이함으로써 포즈의 변화량(△p)을 추정할 수 있다. Mathematically, the local principal component analysis model is the average pose

And the transformation matrix in the t < th >

Can be used. Further, the local principal component analysis can estimate the change amount (p) of the pose by solving a system of linear functions similar to the equation (14).

The local principal component analysis model in a specific frame is a linear function, but local principal component analysis models in different frames use different local principal component analysis models, so local principal component analysis can model the distribution of nonlinear pose.

In this case, local principal component analysis should be calculated in real time, unlike principal component analysis. For example, the regional principal component analysis detects K neighbor poses based on the pose in the tth frame and generates a local principal component analysis model in the tth frame based on the detected K neighbor poses. Therefore, the local principal component analysis can only be performed by knowing the pose in the t-th frame, and the local principal component analysis model should be generated in real time.

The motion estimation method according to an exemplary embodiment may utilize a GPU acceleration based parallel processing technique. Thus, the motion estimation method can shorten the computation time required for modeling using local principal component analysis.

For example, the motion estimation method may calculate a similarity value between a pose in a t-th frame and a plurality of motion fliers stored in a database in parallel. At this time, the motion estimation method may use a GPU processor capable of processing a plurality of threads in parallel.

According to one embodiment, the similarity value between a pause in the t < th > frame and a plurality of motion fiers stored in the database may be expressed as: < EMI ID = 20.0 >

Here, N is the number of joint included in one pose and (if a skeleton model of a human body is composed of 15 joints, N is that number of 15 days), q ₀ is posing in the t frame, q _n Is the nth joint of the jth motion flier stored in the database. Also, d (q ₀ , q _n ) is the distance between the end point of the nth joint included in the pose in the tth frame and the end point of the nth joint of the jth motion primary stored in the database to be. Therefore, the similarity value between the pose in the t-th frame and the j-th motion leader stored in the database, D _j can be expressed as a value obtained by averaging the distance between the respective joints.

Further, the motion estimation method may select K neighboring poses by comparing the similarity values calculated in parallel to the similarity determination threshold. For example, the motion estimation method may select motion fryers whose computed similarity values are less than a preset similarity determination threshold as neighboring poses.

Here, the similarity determination threshold is a value that can be set by the motion estimation method, and the value of K, which is the number of neighboring pauses selected according to the similarity determination threshold, can be changed.

The output of the GPU acceleration based parallel processing technique may include the similarity value of each of the selected K neighboring pauses and the index of the motion fryer corresponding to the neighboring pose. At this time, these results are stored in a column-major format, so that the performance of the GPU acceleration-based parallel processing technique can be utilized to the maximum.

The motion estimation method according to another embodiment may utilize a product quantization technique.

More specifically, the motion estimation method can generate the first quantization results by applying a product quantization technique to a plurality of motion fliers stored in a database. The motion estimation method can generate the second quantization result by applying the product quantization technique to the pose in the t < th > frame.

The motion estimation method may calculate the similarity values between the pauses in the t-th frame and the motion fliers based on the first quantization results and the second quantization result. At this time, the motion estimation method may select K neighboring poses by comparing the calculated similarity values to the similarity determination threshold. For example, the motion estimation method may select motion fryers whose computed similarity values are less than a preset similarity determination threshold as neighboring poses.

Through such a product quantization technique, the motion estimation method can shorten the computation time required for modeling using local principal component analysis.

(3) Automatic skeleton calibration

The motion estimation method according to another embodiment may perform automatic skeleton correction. The motion estimation method can more accurately estimate the motions of target objects having different skeletal sizes (for example, human bodies having different skeletal sizes) by performing automatic skeleton correction.

More specifically, the motion estimation method can automatically predict the length and radius of each of a plurality of joints included in the skeleton model of the target object. At this time, the skeleton model of the target object can model the remaining bone segments except the body including the spine as a cylinder. In the case of the body including the vertebra, it can be modeled as an elliptical column.

The motion estimation method can perform automatic skeleton correction only once when the target object is changed. Hereinafter, the process of performing automatic skeleton correction will be described in detail.

First, referring to FIG. 4A, the motion estimation method may obtain a reference depth image of a target object that takes a predetermined reference pose. Here, the reference pose can be predetermined so as to easily distinguish each of the plurality of joints included in the skeleton model.

The motion estimation method may classify the plurality of pixels included in the reference depth image into any one of a plurality of joints included in the skeleton model of the target object based on information related to a predetermined reference pose.

For example, the motion estimation method can easily distinguish the pixels 410 of the skull portion from the pixels of the clavicle bone portion and the pixels of the neck bone portion in the depth image. In addition, since the reference pose includes both arms folded, the motion estimation method can easily distinguish between the upper-arm pixels 430 and the lower-side pixels 420 with reference to the elbow. Further, the reference pose may include a posture in which both knees are bent, the left foot is directed forward, and the right foot is positioned posteriorly. In this case, the motion estimation method facilitates the use of pixels 440 in the left thigh portion, pixels 450 in the left shin portion, pixels 460 in the right thigh portion, and pixels 470 in the right shin portion .

Also, referring to FIG. 4B, the motion estimation method may determine the radius parameters of each of the plurality of joints such that the number of depth pixels corresponding to each of the plurality of joints is maximized based on the random sampling technique.

More specifically, the motion estimation method can randomly select the radius of each joint and a pair of depth pixels corresponding to that joint. The motion estimation method may define a cylinder on a three-dimensional space based on pairs of radial and depth pixels of randomly selected joints. The motion estimation method can then evaluate the cylinder by counting the number of inliers and the number of outliers in the image projected on a two-dimensional plane of the defined cylinder.

For example, the motion estimation method may evaluate the defined cylinder based on equation (21).

는 i번째 조인트와 관련된 인라이어들의 수이고,

는 i번째 조인트와 관련된 아웃라이어들의 수이다. 따라서, 동작 추정 방법은 각각의 조인트들에서 인라이어들의 수가 최대가 되고, 아웃라이어들의 수가 최소가 되도록 각각의 조인트들의 반지름 파라미터를 결정할 수 있다.here,

Is the number of inducers associated with the ith joint,

Is the number of outliers associated with the i-th joint. Thus, the motion estimation method can determine the radius parameter of each joint to maximize the number of inlayers at each joint and minimize the number of outliers.

The motion estimation method according to another embodiment may utilize the symmetric nature of the skeleton model.

More specifically, based on the symmetry of the skeleton model, the motion estimation method can recognize a joint pair that is symmetric to each other among a plurality of joints. The motion estimation method calculates an average of a radius parameter of the first joint included in the joint pair and a radius parameter of the second joint included in the joint pair and calculates a radius parameter of the first joint and a radius parameter of the second joint, The average can be modified.

Work In the embodiment  Motion estimation apparatus using a depth image according to the present invention

5 is a block diagram illustrating an operation estimating apparatus using a depth image according to an embodiment.

Referring to FIG. 5, a motion estimation apparatus 500 according to an embodiment includes a processor 510 and a depth image acquisition unit 520.

At this time, the depth image acquisition unit 520 may acquire depth images of at least two consecutive frames.

The processor 510 further includes a parameter calculator 511 for calculating first parameters related to the depth flow between adjacent frames based on the depth images, a parameter acquiring unit 512 for acquiring second parameters related to the motion fryers 512) and a pose change amount estimating unit 513 for estimating a change amount of the pause between at least two consecutive frames based on the first parameters and the second parameters.

Here, each of the motion fliers may include information corresponding to a predetermined pose.

1 to 4 may be applied to each of the modules shown in FIG. 5, so that a detailed description thereof will be omitted.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (23)

A motion estimation method using a depth image,
Obtaining depth images of at least two consecutive frames;
Calculating first parameters related to a depth flow between adjacent frames based on the depth images;
Motion priors - acquiring second parameters associated with each of the motion fliers comprising information corresponding to a predetermined pose; And
Estimating a change amount of a pause between the at least two consecutive frames based on the first parameters and the second parameters
/ RTI > The method according to claim 1,
Wherein the amount of change of the pose is modeled by a variation amount of each of a plurality of joint angles, and each of the plurality of joint angles corresponds to any one of a plurality of joints included in a skeleton model of a target object. The method according to claim 1,
The step of estimating the amount of change of the pose
Determining an amount of change in the pose such that a sum of an absolute value associated with the first parameters and an absolute value associated with the second parameters is at a minimum;
/ RTI > The method according to claim 1,
The step of estimating the amount of change of the pose
Determining a variation amount of the pose such that a variation amount of the pose is minimized while considering a constraint associated with the first parameters and a constraint associated with the second parameters
/ RTI > The method according to claim 1,
Wherein the motion fliers are pre-stored in a database and the second parameters are generated by applying Principal Components Analysis (PCA) techniques to the motion fryers. The method according to claim 1,
Wherein the motion fliers are pre-stored in a database and the second parameters are generated by applying a Mixture of Factor Analyzers (MFA) technique to the motion fryers. The method according to claim 1,
The step of acquiring the second parameters
Extracting K neighboring pauses similar to the pose in the tth frame of the motion fryers based on a pose in the tth frame; And
Generating the second parameters by applying a local principal component analysis (LPCA) technique to the K neighboring pose
Lt; / RTI >
Wherein the t < th > frame is included in the at least two consecutive frames, and the value of K is dependent on a predetermined similarity determination threshold. 8. The method of claim 7,
The step of extracting the K neighboring pose
Calculating in parallel a similarity value between a pose in the t < th > frame and at least a portion of the motion fryers using a processor that parallelizes a plurality of threads; And
Selecting the K neighboring pose based on the parallelly calculated similarity values and the predetermined similarity determination threshold
/ RTI > 8. The method of claim 7,
The step of extracting the K neighboring pose
Generating first quantization results and a second quantization result by applying a product quantization technique to each of the motion fliers and the pose in the t < th >frame; And
Calculating pose in the t-th frame and similarity values between the motion fliers based on the first quantization results and the second quantization result
/ RTI > The method according to claim 1,
Obtaining a reference depth image of a target object that takes a predetermined reference pose;
Classifying the plurality of pixels included in the reference depth image into one of a plurality of joints included in the skeleton model of the target object based on the information related to the predetermined reference pose; And
Determining a radius parameter of each of the plurality of joints such that the number of depth pixels corresponding to each of the plurality of joints is maximized based on a random sampling technique
≪ / RTI > 11. The method of claim 10,
Recognizing a joint pair symmetric to each other among the plurality of joints based on a symmetric nature of the skeleton model;
Calculating an average of a radius parameter of a first joint included in the joint pair and a radius parameter of a second joint included in the joint pair; And
Modifying each of the radius parameter of the first joint and the radius parameter of the second joint to the average
≪ / RTI > A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 1 to 11. A motion estimation apparatus using a depth image,
A processor; And
A depth image acquiring unit for acquiring depth images of at least two consecutive frames,
Lt; / RTI >
The processor
A parameter calculation unit for calculating first parameters related to a depth flow between adjacent frames based on the depth images;
Motion priors - a parameter acquiring unit for acquiring second parameters associated with each of the motion fliers, wherein each of the motion fiers includes information corresponding to a predetermined pose; And
And a pose variation amount estimating unit estimating a variation amount of a pause between at least two consecutive frames based on the first parameters and the second parameters,
And a motion estimation unit. 14. The method of claim 13,
Wherein the amount of change of the pose is modeled by a variation amount of each of a plurality of joint angles, and each of the plurality of joint angles corresponds to any one of a plurality of joints included in a skeleton model of a target object. 14. The method of claim 13,
Wherein the pose change amount estimating unit determines the amount of change of the pose such that a sum of an absolute value associated with the first parameters and an absolute value associated with the second parameters is minimum. 14. The method of claim 13,
Wherein the pose change amount estimating unit determines a change amount of the pose such that a change amount of the pose is minimized while considering a constraint related to the first parameters and a constraint associated with the second parameters. 14. The method of claim 13,
Wherein the motion fliers are stored in advance in a database, and the second parameters are generated by applying Principal Components Analysis (PCA) techniques to the motion fryers. 14. The method of claim 13,
Wherein the motion fliers are pre-stored in a database and the second parameters are generated by applying a Mixture of Factor Analyzers (MFA) technique to the motion fryers. 14. The method of claim 13,
The parameter obtaining unit
A neighbor pose extractor for extracting K neighbor pose similar to the pose in the t-th frame of the motion fics based on the pose in the t-th frame; And
And generating the second parameters by applying a Local Principal Components Analysis (LPCA) technique to the K neighboring pose (s)
Lt; / RTI >
Wherein the t < th > frame is included in the at least two consecutive frames, and the value of K is dependent on a predetermined similarity determination threshold. 20. The method of claim 19,
The neighboring pose extracting unit
A parallel processing unit for calculating in parallel the similarity values between a pose in the t-th frame and at least a part of the motion fryers; And
A neighbor pose selection unit for selecting the K neighbor pose based on the parallel-calculated similarity values and the predetermined similarity determination threshold,
And a motion estimation unit. 20. The method of claim 19,
The neighboring pose extracting unit
A product quantizer for generating a first quantization result and a second quantization result by applying a product quantization technique to each of the motion fliers and the pose in the t-th frame; And
Calculating a similarity value between the motion fliers and the pose in the t-th frame based on the first quantization results and the second quantization result,
And a motion estimation unit. 14. The method of claim 13,
Wherein the depth image obtaining unit obtains a reference depth image of a target object that takes a predetermined reference pose,
The processor
A classifier for classifying a plurality of pixels included in the reference depth image into one of a plurality of joints included in the skeleton model of the target object based on information associated with the predetermined reference pose; And
A radius parameter determining unit for determining a radius parameter of each of the plurality of joints so that the number of depth pixels corresponding to each of the plurality of joints becomes a maximum based on a random sampling technique,
Further comprising: 23. The method of claim 22,
The processor
Recognizing a symmetric nature of the skeleton model, recognizing a symmetric pair of the plurality of joints, determining a radial parameter of the first joint included in the joint pair, Calculating an average of the radius parameters of the joint and modifying each of the radius parameter of the first joint and the radius parameter of the second joint to the average.

Images